Enhance API with OIDC support, including login and callback endpoints. Update environment variables for OIDC configuration in .env.example. Add new features to the catalog service for listing software families, Minecraft versions, and deployment regions. Implement server management actions such as kill, delete, and update in the servers module. Integrate feature flags for maintenance mode in server operations. Update pnpm-lock.yaml with new dependencies and versions.

2026-07-05 18:39:53 +02:00
parent bf36cb3159
commit 50cd4b3ffd
225 changed files with 17824 additions and 436 deletions
--- a/docs/IMPLEMENTATION_STATUS.md
+++ b/docs/IMPLEMENTATION_STATUS.md
@@ -1,8 +1,32 @@
 # HexaHost GameCloud — Implementation Status

-Last updated: 2026-06-26
+Last updated: 2026-07-05

-## Current phase: WHMCS hardening — mTLS + Product mappings ✅
+## Current phase: MVP complete — hardening & production readiness
+
+### Sprint 3 (2026-07-05)
+
+| Area | Status |
+|------|--------|
+| Members UI (server access tab) | Done |
+| Organizations / Support / Abuse APIs | Done |
+| Compliance export (S3) + account deletion | Done |
+| Cron-parser schedules (API + worker) | Done |
+| Global idempotency interceptor | Done |
+| Feature flag maintenance_mode | Done |
+| WHMCS PHP start/stop/restart/backup | Done |
+| Worker catalog-sync + notification dispatch | Done |
+| Gap-closure DB migration (40 tables) | Done |
+| E2E smoke tests extended | Done |
+
+### Next: Production hardening
+
+- Penetration test, integration API IP allowlist
+- Client-side backup encryption, Seccomp/AppArmor profiles
+- ClamAV upload scanning
+- Full E2E regression suite in CI
+
+## Previous phase: WHMCS hardening — mTLS + Product mappings ✅

 ### mTLS integration API

--- a/docs/MASTERPROMPT_GAP_ANALYSIS.md
+++ b/docs/MASTERPROMPT_GAP_ANALYSIS.md
@@ -0,0 +1,86 @@
+# Masterprompt Gap-Analyse — HexaHost GameCloud
+
+Stand: 2026-07-05 (Sprint 1–3 abgeschlossen)
+
+Vergleich mit `HexaHost_GameCloud_Cursor_Masterprompt.md`. Legende: ✅ erledigt · 🟡 teilweise · ❌ offen · 🔜 Post-MVP
+
+---
+
+## Zusammenfassung
+
+| Bereich | Status |
+|---------|--------|
+| Auth, 2FA, Sessions, CSRF | ✅ |
+| Server-Lifecycle, Queue, Multi-Node | ✅ |
+| Console, Files, Properties, Worlds, Backups | ✅ |
+| Add-ons (Modrinth), Catalog | ✅ |
+| Billing / Idle-Stop / Credits | ✅ |
+| WHMCS Integration + mTLS + Actions | ✅ |
+| Members, Schedules, Players, Software-Reinstall | ✅ |
+| Notifications (In-App, E-Mail, Webhook, Discord) | ✅ |
+| Organizations, Support, Abuse | ✅ |
+| Compliance (Export, Deletion) | ✅ |
+| Feature Flags, Idempotency, OIDC, OTel | ✅ |
+| Radix UI, E2E, apps/docs | ✅ |
+| DB-Migration Gap-Closure | ✅ (SQL vorhanden) |
+
+---
+
+## 30. Abgeschlossene Sprint-Prioritäten
+
+### Sprint 1 (Gap-Closure)
+Login-Fix, CSRF, Admin-UI, öffentliche Seiten, Permissions, Server-State, WHMCS Renew/SSO, Deploy-Stubs, 25 Ops-Docs, Container-Härtung Node-Agent.
+
+### Sprint 2 (Feature-Completion)
+Datei-Upload/Archiv, Spieler-Mutationen, Software-Reinstall, Schedules-API, Notifications-REST, OTel, Playwright, FeatureFlags, OIDC, CurseForge, Radix UI, apps/docs, Idempotency, WHMCS actions API.
+
+### Sprint 3 (Hardening)
+- **Members-UI** (`server-access.tsx`) an Members-API
+- **Organizations / Support / Abuse** REST-Module
+- **Compliance**: S3-Export, Kontolöschung mit Anonymisierung + Server-Delete-Queue
+- **Cron-Parser** (API + Worker) für Schedules
+- **Idempotency** global registriert
+- **Feature Flag** `maintenance_mode`
+- **WHMCS PHP**: Start/Stop/Restart/Backup Buttons + ApiClient
+- **Worker**: Catalog-Sync (Modrinth), Notification-Dispatch (E-Mail/Webhook/Discord), Compliance-Tick
+- **Migration** `20260705160000_gap_closure_tables` (40 Tabellen)
+- **Security-Docs**: backup-encryption, container-hardening
+- **Impressum**, erweiterte E2E-Smoke-Tests
+
+---
+
+## 31. Post-MVP / Produktions-Härtung
+
+| Thema | Status |
+|-------|--------|
+| Kubernetes, Live-Migration | 🔜 |
+| Passkeys, Werbenetzwerke | 🔜 |
+| ZFS/Ceph Storage-Adapter | 🔜 |
+| Backup Client-Side-Verschlüsselung | 🔜 (Bucket-SSE dokumentiert) |
+| Seccomp/AppArmor Profile | 🔜 (Baseline Docker-Härtung ✅) |
+| ClamAV Upload-Scan | 🔜 |
+| Penetrationstest | 🔜 |
+| Integration-API IP-Allowlist | 🔜 |
+
+---
+
+## 32. Lokales Setup
+
+```bash
+cp .env.example .env
+pnpm dev:infra
+pnpm db:migrate    # oder pnpm db:push bei frischer DB
+pnpm db:seed
+pnpm bootstrap:admin -- --email admin@example.com --password 'Relation123!'
+pnpm dev
+```
+
+Login: `admin@example.com` / `Relation123!`
+
+---
+
+## 33. Pflege-Hinweis
+
+Bei weiteren Änderungen diese Datei und `docs/IMPLEMENTATION_STATUS.md` aktualisieren.
+
+*Erstellt: 2026-07-05 · Sprint 3: 2026-07-05*
--- a/docs/api/README.md
+++ b/docs/api/README.md
@@ -0,0 +1,51 @@
+# GameCloud API documentation
+
+The HexaHost GameCloud REST API is documented with OpenAPI (Swagger) and served by the API process.
+
+## Interactive documentation
+
+When the API is running:
+
+| Environment | URL |
+|-------------|-----|
+| Local development | http://localhost:3001/api/v1/docs |
+| Production | https://api.example.net/api/v1/docs |
+
+Replace `api.example.net` with your `API_HOST` / `API_URL` hostname.
+
+The OpenAPI JSON spec is available at `/api/v1/docs-json` on the same host.
+
+## Authentication
+
+| Audience | Method |
+|----------|--------|
+| Customer panel / users | Session cookie or bearer token after `POST /api/v1/auth/login` |
+| WHMCS integration | HMAC-signed requests — see [WHMCS security](../integrations/whmcs/security.md) |
+| Node agent | mTLS + enrollment token on WebSocket |
+| Admin | Elevated role on session |
+
+Public health endpoints (`/api/v1/health/*`) require no authentication.
+
+## Versioning
+
+All routes are prefixed with `/api/v1/`. Breaking changes require a new major prefix; WHMCS modules declare compatible `APIVersion` in module metadata.
+
+## Code location
+
+- Controllers: `apps/api/src/**/*.controller.ts`
+- Swagger setup: `apps/api/src/main.ts` (`SwaggerModule.setup('api/v1/docs', ...)`)
+- DTOs: colocated with controllers and in `packages/shared`
+
+## Generating a static export
+
+```bash
+curl -s http://localhost:3001/api/v1/docs-json -o openapi.json
+```
+
+Use with codegen tools or import into Postman/Insomnia.
+
+## Related
+
+- [Control plane operations](../operations/control-plane.md)
+- [WHMCS integration](../integrations/whmcs/installation.md)
+- [Threat model](../security/threat-model.md)
--- a/docs/integrations/whmcs/addons.md
+++ b/docs/integrations/whmcs/addons.md
@@ -0,0 +1,56 @@
+# WHMCS addon products
+
+Bill optional GameCloud features as WHMCS addon products linked to the main Minecraft service.
+
+## Supported addon types
+
+| WHMCS addon | GameCloud feature | Module action |
+|-------------|-------------------|---------------|
+| Extra backup retention | Extended backup slots | Usage / plan metadata sync |
+| Dedicated IP (future) | Reserved port/IP on node | Manual / Phase 10 |
+| Priority support | No technical change | Billing only |
+
+Technical addons call GameCloud APIs during `CreateAccount` / `TerminateAccount` on the addon service or via usage records — see [usage billing](../usage-billing.md).
+
+## Setup
+
+**Setup → Products/Services → Product Addons → Create New Addon**
+
+1. Assign to Minecraft parent products
+2. Module: **HexaHost GameCloud** (if addon provisioning implemented) or **None** for pure billing markers
+3. Map addon ID in GameCloud addon module **Mappings** (addon tab) when available
+
+## Linking to parent service
+
+GameCloud correlates via:
+
+- `externalServiceId` — parent WHMCS service ID
+- `externalAddonId` — addon service ID (usage export)
+
+Ensure parent service is provisioned before addon activation.
+
+## Lifecycle
+
+| WHMCS event | Expected behavior |
+|-------------|-------------------|
+| Addon ordered | Enable feature flag on server |
+| Addon suspended | Disable feature; do not delete data |
+| Addon terminated | Revert limits; purge addon-specific data |
+
+## Metered addons
+
+For usage-based billing (playtime, storage overage):
+
+1. GameCloud worker aggregates usage
+2. WHMCS polls via addon module or server module metric export
+3. WHMCS generates invoices — [usage billing](../usage-billing.md)
+
+## Reconciliation
+
+Orphaned addons (parent terminated, addon active) appear in **Reconciliation** dashboard. Run dry-run before live fix.
+
+## Related
+
+- [Products](products.md)
+- [Usage billing](../usage-billing.md)
+- [Reconciliation](../reconciliation.md)
--- a/docs/integrations/whmcs/configurable-options.md
+++ b/docs/integrations/whmcs/configurable-options.md
@@ -0,0 +1,64 @@
+# WHMCS configurable options
+
+Map WHMCS configurable options to GameCloud server parameters at provision and upgrade time.
+
+## Overview
+
+Configurable options let customers choose RAM tiers, Minecraft versions, or software types at order time. The server module reads selected options during `CreateAccount` and `ChangePackage`.
+
+**Preferred approach:** Use addon product mappings for plan slugs; use configurable options for customer-visible upgrades that WHMCS invoices.
+
+## Standard option layout
+
+| Option name | Type | Maps to |
+|-------------|------|---------|
+| Plan | Dropdown | GameCloud plan slug (via mapping resolver) |
+| Minecraft Version | Dropdown | `minecraftVersion` |
+| Software | Dropdown | `softwareFamily` |
+| Extra Backup Slot | Quantity | WHMCS addon product — see [addons](addons.md) |
+
+Option order matters if using fallback config option indices in legacy setups. Mappings table overrides text fields when an active row exists.
+
+## Dropdown values
+
+### Software family
+
+Values must match GameCloud enum exactly:
+
+`VANILLA`, `PAPER`, `PURPUR`, `FABRIC`, `FORGE`, `NEOFORGE`, `QUILT`
+
+### Minecraft version
+
+Use semver strings present in GameCloud catalog. Invalid versions fail provisioning with module log error.
+
+## Upgrade path
+
+When a customer upgrades via WHMCS configurable option change:
+
+1. WHMCS invokes `ChangePackage`
+2. Module sends new `planSlug` to GameCloud
+3. GameCloud resizes limits; may require server restart
+
+Communicate restart requirement in WHMCS product description.
+
+## Validation
+
+The addon **Mappings** page validates plan slugs before save. Configurable option text is **not** validated until order time — typos cause provisioning failures.
+
+## Pricing
+
+WHMCS handles all pricing. GameCloud never stores invoice amounts in WHMCS mode (`BILLING_PROVIDER=whmcs`).
+
+## Example: RAM upgrade option
+
+1. WHMCS configurable option group "Upgrades"
+2. Option "RAM Tier" with values mapping to plan slugs `starter`, `pro`, `enterprise`
+3. Product mapping links WHMCS product + option combination to slug via `ProductMapping.php` resolver
+
+For complex matrices, use separate WHMCS products instead of many conditional options.
+
+## Related
+
+- [Products](products.md)
+- [Product mappings](../product-mappings.md)
+- [Troubleshooting](troubleshooting.md)
--- a/docs/integrations/whmcs/configuration.md
+++ b/docs/integrations/whmcs/configuration.md
@@ -0,0 +1,77 @@
+# WHMCS configuration
+
+Global and per-product settings for the HexaHost GameCloud WHMCS integration.
+
+## Addon module (global)
+
+**Setup → Addon Modules → HexaHost GameCloud → Configure**
+
+| Setting | GameCloud env | Description |
+|---------|---------------|-------------|
+| GameCloud API URL | `API_URL` | Base URL, e.g. `https://api.example.net` — no trailing slash |
+| Integration ID | `WHMCS_INTEGRATION_ID` | Unique installation identifier |
+| API Secret | `WHMCS_API_SECRET` | HMAC shared secret (≥ 32 chars) |
+| Use mTLS client certificate | `INTEGRATION_MTLS_ENABLED` | Enable client cert — see [mtls](../mtls.md) |
+| Client certificate path | — | PEM path readable by PHP |
+| Client private key path | — | PEM path, restricted permissions |
+| CA bundle path | — | Trust anchor for GameCloud API |
+| Auto-reconcile on daily cron | — | Dry-run reconciliation after event poll |
+| Event poll batch size | — | Default 50 |
+
+## GameCloud side
+
+```env
+BILLING_PROVIDER=whmcs
+WHMCS_INTEGRATION_ID=prod-whmcs-main
+WHMCS_API_SECRET=<generated-secret>
+WHMCS_WEBHOOK_SECRET=<optional-for-future-webhooks>
+INTEGRATION_MTLS_ENABLED=true
+APP_URL=https://panel.example.net
+API_URL=https://api.example.net
+```
+
+Restart `api` after credential changes.
+
+## Server module (per product)
+
+Each WHMCS product using module `hexagamecloud` needs:
+
+1. **Module Name:** HexaHost GameCloud
+2. **Server:** Server entry pointing to API hostname
+3. **Config options** or addon mappings — see [products](products.md) and [product mappings](../product-mappings.md)
+
+### Default config options
+
+| Option | Purpose |
+|--------|---------|
+| GameCloud Plan Slug | Fallback plan if no mapping row |
+| Minecraft Version | Default version at provision |
+| Software Family | VANILLA, PAPER, etc. |
+
+Prefer **addon mappings** over raw config option text for production.
+
+## Cron
+
+WHMCS daily cron invokes:
+
+- Pending event poll from GameCloud
+- Optional reconciliation dry-run
+
+Ensure WHMCS cron runs at least every 5 minutes for timely suspensions; daily hook handles bulk sync.
+
+## Reverse proxy
+
+If WHMCS reaches GameCloud through nginx/Traefik, preserve headers for mTLS fingerprint forwarding. Example: `deploy/nginx/integration-mtls.conf.example`.
+
+## Staging checklist
+
+- [ ] Integration ID unique per environment
+- [ ] Test order on hidden WHMCS product
+- [ ] SSO login to panel works ([sso](../sso.md))
+- [ ] Suspend / unsuspend updates server state within 60s
+
+## Related
+
+- [Installation](installation.md)
+- [Security](security.md)
+- [Troubleshooting](troubleshooting.md)
--- a/docs/integrations/whmcs/installation.md
+++ b/docs/integrations/whmcs/installation.md
@@ -0,0 +1,93 @@
+# WHMCS installation
+
+Install the HexaHost GameCloud provisioning and addon modules on WHMCS 8.x+.
+
+## Prerequisites
+
+- WHMCS with SSL (HTTPS admin and client area)
+- GameCloud control plane deployed (`deploy/compose/compose.prod.yml`)
+- `BILLING_PROVIDER=whmcs` on GameCloud
+- Matching credentials prepared (`WHMCS_INTEGRATION_ID`, `WHMCS_API_SECRET`)
+
+## Package contents
+
+Source tree: `integrations/whmcs/`
+
+| Component | WHMCS path |
+|-----------|------------|
+| Server module | `/modules/servers/hexagamecloud/` |
+| Addon module | `/modules/addons/hexagamecloud/` |
+| Hooks | `/includes/hooks/hexagamecloud.php` |
+
+Build a distributable ZIP:
+
+```bash
+cd integrations/whmcs/packaging
+./build-package.sh
+```
+
+## Installation steps
+
+### 1. Copy server module
+
+```bash
+cp -r integrations/whmcs/modules/servers/hexagamecloud /path/to/whmcs/modules/servers/
+```
+
+**Setup → Products/Services → Servers → Create New Server**
+
+- Type: **HexaHost GameCloud**
+- Hostname: `api.example.net` (GameCloud API host only — no path)
+- Username / Password: leave empty (credentials are per-product module settings)
+
+### 2. Copy addon module
+
+```bash
+cp -r integrations/whmcs/modules/addons/hexagamecloud /path/to/whmcs/modules/addons/
+```
+
+**Setup → Addon Modules → HexaHost GameCloud → Activate**
+
+Configure global settings — see [configuration](configuration.md).
+
+### 3. Install hooks
+
+```bash
+cp integrations/whmcs/hooks/hexagamecloud.php /path/to/whmcs/includes/hooks/
+```
+
+Hooks enable daily event polling and optional auto-reconciliation.
+
+### 4. Configure GameCloud
+
+On the control plane `.env.prod`:
+
+```env
+BILLING_PROVIDER=whmcs
+WHMCS_INTEGRATION_ID=prod-whmcs-main
+WHMCS_API_SECRET=<same-as-addon>
+INTEGRATION_MTLS_ENABLED=true
+```
+
+Restart API after changes.
+
+### 5. Verify connectivity
+
+In WHMCS: **Addons → HexaHost GameCloud → Dashboard**
+
+Expected: green health, catalog plan count, zero pending critical events.
+
+## File permissions
+
+WHMCS web user must read module PHP files. Do not make `lib/` or `templates/` web-writable.
+
+## Multi-environment
+
+Use separate `WHMCS_INTEGRATION_ID` values for staging and production GameCloud installations. Never point staging WHMCS at production GameCloud.
+
+## Related
+
+- [Configuration](configuration.md)
+- [Products](products.md)
+- [Security](security.md)
+- [Addon module](../addon-module.md)
--- a/docs/integrations/whmcs/products.md
+++ b/docs/integrations/whmcs/products.md
@@ -0,0 +1,71 @@
+# WHMCS products
+
+Creating WHMCS products that provision HexaHost GameCloud Minecraft servers.
+
+## Product setup
+
+**Setup → Products/Services → Create New Product**
+
+| Field | Value |
+|-------|-------|
+| Product type | Other |
+| Module | HexaHost GameCloud (`hexagamecloud`) |
+| Server group | Server entry with GameCloud API hostname |
+
+## Plan mapping
+
+Each product must resolve to a GameCloud **plan slug** (defines RAM, CPU, backup slots, etc.).
+
+**Recommended:** Configure via addon **Mappings** tab — [product mappings](../product-mappings.md)
+
+**Fallback:** Set config option **GameCloud Plan Slug** on the product module settings.
+
+Validate slugs against live catalog:
+
+```http
+GET /api/v1/catalog/plans
+```
+
+Only active plans appear in the addon mapping dropdown.
+
+## Example products
+
+| WHMCS product name | Plan slug | Target audience |
+|--------------------|-----------|-----------------|
+| Minecraft Starter | `starter` | 2 GB RAM, 10 slots |
+| Minecraft Pro | `pro` | 4 GB RAM, backup included |
+| Minecraft Enterprise | `enterprise` | Custom limits |
+
+Plan slugs are defined in GameCloud admin/catalog — not in WHMCS.
+
+## Software defaults
+
+Set per mapping or config options:
+
+- **Minecraft Version** — e.g. `1.21.1`
+- **Software Family** — `VANILLA`, `PAPER`, `PURPUR`, `FABRIC`, `FORGE`, etc.
+
+Customers may change software via panel after provision if plan allows.
+
+## Billing cycle
+
+WHMCS owns billing cycles. GameCloud receives `externalServiceId` and syncs lifecycle (suspend on overdue invoice via WHMCS module actions or events).
+
+Usage-based overages: see [usage billing](../usage-billing.md).
+
+## Client area
+
+The server module renders status and **Login to Panel** (SSO). Domain field is used as display name default (`server-{serviceId}` if empty).
+
+## Testing
+
+1. Create admin-only product group
+2. Place test order with 100% discount
+3. Confirm `CreateAccount` success in module log
+4. SSO into panel and verify server RUNNING or STARTING
+
+## Related
+
+- [Configurable options](configurable-options.md)
+- [Addons](addons.md)
+- [Installation](installation.md)
--- a/docs/integrations/whmcs/security.md
+++ b/docs/integrations/whmcs/security.md
@@ -0,0 +1,79 @@
+# WHMCS integration security
+
+Security controls for the API-only link between WHMCS and HexaHost GameCloud.
+
+## Trust model
+
+- WHMCS is the **commercial system of record** (payments, invoices)
+- GameCloud is the **technical system of record** (servers, nodes)
+- Integration is **one-way authenticated API** — no shared database
+
+## Authentication layers
+
+### 1. HMAC request signing (required)
+
+Every request to `/api/v1/integrations/whmcs/*` includes:
+
+| Header | Purpose |
+|--------|---------|
+| `X-HGC-Integration-Id` | Installation identifier |
+| `X-HGC-Timestamp` | Unix epoch seconds |
+| `X-HGC-Signature` | HMAC-SHA256 of canonical request |
+
+Secret: `WHMCS_API_SECRET` — rotate via [secrets](../../security/secrets.md).
+
+Clock skew tolerance: ± 300 seconds. Sync NTP on both hosts.
+
+### 2. mTLS (production recommended)
+
+When `INTEGRATION_MTLS_ENABLED=true`:
+
+- Client certificate required from WHMCS host
+- Fingerprint registered in GameCloud installation record
+- See [mtls](../mtls.md)
+
+### 3. Network controls
+
+- Allowlist WHMCS egress IP at firewall or Traefik
+- No public exposure of WHMCS admin URL without 2FA
+- API only on HTTPS (`api.example.net`)
+
+## WHMCS hardening
+
+| Control | Recommendation |
+|---------|----------------|
+| Admin 2FA | Required for all staff |
+| File permissions | Module PHP not writable by web user |
+| API Secret storage | WHMCS encrypted settings storage |
+| Module logs | Restrict **Utilities → Logs** access |
+| Hooks file | `includes/hooks/hexagamecloud.php` owned by root/deploy user |
+
+## Least privilege
+
+WHMCS server module credentials are **not** stored in WHMCS server username/password fields — only in addon encrypted settings. GameCloud API keys for customers are separate from integration secret.
+
+## Audit
+
+GameCloud logs integration calls with integration ID and action. WHMCS module log redacts secrets. Correlate by timestamp and `externalServiceId`.
+
+## Incident response
+
+On suspected secret leak:
+
+1. Rotate `WHMCS_API_SECRET` on GameCloud first, then WHMCS addon
+2. Review module log for anomalous provisioning
+3. Run reconciliation dry-run
+4. Re-register mTLS fingerprint if cert compromised
+
+See [incident response](../../operations/incident-response.md).
+
+## Compliance
+
+- PCI: WHMCS handles card data; GameCloud never receives PAN
+- GDPR: Customer PII flows in `upsertClient` — subject to [data retention](../../security/data-retention.md)
+
+## Related
+
+- [Configuration](configuration.md)
+- [mTLS](../mtls.md)
+- [Troubleshooting](troubleshooting.md)
--- a/docs/integrations/whmcs/troubleshooting.md
+++ b/docs/integrations/whmcs/troubleshooting.md
@@ -0,0 +1,101 @@
+# WHMCS troubleshooting
+
+Common integration failures between WHMCS and HexaHost GameCloud.
+
+## Diagnostic tools
+
+| Tool | Location |
+|------|----------|
+| WHMCS module log | Utilities → Logs → Module Log |
+| GameCloud addon dashboard | Addons → HexaHost GameCloud |
+| API health | `GET https://api.example.net/api/v1/health/live` |
+| OpenAPI | `/api/v1/docs` — [API README](../../api/README.md) |
+| Reconciliation | Addon → Reconciliation tab |
+
+## Authentication errors
+
+### `Invalid signature` / 401
+
+- Confirm `WHMCS_API_SECRET` matches exactly on both sides (no trailing newline)
+- Check `WHMCS_INTEGRATION_ID` matches addon **Integration ID**
+- Verify server clock (NTP): `X-HGC-Timestamp` skew > 5 min fails
+- Ensure **GameCloud API URL** has no trailing slash
+
+### mTLS / certificate errors
+
+- `INTEGRATION_MTLS_ENABLED=true` but fingerprint not registered → register in addon **mTLS** tab
+- Wrong CA bundle on WHMCS → point to GameCloud CA PEM
+- Reverse proxy not forwarding `X-HGC-Client-Cert-Fingerprint` → see [mtls](../mtls.md)
+
+## Provisioning failures
+
+### `CreateAccount` returns error string
+
+1. Read WHMCS module log full message
+2. Common causes:
+   - Invalid plan slug — fix [product mapping](../product-mappings.md)
+   - Catalog plan inactive
+   - No eligible game node (`NO_NODE_AVAILABLE`)
+   - Invalid Minecraft version for software family
+
+### Server stuck Pending in WHMCS
+
+- GameCloud provision may have succeeded — check addon dashboard service count
+- Run reconciliation dry-run
+- Manual SSO to panel to verify server exists
+
+## Lifecycle sync
+
+### Suspend / unsuspend not applied
+
+- WHMCS module action failed silently — check module log
+- GameCloud worker backlog — check Redis / worker logs
+- Event poll not running — verify WHMCS cron active
+
+### Terminate left data on node
+
+- Terminate API may have failed mid-flight — admin force-terminate in panel
+- Node offline during terminate — retry when node ONLINE
+
+## SSO issues
+
+See dedicated [SSO guide](../sso.md). Typical fixes:
+
+- `APP_URL` mismatch between GameCloud and expected redirect
+- Clock skew
+- Service not linked to correct `externalServiceId`
+
+## Usage billing gaps
+
+- Usage export cron not running
+- Mismatch between WHMCS service ID and GameCloud external ID — reconciliation
+
+## Network
+
+```bash
+# From WHMCS host
+curl -v https://api.example.net/api/v1/health/live
+openssl s_client -connect api.example.net:443 -servername api.example.net
+```
+
+Firewall must allow outbound 443 from WHMCS to GameCloud API.
+
+## Enable debug logging
+
+WHMCS: enable module debug in **System Settings → General Settings** (temporarily).
+
+GameCloud: set `LOG_LEVEL=debug` on API, reproduce, revert to `info`.
+
+## Escalation data to collect
+
+- WHMCS service ID and timestamp of failure
+- Module log entry (redact secrets)
+- GameCloud request ID from API response header if present
+- Integration ID and API version
+
+## Related
+
+- [Security](security.md)
+- [Configuration](configuration.md)
+- [Incident response](../../operations/incident-response.md)
+- [Reconciliation](../reconciliation.md)
--- a/docs/integrations/whmcs/upgrades.md
+++ b/docs/integrations/whmcs/upgrades.md
@@ -0,0 +1,74 @@
+# WHMCS module upgrades
+
+Upgrade paths for `integrations/whmcs` server and addon modules alongside GameCloud API releases.
+
+## Version compatibility
+
+Check release notes for minimum pairs:
+
+| GameCloud API | Server module | Addon module |
+|---------------|---------------|--------------|
+| 1.0.x | 1.0.x | 1.0.x |
+| 1.1.x | 1.1.x | 1.1.0+ (SQL migrations) |
+
+Module metadata declares `APIVersion` in `hexagamecloud_MetaData()`.
+
+## Upgrade order
+
+1. **GameCloud API** — deploy new control plane ([upgrades](../../operations/upgrades.md))
+2. **Database migrations** — addon SQL in `modules/addons/hexagamecloud/sql/upgrade-*.sql`
+3. **WHMCS addon module** — copy files, run upgrade hook
+4. **WHMCS server module** — copy files (usually no DB change)
+5. **Verify** — dashboard health, test suspend/unsuspend
+
+Never upgrade WHMCS modules before the API when release notes specify API-first ordering.
+
+## Addon SQL migrations
+
+WHMCS runs `hexagamecloud_upgrade()` on version bump. Manual apply if needed:
+
+```bash
+mysql whmcs < integrations/whmcs/modules/addons/hexagamecloud/sql/upgrade-1.1.0.sql
+```
+
+Back up WHMCS database before upgrade.
+
+## File deployment
+
+```bash
+# From packaging script output or git
+cp -r integrations/whmcs/modules/addons/hexagamecloud /path/to/whmcs/modules/addons/
+cp -r integrations/whmcs/modules/servers/hexagamecloud /path/to/whmcs/modules/servers/
+cp integrations/whmcs/hooks/hexagamecloud.php /path/to/whmcs/includes/hooks/
+```
+
+Clear WHMCS template cache if admin UI looks stale.
+
+## Zero-downtime considerations
+
+- Running game servers unaffected by module file update
+- Brief API restart during GameCloud upgrade may delay WHMCS module calls — retry logic in `ApiClient.php` handles transient 502
+- Place WHMCS in maintenance mode only if API breaking change requires it
+
+## Rollback
+
+1. Restore previous module files from backup
+2. Roll back GameCloud images if API incompatible
+3. Restore WHMCS DB if SQL migration ran
+
+Product mappings table is forward-compatible unless release notes say otherwise.
+
+## Packaging
+
+```bash
+cd integrations/whmcs/packaging
+./build-package.sh
+```
+
+Distribute ZIP to production WHMCS via change-controlled deploy.
+
+## Related
+
+- [Installation](installation.md)
+- [Troubleshooting](troubleshooting.md)
+- [Control plane upgrades](../../operations/upgrades.md)
--- a/docs/operations/backup-restore.md
+++ b/docs/operations/backup-restore.md
@@ -0,0 +1,108 @@
+# Backup and restore
+
+This runbook covers backing up and restoring HexaHost GameCloud control plane data. Game server world data is backed up separately via the in-product backup feature (stored in object storage).
+
+## Scope
+
+| Component | Method | Frequency |
+|-----------|--------|-----------|
+| PostgreSQL | `pg_dump` + WAL archiving | Daily full; continuous WAL (recommended) |
+| Redis | RDB snapshot (`BGSAVE`) | Hourly if non-transient keys matter |
+| Object storage (MinIO/S3) | Replication or `mc mirror` | Continuous or daily |
+| Traefik ACME | Copy `acme.json` | After each cert issue |
+| `.env.prod` / secrets | Encrypted off-site vault | On change |
+| WHMCS mappings | DB export of `mod_hexagamecloud_*` | Weekly |
+
+## PostgreSQL backup
+
+### Logical dump (single host)
+
+```bash
+docker exec hexahost-gamecloud-prod-postgres-1 \
+  pg_dump -U gamecloud -Fc gamecloud > "gamecloud-$(date +%F).dump"
+```
+
+Store dumps encrypted off-site. Retain at least 30 daily backups for production.
+
+### Restore from dump
+
+1. Stop API and worker to prevent writes:
+
+   ```bash
+   docker compose -f deploy/compose/compose.prod.yml stop api worker
+   ```
+
+2. Drop and recreate database (destructive):
+
+   ```bash
+   docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     psql -U gamecloud -c "DROP DATABASE gamecloud WITH (FORCE);"
+   docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     psql -U gamecloud -c "CREATE DATABASE gamecloud OWNER gamecloud;"
+   ```
+
+3. Restore:
+
+   ```bash
+   cat gamecloud-2026-07-01.dump | docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     pg_restore -U gamecloud -d gamecloud --no-owner --role=gamecloud
+   ```
+
+4. Start services and verify:
+
+   ```bash
+   docker compose -f deploy/compose/compose.prod.yml start api worker
+   curl -s https://api.example.net/api/v1/health/live
+   ```
+
+### Point-in-time recovery (PITR)
+
+For managed PostgreSQL or self-hosted WAL archiving, enable `archive_mode` and restore to a timestamp before the incident. Document your WAL retention in [disaster-recovery](disaster-recovery.md).
+
+## Redis backup
+
+Redis holds job queues and ephemeral cache. For queue durability during maintenance:
+
+```bash
+docker exec hexahost-gamecloud-prod-redis-1 redis-cli BGSAVE
+docker cp hexahost-gamecloud-prod-redis-1:/data/dump.rdb ./redis-$(date +%F).rdb
+```
+
+Restore by stopping Redis, replacing `dump.rdb`, and restarting. Expect in-flight jobs to be reprocessed or dead-lettered.
+
+## Object storage backup
+
+```bash
+mc alias set prod http://minio:9000 "$S3_ACCESS_KEY" "$S3_SECRET_KEY"
+mc mirror prod/gamecloud /backup/gamecloud-$(date +%F)/
+```
+
+Verify checksums on a sample of backup objects (`backups/*/manifest.json`).
+
+## Game server backups (tenant data)
+
+Customer-initiated backups are stored under `s3://gamecloud/backups/{serverId}/`. Restore through the panel or admin API — not via PostgreSQL restore alone.
+
+## Pre-restore checklist
+
+- [ ] Incident ticket with restore target time documented
+- [ ] WHMCS placed in maintenance mode if billing sync is active
+- [ ] Node agents will reconnect automatically after API restore
+- [ ] DNS records remain valid (no restore needed for RFC2136 provider state in DB)
+
+## Verification after restore
+
+1. Admin login and random server detail page
+2. WHMCS addon dashboard health (`Addons → HexaHost GameCloud`)
+3. Worker queue processing (`failed` jobs near zero)
+4. Sample backup download from object storage
+
+## Automation
+
+Integrate dumps with your existing backup tool (restic, Borg, Velero). Ansible playbooks in `deploy/ansible/` provision hosts; schedule backups via cron or systemd timers on `cp-db` or the compose host.
+
+## Related
+
+- [Disaster recovery](disaster-recovery.md)
+- [Object storage](object-storage.md)
+- [Data retention](../security/data-retention.md)
--- a/docs/operations/disaster-recovery.md
+++ b/docs/operations/disaster-recovery.md
@@ -0,0 +1,80 @@
+# Disaster recovery
+
+Recovery objectives and procedures when a HexaHost GameCloud control plane or game node is lost.
+
+## Recovery objectives (default targets)
+
+| Tier | RPO | RTO | Notes |
+|------|-----|-----|-------|
+| Control plane (API, DB) | 1 h | 4 h | With WAL + hourly Redis snapshot |
+| Object storage (backups) | 15 min | 4 h | With cross-region replication |
+| Single game node | 0 (running servers) | 30 min | Reschedule + DNS unchanged if edge IP stable |
+| Full region loss | 24 h | 24 h | Requires warm standby or restore from off-site |
+
+Adjust these in your internal SLA and document approved deviations.
+
+## Failure scenarios
+
+### Control plane host lost
+
+1. Provision replacement via `deploy/ansible/control-plane.yml`
+2. Restore PostgreSQL from latest dump + WAL ([backup-restore](backup-restore.md))
+3. Restore MinIO data or repoint `S3_*` to replica bucket
+4. Deploy compose: `deploy/compose/compose.prod.yml`
+5. Restore Traefik `acme.json` or re-issue certificates
+6. Verify WHMCS integration (HMAC secret unchanged)
+
+### Database corruption
+
+1. Stop writes (`api`, `worker`)
+2. Restore to last known-good PITR timestamp
+3. Run WHMCS reconciliation dry-run from addon module
+4. Review audit log gap for the corruption window
+
+### Object storage unavailable
+
+- API continues serving metadata; backup restore and file uploads fail
+- Fail over to replica endpoint or restore MinIO volume from snapshot
+- Workers retry S3 operations with exponential backoff
+
+### All game nodes offline
+
+- Customers cannot start servers; running containers on nodes may continue until stopped
+- Edge gateway returns connection errors for join-to-start
+- Restore nodes in priority order; scheduler assigns new provisions when nodes return
+
+### WHMCS host lost
+
+GameCloud remains authoritative for technical state. Reinstall WHMCS module from `integrations/whmcs/`, restore WHMCS database, re-import product mappings from backup export.
+
+## Standby strategy (recommended)
+
+| Component | Standby approach |
+|-----------|------------------|
+| PostgreSQL | Streaming replica or managed HA |
+| Redis | Sentinel or managed Redis |
+| S3 | Cross-region replication |
+| Control plane VMs | IaC templates in `deploy/ansible/` |
+| Traefik certs | Shared `acme.json` on persistent volume |
+
+## Communication
+
+During DR:
+
+1. Status page update (if applicable)
+2. Internal war room with roles: coordinator, DB, network, WHMCS
+3. Customer comms after RTO estimate is known — WHMCS can send mass email
+
+## DR drill checklist (quarterly)
+
+- [ ] Restore PostgreSQL dump to staging environment
+- [ ] Restore sample backup from object storage to test server
+- [ ] Fail over DNS to standby edge IP (dry run)
+- [ ] WHMCS provisioning test order on staging
+- [ ] Document actual RPO/RTO vs targets
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Server migration](server-migration.md)
+- [Incident response](incident-response.md)
--- a/docs/operations/incident-response.md
+++ b/docs/operations/incident-response.md
@@ -0,0 +1,84 @@
+# Incident response
+
+Operational playbook for security and availability incidents affecting HexaHost GameCloud.
+
+## Severity levels
+
+| Level | Examples | Response time | Examples |
+|-------|----------|---------------|----------|
+| SEV-1 | API down, data breach suspected, all nodes offline | 15 min | Full platform outage |
+| SEV-2 | Single node down, WHMCS provision failures, DNS sync broken | 1 h | Partial customer impact |
+| SEV-3 | Elevatoration, non-critical bug, one customer server stuck | Next business day | Limited blast radius |
+
+## Initial response (all severities)
+
+1. **Acknowledge** — Assign incident commander and scribe
+2. **Triage** — Customer-facing vs internal-only impact
+3. **Communicate** — Internal channel + status page if SEV-1/2
+4. **Preserve evidence** — Do not reboot hosts before log capture if security-related
+
+## Common incidents
+
+### API unavailable (502/503)
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml ps
+docker compose -f deploy/compose/compose.prod.yml logs --tail=200 api
+curl -v https://api.example.net/api/v1/health/live
+```
+
+Check Traefik routing ([traefik](traefik.md)), PostgreSQL connectivity, disk full.
+
+### Worker queue backlog
+
+Inspect Redis queue depth and failed jobs in logs. Scale worker container or restart after fixing root cause (DNS provider down, S3 errors).
+
+### Node heartbeat stale
+
+Nodes not seen > 90s trigger alerts. Verify node-agent systemd status, outbound firewall to `wss://api.example.net/api/v1/nodes/ws`, certificate expiry.
+
+### Suspected credential leak
+
+1. Rotate affected secrets immediately ([secrets](../security/secrets.md)):
+   - `SESSION_SECRET` (invalidates all sessions)
+   - `WHMCS_API_SECRET` per installation
+   - `NODE_TOKEN` for compromised node
+   - `S3_ACCESS_KEY` / database password
+2. Review audit logs and WHMCS module call logs
+3. Enable mTLS if not already active for WHMCS integration
+
+### Tenant isolation concern (SEV-1)
+
+1. Disable affected API endpoints if exploit is active
+2. Capture request IDs from audit log
+3. Engage security lead; reference [threat model](../security/threat-model.md) T21
+
+## WHMCS-specific
+
+Provisioning failures often appear in WHMCS **Utilities → Logs → Module Log**. Cross-check GameCloud API logs with `X-HGC-Integration-Id` header.
+
+Run reconciliation dry-run before live fixes: [reconciliation](../integrations/whmcs/reconciliation.md).
+
+## Post-incident
+
+Within 5 business days:
+
+- [ ] Timeline document (UTC)
+- [ ] Root cause (5 whys)
+- [ ] Action items with owners
+- [ ] Update runbooks if gaps found
+
+## Contacts and escalation
+
+Document internally:
+
+- On-call rotation
+- WHMCS admin contact
+- DNS provider support
+- Hosting provider (game nodes)
+
+## Related
+
+- [Disaster recovery](disaster-recovery.md)
+- [Threat model](../security/threat-model.md)
+- [WHMCS troubleshooting](../integrations/whmcs/troubleshooting.md)
--- a/docs/operations/node-drain.md
+++ b/docs/operations/node-drain.md
@@ -0,0 +1,101 @@
+# Game node drain
+
+Gracefully remove a game node from scheduling before maintenance, decommissioning, or migration.
+
+## When to drain
+
+- OS kernel / Docker upgrade on the node
+- Hardware maintenance or rack move
+- Shrinking cluster capacity
+- Replacing a node with new hardware ([server migration](server-migration.md))
+
+Do **not** drain the last node in a region if active servers have no migration target.
+
+## Procedure
+
+### 1. Set node to DRAINING
+
+Via admin API:
+
+```http
+PATCH /api/v1/admin/nodes/{nodeId}
+Authorization: Bearer <admin-token>
+Content-Type: application/json
+
+{ "status": "DRAINING" }
+```
+
+The scheduler stops placing new servers on this node. Existing running servers continue until stopped by users or idle shutdown.
+
+### 2. Wait for natural shutdown (optional)
+
+If maintenance window allows, wait for idle shutdown policy to stop inactive servers. Monitor:
+
+```sql
+SELECT id, status, "nodeId" FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status = 'RUNNING';
+```
+
+### 3. Migrate or stop remaining servers
+
+For each running server:
+
+- **Preferred:** Customer stops server via panel; reprovision on another node on next start (scheduler picks eligible node)
+- **Admin:** Force stop via admin API, then start — triggers reschedule if original node is DRAINING
+- **Maintenance:** Communicate window to affected customers
+
+### 4. Confirm empty node
+
+```sql
+SELECT COUNT(*) FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status IN ('RUNNING', 'STARTING');
+```
+
+Expected: `0` before maintenance on Docker host.
+
+### 5. Stop node-agent
+
+```bash
+sudo systemctl stop hgc-node-agent
+```
+
+Configured in `deploy/systemd/node-agent.service`.
+
+### 6. Perform maintenance
+
+Apply updates via `deploy/ansible/game-node.yml` or manual steps. See [game node operations](game-node.md).
+
+### 7. Return to service or decommission
+
+**Return:**
+
+```http
+PATCH /api/v1/admin/nodes/{nodeId}
+{ "status": "ONLINE" }
+```
+
+```bash
+sudo systemctl start hgc-node-agent
+```
+
+**Decommission:**
+
+1. Revoke enrollment token in control plane
+2. Remove `GameNode` row or mark `OFFLINE` permanently
+3. Wipe `/var/lib/hgc-node` data if disks are reused
+
+## Edge cases
+
+| Situation | Action |
+|-----------|--------|
+| Server stuck STARTING during drain | Admin stop + investigate node-agent logs |
+| Port range exhausted on other nodes | Add node or expand range before drain |
+| Node loses connectivity mid-drain | Servers marked UNKNOWN; reconcile when agent returns |
+
+## Automation
+
+Future: automated drain API with `--wait-empty` timeout. Until then, use SQL + admin API as above.
+
+## Related
+
+- [Game node operations](game-node.md)
+- [Server migration](server-migration.md)
+- [Upgrades](upgrades.md)
--- a/docs/operations/object-storage.md
+++ b/docs/operations/object-storage.md
@@ -0,0 +1,90 @@
+# Object storage operations
+
+HexaHost GameCloud stores backups, world exports, mod archives, and other large blobs in S3-compatible object storage. Development uses MinIO via `deploy/compose/compose.dev.yml`; production uses MinIO or an external S3 provider.
+
+## Environment variables
+
+| Variable | Description |
+|----------|-------------|
+| `S3_ENDPOINT` | Internal URL, e.g. `http://minio:9000` in compose |
+| `S3_REGION` | Region identifier (`us-east-1` for MinIO) |
+| `S3_BUCKET` | Primary bucket (default `gamecloud`) |
+| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | Credentials — rotate regularly |
+| `S3_FORCE_PATH_STYLE` | `true` for MinIO; `false` for AWS S3 |
+
+## Production (MinIO in compose)
+
+The production stack in `deploy/compose/compose.prod.yml` runs MinIO on the internal network only — no public ports. API and worker reach it at `http://minio:9000`.
+
+Initial bucket creation is handled by `minio-init` on first deploy:
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up minio-init
+```
+
+### Admin console access
+
+Do not expose the MinIO console to the public internet. Options:
+
+- SSH tunnel: `ssh -L 9001:127.0.0.1:9001 cp-01`
+- VPN-only route to the control plane management network
+- Separate MinIO deployment with IAM and audit logging
+
+## External S3 (AWS, Wasabi, Hetzner Object Storage)
+
+Point `S3_ENDPOINT` at the provider URL and set `S3_FORCE_PATH_STYLE=false` where appropriate. Create the bucket manually and apply a bucket policy that:
+
+- Denies public read/write
+- Allows only the GameCloud service principal (access key)
+- Enables versioning for backup objects
+- Optionally enforces SSE-S3 or SSE-KMS
+
+Remove the `minio` and `minio-init` services from compose when using external storage.
+
+## Object layout
+
+| Prefix | Content | Retention |
+|--------|---------|-----------|
+| `backups/{serverId}/` | Scheduled server backups | Per plan + [data retention policy](../security/data-retention.md) |
+| `worlds/{serverId}/` | World export ZIPs | 7 days after download link expiry |
+| `uploads/{serverId}/` | User file uploads (staging) | Cleaned after successful install |
+| `catalog/` | Cached mod/plugin artifacts | Managed by catalog sync jobs |
+
+## Health and monitoring
+
+MinIO health endpoint: `GET /minio/health/live`
+
+Monitor:
+
+- Bucket size growth (per tenant and total)
+- Failed PUT/GET rates from API logs
+- Disk usage on MinIO volume `hexahost-gamecloud-minio`
+
+## Backup of the storage layer
+
+For self-hosted MinIO:
+
+1. Enable bucket versioning
+2. Replicate to a second bucket or provider (MinIO site replication or `mc mirror`)
+3. Include object storage in [backup-restore](backup-restore.md) runbooks
+
+For external S3, rely on provider replication and lifecycle rules; document RPO/RTO in [disaster-recovery](disaster-recovery.md).
+
+## Troubleshooting
+
+### `NoSuchBucket`
+
+Run `minio-init` or create the bucket manually with the configured `S3_BUCKET` name.
+
+### Signature errors
+
+Verify clock sync (NTP) on API/worker hosts. Check `S3_FORCE_PATH_STYLE` matches the provider.
+
+### Slow backup uploads
+
+Check network between game nodes (backup source) and storage. Large backups stream from node-agent → API → S3; consider dedicated storage nodes for high-volume deployments.
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Secrets management](../security/secrets.md)
--- a/docs/operations/server-migration.md
+++ b/docs/operations/server-migration.md
@@ -0,0 +1,79 @@
+# Server and host migration
+
+Moving HexaHost GameCloud workloads between hosts: control plane relocation, game node replacement, and WHMCS host migration.
+
+## Control plane migration
+
+Move from `cp-01-old` to `cp-01-new`:
+
+1. **Prepare new host** — `ansible-playbook -i inventories/production deploy/ansible/control-plane.yml`
+2. **Stop writes on old host** — `docker compose stop api worker`
+3. **Backup** — PostgreSQL dump, Redis RDB, MinIO mirror, `/opt/hexahost-gamecloud/.env.prod`, Traefik `acme.json`
+4. **Transfer** — Secure copy to new host (rsync over SSH, encrypted archive)
+5. **Restore** — Follow [backup-restore](backup-restore.md) on new host
+6. **Update DNS** — `panel.example.net` and `api.example.net` A/AAAA records to new IP
+7. **Start stack** — `docker compose -f deploy/compose/compose.prod.yml up -d`
+8. **Verify** — Health checks, WHMCS addon dashboard, node WebSocket reconnections
+9. **Decommission old host** — Secure wipe after 72h parallel run (optional)
+
+WHMCS requires no URL change if hostnames are unchanged. Update `GameCloud API URL` in addon settings only if `api.example.net` changes.
+
+## Game node migration
+
+Replace `node-01` with `node-02` while preserving customer servers:
+
+### Add replacement node first
+
+1. Create `GameNode` in admin API
+2. Deploy agent: `deploy/ansible/game-node.yml`
+3. Confirm `ONLINE` status
+
+### Drain old node
+
+Follow [node-drain](node-drain.md). Running servers must stop and start on the new node.
+
+### Data considerations
+
+- World data lives in Docker volumes on the node — **not** automatically migrated
+- Customers should create a backup before migration window
+- Admin can trigger backup via API, then restore after server starts on new node
+
+### DNS
+
+Join hostnames point to the **edge gateway**, not individual nodes. No DNS change required for node migration unless edge IP changes.
+
+## Edge gateway migration
+
+If `EDGE_PUBLIC_HOST` changes:
+
+1. Update `RFC2136` records or manual DNS for `*.play.example.net`
+2. Deploy edge gateway on new IP
+3. Worker DNS sync reconciles `server_dns_records` within 60s
+
+## WHMCS host migration
+
+1. Export WHMCS database and `mod_hexagamecloud_*` tables
+2. Copy `integrations/whmcs/modules/` to new WHMCS install
+3. Reconfigure addon: Integration ID, API Secret (unchanged on GameCloud side)
+4. Run reconciliation dry-run: **Addons → HexaHost GameCloud → Reconciliation**
+5. Re-register mTLS fingerprint if client certificate changes
+
+See [WHMCS installation](../integrations/whmcs/installation.md).
+
+## MinIO / S3 migration
+
+```bash
+mc mirror old/gamecloud new/gamecloud
+```
+
+Update `S3_ENDPOINT` and credentials in `.env.prod`, restart API and worker.
+
+## Rollback
+
+Keep old host powered but isolated for 72 hours. Rollback = revert DNS + restart old compose stack from pre-migration backup.
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Disaster recovery](disaster-recovery.md)
+- [Node drain](node-drain.md)
--- a/docs/operations/traefik.md
+++ b/docs/operations/traefik.md
@@ -0,0 +1,96 @@
+# Traefik operations
+
+HexaHost GameCloud exposes the customer panel (`web`) and REST API (`api`) through Traefik on the control plane host. Game nodes and the edge gateway are **not** routed through this stack.
+
+## Layout
+
+| Path | Purpose |
+|------|---------|
+| `deploy/traefik/traefik.yml` | Static config — entry points, ACME, providers |
+| `deploy/traefik/dynamic/gamecloud.yml` | Middleware, TLS options, optional file routers |
+| `deploy/compose/compose.prod.yml` | Docker labels for primary routing |
+
+## Prerequisites
+
+```bash
+docker network create traefik-network
+```
+
+Set in `.env.prod`:
+
+```env
+TRAEFIK_NETWORK=traefik-network
+WEB_HOST=panel.example.net
+API_HOST=api.example.net
+ACME_EMAIL=ops@example.net
+APP_URL=https://panel.example.net
+API_URL=https://api.example.net
+TRUSTED_PROXY_COUNT=1
+```
+
+## Run Traefik
+
+Example standalone container (adjust paths):
+
+```bash
+docker run -d \
+  --name traefik \
+  --restart always \
+  -p 80:80 -p 443:443 \
+  -v /var/run/docker.sock:/var/run/docker.sock:ro \
+  -v /opt/hexahost-gamecloud/deploy/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
+  -v /opt/hexahost-gamecloud/deploy/traefik/dynamic:/etc/traefik/dynamic:ro \
+  -v /opt/hexahost-gamecloud/letsencrypt:/letsencrypt \
+  --network traefik-network \
+  traefik:v3.2
+```
+
+Then start the application stack:
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d
+```
+
+## Routing
+
+| Host | Service | Health check |
+|------|---------|--------------|
+| `panel.example.net` | `web:3000` | `GET /de` |
+| `api.example.net` | `api:3001` | `GET /api/v1/health/live` |
+
+HTTP on port 80 redirects to HTTPS. API requests receive rate limiting (`100 req/s` average, burst `200`).
+
+## WHMCS integration mTLS
+
+WHMCS integration traffic uses the same `api.example.net` host. When `INTEGRATION_MTLS_ENABLED=true`, terminate client certificates at Traefik or nginx and forward `X-HGC-Client-Cert-Fingerprint`. See [WHMCS mTLS](../integrations/whmcs/mtls.md) and `deploy/nginx/integration-mtls.conf.example`.
+
+## Certificate renewal
+
+Let's Encrypt certificates are stored in `/letsencrypt/acme.json`. Back up this file before host migrations. Traefik renews automatically; monitor expiry via the Traefik dashboard or Prometheus metrics.
+
+## Troubleshooting
+
+### 502 Bad Gateway
+
+1. Confirm `api` / `web` containers are healthy: `docker compose ps`
+2. Verify both services attach to `traefik-network`
+3. Check label `traefik.docker.network` matches the external network name
+
+### Certificate errors
+
+1. Ensure ports 80 and 443 are reachable from the internet for HTTP-01 challenge
+2. Confirm `ACME_EMAIL` is valid
+3. Inspect Traefik logs: `docker logs traefik 2>&1 | jq .`
+
+### Wrong client IP in audit logs
+
+Increase `TRUSTED_PROXY_COUNT` on the API to match the number of reverse-proxy hops.
+
+## Ansible
+
+Control plane provisioning is described in `deploy/ansible/control-plane.yml`. After OS setup, copy Traefik and compose files to `/opt/hexahost-gamecloud` and enable systemd or compose restart policies.
+
+## Related
+
+- [Control plane operations](control-plane.md)
+- [Installation (development)](installation.md)
--- a/docs/operations/upgrades.md
+++ b/docs/operations/upgrades.md
@@ -0,0 +1,86 @@
+# Control plane upgrades
+
+Rolling upgrades for HexaHost GameCloud API, worker, web, and infrastructure containers on the control plane.
+
+## Version pinning
+
+Set in `.env.prod`:
+
+```env
+GAMECLOUD_IMAGE_TAG=1.2.0
+```
+
+Build and tag images in CI, or pull from your registry. Never deploy `:latest` in production without a rollback plan.
+
+## Pre-upgrade checklist
+
+- [ ] Read release notes and migration requirements (`docs/IMPLEMENTATION_STATUS.md`)
+- [ ] Fresh PostgreSQL backup ([backup-restore](backup-restore.md))
+- [ ] WHMCS addon/server module compatibility verified
+- [ ] Maintenance window communicated if API downtime expected
+- [ ] Staging deploy completed successfully
+
+## Standard rolling upgrade
+
+```bash
+cd /opt/hexahost-gamecloud
+
+# Pull/build new images
+export GAMECLOUD_IMAGE_TAG=1.2.0
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod pull
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod build
+
+# Run database migrations (if applicable — via API container one-shot)
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod run --rm api \
+  pnpm --filter @hexahost/api prisma migrate deploy
+
+# Recreate app containers one at a time
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d --no-deps worker
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d --no-deps api
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d --no-deps web
+```
+
+## Infrastructure upgrades
+
+| Service | Notes |
+|---------|-------|
+| PostgreSQL | Major version requires `pg_upgrade` or dump/restore — plan maintenance window |
+| Redis | Minor upgrades usually safe; snapshot before upgrade |
+| MinIO | Follow MinIO release notes; backup volume first |
+| Traefik | Test dynamic config with `traefik validate` equivalent (dry-run container) |
+
+## Zero-downtime considerations
+
+- Run at least two API instances behind Traefik for true zero-downtime (single-node compose has brief API restart)
+- Worker: only one active consumer per-correct for some jobs; use graceful shutdown (SIGTERM) before replace
+- Web: Next.js standalone reload causes short 502; upgrade during low traffic
+
+## Rollback
+
+```bash
+export GAMECLOUD_IMAGE_TAG=1.1.0
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d api worker web
+```
+
+If migrations ran forward-only, restore PostgreSQL from pre-upgrade dump.
+
+## WHMCS module upgrades
+
+See [WHMCS upgrades](../integrations/whmcs/upgrades.md). Upgrade GameCloud API before WHMCS server module when release notes specify API-first ordering.
+
+## Post-upgrade verification
+
+```bash
+curl -sf https://api.example.net/api/v1/health/live
+curl -sf https://panel.example.net/de
+```
+
+- Admin panel login
+- WHMCS test suspend/unsuspend on staging service
+- Worker queue depth normal
+- Node heartbeats within 30s
+
+## Related
+
+- [Control plane operations](control-plane.md)
+- [Node drain](node-drain.md) — upgrade game nodes separately
--- a/docs/security/backup-encryption.md
+++ b/docs/security/backup-encryption.md
@@ -0,0 +1,9 @@
+# Backup Encryption at Rest
+
+GameCloud backups are stored in S3-compatible object storage. Production deployments should enable:
+
+1. **Bucket SSE** — Server-side encryption (AES-256) on the backup bucket
+2. **Application envelope encryption** (roadmap) — Per-backup data keys wrapped with `ENCRYPTION_KEY`
+3. **Key rotation** — Documented in `docs/security/secrets.md`
+
+Current MVP stores backups without client-side encryption. Enable bucket policies before production.
--- a/docs/security/container-hardening.md
+++ b/docs/security/container-hardening.md
@@ -0,0 +1,14 @@
+# Container Hardening
+
+The node-agent applies baseline Docker hardening:
+
+- `cap_drop: ALL` with minimal adds
+- `security_opt: no-new-privileges`
+- PID limits and log rotation
+
+## Roadmap (Post-MVP)
+
+- **Seccomp** — Custom profile per software family under `deploy/seccomp/`
+- **AppArmor** — Node-level profiles for game server containers
+
+See `apps/node-agent/internal/docker/client.go` for current defaults.
--- a/docs/security/container-isolation.md
+++ b/docs/security/container-isolation.md
@@ -0,0 +1,71 @@
+# Container isolation
+
+How HexaHost GameCloud isolates Minecraft server workloads on game nodes.
+
+## Model
+
+Each customer server runs as a **non-privileged Docker container** on a dedicated game node. The node-agent creates and manages containers; customers never receive Docker socket access.
+
+```
+Control plane (trusted)  ──mTLS──►  Node agent (trusted)  ──►  Game container (untrusted)
+```
+
+## Container hardening (target profile)
+
+| Control | Setting |
+|---------|---------|
+| User | Non-root UID inside container |
+| Privileges | `--privileged=false` |
+| Capabilities | Drop ALL, add only required (none for vanilla Java) |
+| seccomp | Docker default or custom profile (Phase 2) |
+| AppArmor | Docker default profile |
+| Network | Bridge network per server; no `host` network mode |
+| Read-only root | Where compatible with selected software family |
+| Memory / CPU | cgroup limits from plan slug |
+| PIDs limit | Prevents fork bombs |
+
+## Network isolation
+
+- Game containers reach the internet for Minecraft auth and mod downloads only through controlled egress
+- No route to control plane PostgreSQL, Redis, or internal APIs
+- Node-agent management port (`9100`) bound to localhost or management VLAN only
+
+See threat T05 and T12 in [threat model](threat-model.md).
+
+## Storage isolation
+
+- Per-server data directory on node disk: `/var/lib/hgc-node/servers/{serverId}/`
+- Path traversal prevented server-side; symlinks rejected on archive extract
+- Backups uploaded to object storage — not readable by other tenants
+
+## Image policy
+
+- Only catalog-approved runtime images (digest-pinned)
+- Customers cannot push custom images
+- Periodic image vulnerability scan in CI (Phase 9)
+
+## Multi-tenant on one node
+
+The scheduler enforces RAM and server count limits per node. There is no shared filesystem between server containers except the read-only image layers.
+
+## Operator responsibilities
+
+| Task | Frequency |
+|------|-----------|
+| Docker security updates on nodes | Monthly |
+| Review AppArmor/seccomp profiles after Minecraft/Java major bumps | Per release |
+| Audit `docker ps` for unexpected privileged containers | Weekly |
+
+Bootstrap hardening via `deploy/ansible/game-node.yml`.
+
+## Verification
+
+- Attempt cross-container network scan from test server (should fail to reach agent port)
+- Confirm `docker inspect` shows `Privileged: false`
+- Penetration test before production go-live
+
+## Related
+
+- [Threat model](threat-model.md)
+- [Game node operations](../operations/game-node.md)
+- [Secrets](secrets.md)
--- a/docs/security/data-retention.md
+++ b/docs/security/data-retention.md
@@ -0,0 +1,66 @@
+# Data retention
+
+Retention periods and deletion procedures for HexaHost GameCloud customer and operational data.
+
+## Policy summary
+
+| Data type | Retention | Deletion trigger |
+|-----------|-----------|------------------|
+| Account profile | Life of account + 90 days | Account deletion request |
+| Game server world (live) | Life of service | Terminate in WHMCS / admin |
+| Automated backups | Per plan (7–30 days default) | Lifecycle job + S3 expiry |
+| Manual backup downloads | Link valid 24 h; object 7 days | Worker cleanup |
+| Audit log | 24 months | Scheduled purge job |
+| Application logs (Loki) | 30 days (`720h` in loki-config) | Loki compactor |
+| Session tokens | 7 days idle / 30 days max | Redis TTL |
+| WHMCS sync events | 90 days acknowledged | Addon retention SQL |
+| DNS record history | 12 months | DB maintenance |
+| Metering / usage samples | 13 months (billing) | Aggregated then purged |
+
+Adjust periods in contracts and WHMCS product terms. Document customer-facing retention in your privacy policy (German: *Datenschutzerklärung*).
+
+## Backup retention
+
+Backup objects live under `s3://{bucket}/backups/{serverId}/`. The worker applies plan-based retention:
+
+- Free/starter plans: 7 daily slots
+- Pro plans: 14 daily + 4 weekly
+- Enterprise: configurable via admin
+
+Expired backups are deleted from object storage; metadata rows in PostgreSQL are soft-deleted then purged after 7 days.
+
+## Termination flow
+
+When WHMCS calls `TerminateAccount`:
+
+1. Server stopped on node
+2. Final backup optional (plan-dependent)
+3. World data on node deleted within 24 h
+4. S3 prefix `backups/{serverId}/` deleted within 7 days
+5. PostgreSQL server row anonymised or hard-deleted per legal requirement
+
+## GDPR / DSGVO considerations
+
+- **Right to erasure:** Export then delete user row; cascade removes owned servers
+- **Data portability:** World export ZIP + backup download via panel
+- **Processor agreement:** Required between you (controller) and HexaHost GameCloud operator if different entity
+
+Logs may contain IP addresses and user IDs — treat as personal data where applicable.
+
+## Operational data
+
+| Store | Purge method |
+|-------|--------------|
+| PostgreSQL audit | `DELETE FROM audit_events WHERE created_at < NOW() - INTERVAL '24 months'` (automated job) |
+| Redis | TTL-based; no long-term PII |
+| MinIO access logs | Provider lifecycle rule |
+
+## Legal hold
+
+Suspend automated deletion for accounts under litigation hold. Flag in admin metadata; worker skips purge for affected `serverId`.
+
+## Related
+
+- [Backup and restore](../operations/backup-restore.md)
+- [Object storage](../operations/object-storage.md)
+- [Threat model](threat-model.md) — T16 backup integrity
--- a/docs/security/secrets.md
+++ b/docs/security/secrets.md
@@ -0,0 +1,95 @@
+# Secrets management
+
+Guidelines for generating, storing, and rotating secrets in HexaHost GameCloud deployments.
+
+## Classification
+
+| Secret | Location | Rotation |
+|--------|----------|----------|
+| `SESSION_SECRET` | `.env.prod` on control plane | Quarterly; invalidates sessions |
+| `ENCRYPTION_KEY` | `.env.prod` | Annually; requires re-encryption plan |
+| `POSTGRES_PASSWORD` | `.env.prod`, compose | Annually |
+| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | `.env.prod`, MinIO IAM | Quarterly |
+| `WHMCS_API_SECRET` | GameCloud `.env.prod` + WHMCS addon | On compromise; per installation |
+| `WHMCS_WEBHOOK_SECRET` | Both sides | On compromise |
+| `NODE_TOKEN` | Per-node enrollment | One-time at enroll; rotate on rebuild |
+| Node mTLS keys | `/etc/hgc-node/` on game nodes | Before cert expiry |
+| `EDGE_INTERNAL_API_KEY` | Edge gateway + API | Quarterly |
+| `RFC2136_KEY_SECRET` | API/worker env | Annually |
+| `STRIPE_*` | API env (if used) | Per Stripe dashboard policy |
+| Traefik `acme.json` | `/letsencrypt/` | Auto-renewed; backup only |
+
+## Generation
+
+```bash
+# 32-byte secrets (SESSION_SECRET, ENCRYPTION_KEY, API secrets)
+openssl rand -base64 32
+
+# WHMCS integration — minimum 32 characters
+openssl rand -hex 24
+```
+
+Never use development defaults from `.env.example` in production.
+
+## Storage rules
+
+1. **Never commit** `.env`, `.env.prod`, or key material to git
+2. Restrict file permissions: `chmod 600 .env.prod`, owner `gamecloud`
+3. Prefer secret manager (HashiCorp Vault, SOPS, cloud provider SM) over plain files for multi-host
+4. Ansible: use `ansible-vault` for inventory secrets referenced in `deploy/ansible/`
+
+## Distribution
+
+| From | To | Channel |
+|------|-----|---------|
+| Operator | Control plane | SSH + encrypted archive |
+| Control plane | WHMCS admin | Out-of-band (password manager share) |
+| Control plane | Game node | Enrollment token via secure ticket |
+
+WHMCS **Integration ID** is not secret but must match exactly on both sides.
+
+## Rotation procedures
+
+### SESSION_SECRET
+
+1. Generate new value
+2. Update `.env.prod`, restart `api` and `web`
+3. All users must log in again
+
+### WHMCS_API_SECRET
+
+1. Update GameCloud `.env.prod` and restart API
+2. Update WHMCS addon **API Secret** immediately after — expect brief auth failures
+3. No WHMCS module reinstall required
+
+### Database password
+
+1. `ALTER USER gamecloud PASSWORD '...'` in PostgreSQL
+2. Update `DATABASE_URL` / `POSTGRES_PASSWORD` in `.env.prod`
+3. Restart `api`, `worker`
+
+### Node token compromise
+
+1. Revoke token in admin API for affected `NODE_ID`
+2. Issue new enrollment token
+3. Reinstall node-agent with new token and fresh mTLS cert
+
+## Logging and redaction
+
+- Structured logs must not print env dumps or Authorization headers
+- WHMCS module calls redact `password`, `secret`, `apiSecret` in module logs
+- OpenTelemetry spans must not include query strings with tokens
+
+## Development vs production
+
+| Variable | Development | Production |
+|----------|-------------|------------|
+| `INTEGRATION_MTLS_ENABLED` | `false` | `true` (recommended) |
+| `NODE_TLS_SKIP_VERIFY` | `true` | `false` |
+| Default MinIO credentials | Allowed | **Forbidden** |
+
+## Related
+
+- [Threat model](threat-model.md)
+- [WHMCS security](../integrations/whmcs/security.md)
+- [Data retention](data-retention.md)