Enhance API with OIDC support, including login and callback endpoints. Update environment variables for OIDC configuration in .env.example. Add new features to the catalog service for listing software families, Minecraft versions, and deployment regions. Implement server management actions such as kill, delete, and update in the servers module. Integrate feature flags for maintenance mode in server operations. Update pnpm-lock.yaml with new dependencies and versions.

2026-07-05 18:39:53 +02:00
parent bf36cb3159
commit 50cd4b3ffd
225 changed files with 17824 additions and 436 deletions
--- a/docs/operations/backup-restore.md
+++ b/docs/operations/backup-restore.md
@@ -0,0 +1,108 @@
+# Backup and restore
+
+This runbook covers backing up and restoring HexaHost GameCloud control plane data. Game server world data is backed up separately via the in-product backup feature (stored in object storage).
+
+## Scope
+
+| Component | Method | Frequency |
+|-----------|--------|-----------|
+| PostgreSQL | `pg_dump` + WAL archiving | Daily full; continuous WAL (recommended) |
+| Redis | RDB snapshot (`BGSAVE`) | Hourly if non-transient keys matter |
+| Object storage (MinIO/S3) | Replication or `mc mirror` | Continuous or daily |
+| Traefik ACME | Copy `acme.json` | After each cert issue |
+| `.env.prod` / secrets | Encrypted off-site vault | On change |
+| WHMCS mappings | DB export of `mod_hexagamecloud_*` | Weekly |
+
+## PostgreSQL backup
+
+### Logical dump (single host)
+
+```bash
+docker exec hexahost-gamecloud-prod-postgres-1 \
+  pg_dump -U gamecloud -Fc gamecloud > "gamecloud-$(date +%F).dump"
+```
+
+Store dumps encrypted off-site. Retain at least 30 daily backups for production.
+
+### Restore from dump
+
+1. Stop API and worker to prevent writes:
+
+   ```bash
+   docker compose -f deploy/compose/compose.prod.yml stop api worker
+   ```
+
+2. Drop and recreate database (destructive):
+
+   ```bash
+   docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     psql -U gamecloud -c "DROP DATABASE gamecloud WITH (FORCE);"
+   docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     psql -U gamecloud -c "CREATE DATABASE gamecloud OWNER gamecloud;"
+   ```
+
+3. Restore:
+
+   ```bash
+   cat gamecloud-2026-07-01.dump | docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     pg_restore -U gamecloud -d gamecloud --no-owner --role=gamecloud
+   ```
+
+4. Start services and verify:
+
+   ```bash
+   docker compose -f deploy/compose/compose.prod.yml start api worker
+   curl -s https://api.example.net/api/v1/health/live
+   ```
+
+### Point-in-time recovery (PITR)
+
+For managed PostgreSQL or self-hosted WAL archiving, enable `archive_mode` and restore to a timestamp before the incident. Document your WAL retention in [disaster-recovery](disaster-recovery.md).
+
+## Redis backup
+
+Redis holds job queues and ephemeral cache. For queue durability during maintenance:
+
+```bash
+docker exec hexahost-gamecloud-prod-redis-1 redis-cli BGSAVE
+docker cp hexahost-gamecloud-prod-redis-1:/data/dump.rdb ./redis-$(date +%F).rdb
+```
+
+Restore by stopping Redis, replacing `dump.rdb`, and restarting. Expect in-flight jobs to be reprocessed or dead-lettered.
+
+## Object storage backup
+
+```bash
+mc alias set prod http://minio:9000 "$S3_ACCESS_KEY" "$S3_SECRET_KEY"
+mc mirror prod/gamecloud /backup/gamecloud-$(date +%F)/
+```
+
+Verify checksums on a sample of backup objects (`backups/*/manifest.json`).
+
+## Game server backups (tenant data)
+
+Customer-initiated backups are stored under `s3://gamecloud/backups/{serverId}/`. Restore through the panel or admin API — not via PostgreSQL restore alone.
+
+## Pre-restore checklist
+
+- [ ] Incident ticket with restore target time documented
+- [ ] WHMCS placed in maintenance mode if billing sync is active
+- [ ] Node agents will reconnect automatically after API restore
+- [ ] DNS records remain valid (no restore needed for RFC2136 provider state in DB)
+
+## Verification after restore
+
+1. Admin login and random server detail page
+2. WHMCS addon dashboard health (`Addons → HexaHost GameCloud`)
+3. Worker queue processing (`failed` jobs near zero)
+4. Sample backup download from object storage
+
+## Automation
+
+Integrate dumps with your existing backup tool (restic, Borg, Velero). Ansible playbooks in `deploy/ansible/` provision hosts; schedule backups via cron or systemd timers on `cp-db` or the compose host.
+
+## Related
+
+- [Disaster recovery](disaster-recovery.md)
+- [Object storage](object-storage.md)
+- [Data retention](../security/data-retention.md)
--- a/docs/operations/disaster-recovery.md
+++ b/docs/operations/disaster-recovery.md
@@ -0,0 +1,80 @@
+# Disaster recovery
+
+Recovery objectives and procedures when a HexaHost GameCloud control plane or game node is lost.
+
+## Recovery objectives (default targets)
+
+| Tier | RPO | RTO | Notes |
+|------|-----|-----|-------|
+| Control plane (API, DB) | 1 h | 4 h | With WAL + hourly Redis snapshot |
+| Object storage (backups) | 15 min | 4 h | With cross-region replication |
+| Single game node | 0 (running servers) | 30 min | Reschedule + DNS unchanged if edge IP stable |
+| Full region loss | 24 h | 24 h | Requires warm standby or restore from off-site |
+
+Adjust these in your internal SLA and document approved deviations.
+
+## Failure scenarios
+
+### Control plane host lost
+
+1. Provision replacement via `deploy/ansible/control-plane.yml`
+2. Restore PostgreSQL from latest dump + WAL ([backup-restore](backup-restore.md))
+3. Restore MinIO data or repoint `S3_*` to replica bucket
+4. Deploy compose: `deploy/compose/compose.prod.yml`
+5. Restore Traefik `acme.json` or re-issue certificates
+6. Verify WHMCS integration (HMAC secret unchanged)
+
+### Database corruption
+
+1. Stop writes (`api`, `worker`)
+2. Restore to last known-good PITR timestamp
+3. Run WHMCS reconciliation dry-run from addon module
+4. Review audit log gap for the corruption window
+
+### Object storage unavailable
+
+- API continues serving metadata; backup restore and file uploads fail
+- Fail over to replica endpoint or restore MinIO volume from snapshot
+- Workers retry S3 operations with exponential backoff
+
+### All game nodes offline
+
+- Customers cannot start servers; running containers on nodes may continue until stopped
+- Edge gateway returns connection errors for join-to-start
+- Restore nodes in priority order; scheduler assigns new provisions when nodes return
+
+### WHMCS host lost
+
+GameCloud remains authoritative for technical state. Reinstall WHMCS module from `integrations/whmcs/`, restore WHMCS database, re-import product mappings from backup export.
+
+## Standby strategy (recommended)
+
+| Component | Standby approach |
+|-----------|------------------|
+| PostgreSQL | Streaming replica or managed HA |
+| Redis | Sentinel or managed Redis |
+| S3 | Cross-region replication |
+| Control plane VMs | IaC templates in `deploy/ansible/` |
+| Traefik certs | Shared `acme.json` on persistent volume |
+
+## Communication
+
+During DR:
+
+1. Status page update (if applicable)
+2. Internal war room with roles: coordinator, DB, network, WHMCS
+3. Customer comms after RTO estimate is known — WHMCS can send mass email
+
+## DR drill checklist (quarterly)
+
+- [ ] Restore PostgreSQL dump to staging environment
+- [ ] Restore sample backup from object storage to test server
+- [ ] Fail over DNS to standby edge IP (dry run)
+- [ ] WHMCS provisioning test order on staging
+- [ ] Document actual RPO/RTO vs targets
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Server migration](server-migration.md)
+- [Incident response](incident-response.md)
--- a/docs/operations/incident-response.md
+++ b/docs/operations/incident-response.md
@@ -0,0 +1,84 @@
+# Incident response
+
+Operational playbook for security and availability incidents affecting HexaHost GameCloud.
+
+## Severity levels
+
+| Level | Examples | Response time | Examples |
+|-------|----------|---------------|----------|
+| SEV-1 | API down, data breach suspected, all nodes offline | 15 min | Full platform outage |
+| SEV-2 | Single node down, WHMCS provision failures, DNS sync broken | 1 h | Partial customer impact |
+| SEV-3 | Elevatoration, non-critical bug, one customer server stuck | Next business day | Limited blast radius |
+
+## Initial response (all severities)
+
+1. **Acknowledge** — Assign incident commander and scribe
+2. **Triage** — Customer-facing vs internal-only impact
+3. **Communicate** — Internal channel + status page if SEV-1/2
+4. **Preserve evidence** — Do not reboot hosts before log capture if security-related
+
+## Common incidents
+
+### API unavailable (502/503)
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml ps
+docker compose -f deploy/compose/compose.prod.yml logs --tail=200 api
+curl -v https://api.example.net/api/v1/health/live
+```
+
+Check Traefik routing ([traefik](traefik.md)), PostgreSQL connectivity, disk full.
+
+### Worker queue backlog
+
+Inspect Redis queue depth and failed jobs in logs. Scale worker container or restart after fixing root cause (DNS provider down, S3 errors).
+
+### Node heartbeat stale
+
+Nodes not seen > 90s trigger alerts. Verify node-agent systemd status, outbound firewall to `wss://api.example.net/api/v1/nodes/ws`, certificate expiry.
+
+### Suspected credential leak
+
+1. Rotate affected secrets immediately ([secrets](../security/secrets.md)):
+   - `SESSION_SECRET` (invalidates all sessions)
+   - `WHMCS_API_SECRET` per installation
+   - `NODE_TOKEN` for compromised node
+   - `S3_ACCESS_KEY` / database password
+2. Review audit logs and WHMCS module call logs
+3. Enable mTLS if not already active for WHMCS integration
+
+### Tenant isolation concern (SEV-1)
+
+1. Disable affected API endpoints if exploit is active
+2. Capture request IDs from audit log
+3. Engage security lead; reference [threat model](../security/threat-model.md) T21
+
+## WHMCS-specific
+
+Provisioning failures often appear in WHMCS **Utilities → Logs → Module Log**. Cross-check GameCloud API logs with `X-HGC-Integration-Id` header.
+
+Run reconciliation dry-run before live fixes: [reconciliation](../integrations/whmcs/reconciliation.md).
+
+## Post-incident
+
+Within 5 business days:
+
+- [ ] Timeline document (UTC)
+- [ ] Root cause (5 whys)
+- [ ] Action items with owners
+- [ ] Update runbooks if gaps found
+
+## Contacts and escalation
+
+Document internally:
+
+- On-call rotation
+- WHMCS admin contact
+- DNS provider support
+- Hosting provider (game nodes)
+
+## Related
+
+- [Disaster recovery](disaster-recovery.md)
+- [Threat model](../security/threat-model.md)
+- [WHMCS troubleshooting](../integrations/whmcs/troubleshooting.md)
--- a/docs/operations/node-drain.md
+++ b/docs/operations/node-drain.md
@@ -0,0 +1,101 @@
+# Game node drain
+
+Gracefully remove a game node from scheduling before maintenance, decommissioning, or migration.
+
+## When to drain
+
+- OS kernel / Docker upgrade on the node
+- Hardware maintenance or rack move
+- Shrinking cluster capacity
+- Replacing a node with new hardware ([server migration](server-migration.md))
+
+Do **not** drain the last node in a region if active servers have no migration target.
+
+## Procedure
+
+### 1. Set node to DRAINING
+
+Via admin API:
+
+```http
+PATCH /api/v1/admin/nodes/{nodeId}
+Authorization: Bearer <admin-token>
+Content-Type: application/json
+
+{ "status": "DRAINING" }
+```
+
+The scheduler stops placing new servers on this node. Existing running servers continue until stopped by users or idle shutdown.
+
+### 2. Wait for natural shutdown (optional)
+
+If maintenance window allows, wait for idle shutdown policy to stop inactive servers. Monitor:
+
+```sql
+SELECT id, status, "nodeId" FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status = 'RUNNING';
+```
+
+### 3. Migrate or stop remaining servers
+
+For each running server:
+
+- **Preferred:** Customer stops server via panel; reprovision on another node on next start (scheduler picks eligible node)
+- **Admin:** Force stop via admin API, then start — triggers reschedule if original node is DRAINING
+- **Maintenance:** Communicate window to affected customers
+
+### 4. Confirm empty node
+
+```sql
+SELECT COUNT(*) FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status IN ('RUNNING', 'STARTING');
+```
+
+Expected: `0` before maintenance on Docker host.
+
+### 5. Stop node-agent
+
+```bash
+sudo systemctl stop hgc-node-agent
+```
+
+Configured in `deploy/systemd/node-agent.service`.
+
+### 6. Perform maintenance
+
+Apply updates via `deploy/ansible/game-node.yml` or manual steps. See [game node operations](game-node.md).
+
+### 7. Return to service or decommission
+
+**Return:**
+
+```http
+PATCH /api/v1/admin/nodes/{nodeId}
+{ "status": "ONLINE" }
+```
+
+```bash
+sudo systemctl start hgc-node-agent
+```
+
+**Decommission:**
+
+1. Revoke enrollment token in control plane
+2. Remove `GameNode` row or mark `OFFLINE` permanently
+3. Wipe `/var/lib/hgc-node` data if disks are reused
+
+## Edge cases
+
+| Situation | Action |
+|-----------|--------|
+| Server stuck STARTING during drain | Admin stop + investigate node-agent logs |
+| Port range exhausted on other nodes | Add node or expand range before drain |
+| Node loses connectivity mid-drain | Servers marked UNKNOWN; reconcile when agent returns |
+
+## Automation
+
+Future: automated drain API with `--wait-empty` timeout. Until then, use SQL + admin API as above.
+
+## Related
+
+- [Game node operations](game-node.md)
+- [Server migration](server-migration.md)
+- [Upgrades](upgrades.md)
--- a/docs/operations/object-storage.md
+++ b/docs/operations/object-storage.md
@@ -0,0 +1,90 @@
+# Object storage operations
+
+HexaHost GameCloud stores backups, world exports, mod archives, and other large blobs in S3-compatible object storage. Development uses MinIO via `deploy/compose/compose.dev.yml`; production uses MinIO or an external S3 provider.
+
+## Environment variables
+
+| Variable | Description |
+|----------|-------------|
+| `S3_ENDPOINT` | Internal URL, e.g. `http://minio:9000` in compose |
+| `S3_REGION` | Region identifier (`us-east-1` for MinIO) |
+| `S3_BUCKET` | Primary bucket (default `gamecloud`) |
+| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | Credentials — rotate regularly |
+| `S3_FORCE_PATH_STYLE` | `true` for MinIO; `false` for AWS S3 |
+
+## Production (MinIO in compose)
+
+The production stack in `deploy/compose/compose.prod.yml` runs MinIO on the internal network only — no public ports. API and worker reach it at `http://minio:9000`.
+
+Initial bucket creation is handled by `minio-init` on first deploy:
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up minio-init
+```
+
+### Admin console access
+
+Do not expose the MinIO console to the public internet. Options:
+
+- SSH tunnel: `ssh -L 9001:127.0.0.1:9001 cp-01`
+- VPN-only route to the control plane management network
+- Separate MinIO deployment with IAM and audit logging
+
+## External S3 (AWS, Wasabi, Hetzner Object Storage)
+
+Point `S3_ENDPOINT` at the provider URL and set `S3_FORCE_PATH_STYLE=false` where appropriate. Create the bucket manually and apply a bucket policy that:
+
+- Denies public read/write
+- Allows only the GameCloud service principal (access key)
+- Enables versioning for backup objects
+- Optionally enforces SSE-S3 or SSE-KMS
+
+Remove the `minio` and `minio-init` services from compose when using external storage.
+
+## Object layout
+
+| Prefix | Content | Retention |
+|--------|---------|-----------|
+| `backups/{serverId}/` | Scheduled server backups | Per plan + [data retention policy](../security/data-retention.md) |
+| `worlds/{serverId}/` | World export ZIPs | 7 days after download link expiry |
+| `uploads/{serverId}/` | User file uploads (staging) | Cleaned after successful install |
+| `catalog/` | Cached mod/plugin artifacts | Managed by catalog sync jobs |
+
+## Health and monitoring
+
+MinIO health endpoint: `GET /minio/health/live`
+
+Monitor:
+
+- Bucket size growth (per tenant and total)
+- Failed PUT/GET rates from API logs
+- Disk usage on MinIO volume `hexahost-gamecloud-minio`
+
+## Backup of the storage layer
+
+For self-hosted MinIO:
+
+1. Enable bucket versioning
+2. Replicate to a second bucket or provider (MinIO site replication or `mc mirror`)
+3. Include object storage in [backup-restore](backup-restore.md) runbooks
+
+For external S3, rely on provider replication and lifecycle rules; document RPO/RTO in [disaster-recovery](disaster-recovery.md).
+
+## Troubleshooting
+
+### `NoSuchBucket`
+
+Run `minio-init` or create the bucket manually with the configured `S3_BUCKET` name.
+
+### Signature errors
+
+Verify clock sync (NTP) on API/worker hosts. Check `S3_FORCE_PATH_STYLE` matches the provider.
+
+### Slow backup uploads
+
+Check network between game nodes (backup source) and storage. Large backups stream from node-agent → API → S3; consider dedicated storage nodes for high-volume deployments.
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Secrets management](../security/secrets.md)
--- a/docs/operations/server-migration.md
+++ b/docs/operations/server-migration.md
@@ -0,0 +1,79 @@
+# Server and host migration
+
+Moving HexaHost GameCloud workloads between hosts: control plane relocation, game node replacement, and WHMCS host migration.
+
+## Control plane migration
+
+Move from `cp-01-old` to `cp-01-new`:
+
+1. **Prepare new host** — `ansible-playbook -i inventories/production deploy/ansible/control-plane.yml`
+2. **Stop writes on old host** — `docker compose stop api worker`
+3. **Backup** — PostgreSQL dump, Redis RDB, MinIO mirror, `/opt/hexahost-gamecloud/.env.prod`, Traefik `acme.json`
+4. **Transfer** — Secure copy to new host (rsync over SSH, encrypted archive)
+5. **Restore** — Follow [backup-restore](backup-restore.md) on new host
+6. **Update DNS** — `panel.example.net` and `api.example.net` A/AAAA records to new IP
+7. **Start stack** — `docker compose -f deploy/compose/compose.prod.yml up -d`
+8. **Verify** — Health checks, WHMCS addon dashboard, node WebSocket reconnections
+9. **Decommission old host** — Secure wipe after 72h parallel run (optional)
+
+WHMCS requires no URL change if hostnames are unchanged. Update `GameCloud API URL` in addon settings only if `api.example.net` changes.
+
+## Game node migration
+
+Replace `node-01` with `node-02` while preserving customer servers:
+
+### Add replacement node first
+
+1. Create `GameNode` in admin API
+2. Deploy agent: `deploy/ansible/game-node.yml`
+3. Confirm `ONLINE` status
+
+### Drain old node
+
+Follow [node-drain](node-drain.md). Running servers must stop and start on the new node.
+
+### Data considerations
+
+- World data lives in Docker volumes on the node — **not** automatically migrated
+- Customers should create a backup before migration window
+- Admin can trigger backup via API, then restore after server starts on new node
+
+### DNS
+
+Join hostnames point to the **edge gateway**, not individual nodes. No DNS change required for node migration unless edge IP changes.
+
+## Edge gateway migration
+
+If `EDGE_PUBLIC_HOST` changes:
+
+1. Update `RFC2136` records or manual DNS for `*.play.example.net`
+2. Deploy edge gateway on new IP
+3. Worker DNS sync reconciles `server_dns_records` within 60s
+
+## WHMCS host migration
+
+1. Export WHMCS database and `mod_hexagamecloud_*` tables
+2. Copy `integrations/whmcs/modules/` to new WHMCS install
+3. Reconfigure addon: Integration ID, API Secret (unchanged on GameCloud side)
+4. Run reconciliation dry-run: **Addons → HexaHost GameCloud → Reconciliation**
+5. Re-register mTLS fingerprint if client certificate changes
+
+See [WHMCS installation](../integrations/whmcs/installation.md).
+
+## MinIO / S3 migration
+
+```bash
+mc mirror old/gamecloud new/gamecloud
+```
+
+Update `S3_ENDPOINT` and credentials in `.env.prod`, restart API and worker.
+
+## Rollback
+
+Keep old host powered but isolated for 72 hours. Rollback = revert DNS + restart old compose stack from pre-migration backup.
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Disaster recovery](disaster-recovery.md)
+- [Node drain](node-drain.md)
--- a/docs/operations/traefik.md
+++ b/docs/operations/traefik.md
@@ -0,0 +1,96 @@
+# Traefik operations
+
+HexaHost GameCloud exposes the customer panel (`web`) and REST API (`api`) through Traefik on the control plane host. Game nodes and the edge gateway are **not** routed through this stack.
+
+## Layout
+
+| Path | Purpose |
+|------|---------|
+| `deploy/traefik/traefik.yml` | Static config — entry points, ACME, providers |
+| `deploy/traefik/dynamic/gamecloud.yml` | Middleware, TLS options, optional file routers |
+| `deploy/compose/compose.prod.yml` | Docker labels for primary routing |
+
+## Prerequisites
+
+```bash
+docker network create traefik-network
+```
+
+Set in `.env.prod`:
+
+```env
+TRAEFIK_NETWORK=traefik-network
+WEB_HOST=panel.example.net
+API_HOST=api.example.net
+ACME_EMAIL=ops@example.net
+APP_URL=https://panel.example.net
+API_URL=https://api.example.net
+TRUSTED_PROXY_COUNT=1
+```
+
+## Run Traefik
+
+Example standalone container (adjust paths):
+
+```bash
+docker run -d \
+  --name traefik \
+  --restart always \
+  -p 80:80 -p 443:443 \
+  -v /var/run/docker.sock:/var/run/docker.sock:ro \
+  -v /opt/hexahost-gamecloud/deploy/traefik/traefik.yml:/etc/traefik/traefik.yml:ro \
+  -v /opt/hexahost-gamecloud/deploy/traefik/dynamic:/etc/traefik/dynamic:ro \
+  -v /opt/hexahost-gamecloud/letsencrypt:/letsencrypt \
+  --network traefik-network \
+  traefik:v3.2
+```
+
+Then start the application stack:
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d
+```
+
+## Routing
+
+| Host | Service | Health check |
+|------|---------|--------------|
+| `panel.example.net` | `web:3000` | `GET /de` |
+| `api.example.net` | `api:3001` | `GET /api/v1/health/live` |
+
+HTTP on port 80 redirects to HTTPS. API requests receive rate limiting (`100 req/s` average, burst `200`).
+
+## WHMCS integration mTLS
+
+WHMCS integration traffic uses the same `api.example.net` host. When `INTEGRATION_MTLS_ENABLED=true`, terminate client certificates at Traefik or nginx and forward `X-HGC-Client-Cert-Fingerprint`. See [WHMCS mTLS](../integrations/whmcs/mtls.md) and `deploy/nginx/integration-mtls.conf.example`.
+
+## Certificate renewal
+
+Let's Encrypt certificates are stored in `/letsencrypt/acme.json`. Back up this file before host migrations. Traefik renews automatically; monitor expiry via the Traefik dashboard or Prometheus metrics.
+
+## Troubleshooting
+
+### 502 Bad Gateway
+
+1. Confirm `api` / `web` containers are healthy: `docker compose ps`
+2. Verify both services attach to `traefik-network`
+3. Check label `traefik.docker.network` matches the external network name
+
+### Certificate errors
+
+1. Ensure ports 80 and 443 are reachable from the internet for HTTP-01 challenge
+2. Confirm `ACME_EMAIL` is valid
+3. Inspect Traefik logs: `docker logs traefik 2>&1 | jq .`
+
+### Wrong client IP in audit logs
+
+Increase `TRUSTED_PROXY_COUNT` on the API to match the number of reverse-proxy hops.
+
+## Ansible
+
+Control plane provisioning is described in `deploy/ansible/control-plane.yml`. After OS setup, copy Traefik and compose files to `/opt/hexahost-gamecloud` and enable systemd or compose restart policies.
+
+## Related
+
+- [Control plane operations](control-plane.md)
+- [Installation (development)](installation.md)
--- a/docs/operations/upgrades.md
+++ b/docs/operations/upgrades.md
@@ -0,0 +1,86 @@
+# Control plane upgrades
+
+Rolling upgrades for HexaHost GameCloud API, worker, web, and infrastructure containers on the control plane.
+
+## Version pinning
+
+Set in `.env.prod`:
+
+```env
+GAMECLOUD_IMAGE_TAG=1.2.0
+```
+
+Build and tag images in CI, or pull from your registry. Never deploy `:latest` in production without a rollback plan.
+
+## Pre-upgrade checklist
+
+- [ ] Read release notes and migration requirements (`docs/IMPLEMENTATION_STATUS.md`)
+- [ ] Fresh PostgreSQL backup ([backup-restore](backup-restore.md))
+- [ ] WHMCS addon/server module compatibility verified
+- [ ] Maintenance window communicated if API downtime expected
+- [ ] Staging deploy completed successfully
+
+## Standard rolling upgrade
+
+```bash
+cd /opt/hexahost-gamecloud
+
+# Pull/build new images
+export GAMECLOUD_IMAGE_TAG=1.2.0
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod pull
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod build
+
+# Run database migrations (if applicable — via API container one-shot)
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod run --rm api \
+  pnpm --filter @hexahost/api prisma migrate deploy
+
+# Recreate app containers one at a time
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d --no-deps worker
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d --no-deps api
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d --no-deps web
+```
+
+## Infrastructure upgrades
+
+| Service | Notes |
+|---------|-------|
+| PostgreSQL | Major version requires `pg_upgrade` or dump/restore — plan maintenance window |
+| Redis | Minor upgrades usually safe; snapshot before upgrade |
+| MinIO | Follow MinIO release notes; backup volume first |
+| Traefik | Test dynamic config with `traefik validate` equivalent (dry-run container) |
+
+## Zero-downtime considerations
+
+- Run at least two API instances behind Traefik for true zero-downtime (single-node compose has brief API restart)
+- Worker: only one active consumer per-correct for some jobs; use graceful shutdown (SIGTERM) before replace
+- Web: Next.js standalone reload causes short 502; upgrade during low traffic
+
+## Rollback
+
+```bash
+export GAMECLOUD_IMAGE_TAG=1.1.0
+docker compose -f deploy/compose/compose.prod.yml --env-file .env.prod up -d api worker web
+```
+
+If migrations ran forward-only, restore PostgreSQL from pre-upgrade dump.
+
+## WHMCS module upgrades
+
+See [WHMCS upgrades](../integrations/whmcs/upgrades.md). Upgrade GameCloud API before WHMCS server module when release notes specify API-first ordering.
+
+## Post-upgrade verification
+
+```bash
+curl -sf https://api.example.net/api/v1/health/live
+curl -sf https://panel.example.net/de
+```
+
+- Admin panel login
+- WHMCS test suspend/unsuspend on staging service
+- Worker queue depth normal
+- Node heartbeats within 30s
+
+## Related
+
+- [Control plane operations](control-plane.md)
+- [Node drain](node-drain.md) — upgrade game nodes separately