Enhance API with OIDC support, including login and callback endpoints. Update environment variables for OIDC configuration in .env.example. Add new features to the catalog service for listing software families, Minecraft versions, and deployment regions. Implement server management actions such as kill, delete, and update in the servers module. Integrate feature flags for maintenance mode in server operations. Update pnpm-lock.yaml with new dependencies and versions.

docs/operations/disaster-recovery.md

@@ -0,0 +1,80 @@
+# Disaster recovery
+
+Recovery objectives and procedures when a HexaHost GameCloud control plane or game node is lost.
+
+## Recovery objectives (default targets)
+
+| Tier | RPO | RTO | Notes |
+|------|-----|-----|-------|
+| Control plane (API, DB) | 1 h | 4 h | With WAL + hourly Redis snapshot |
+| Object storage (backups) | 15 min | 4 h | With cross-region replication |
+| Single game node | 0 (running servers) | 30 min | Reschedule + DNS unchanged if edge IP stable |
+| Full region loss | 24 h | 24 h | Requires warm standby or restore from off-site |
+
+Adjust these in your internal SLA and document approved deviations.
+
+## Failure scenarios
+
+### Control plane host lost
+
+1. Provision replacement via `deploy/ansible/control-plane.yml`
+2. Restore PostgreSQL from latest dump + WAL ([backup-restore](backup-restore.md))
+3. Restore MinIO data or repoint `S3_*` to replica bucket
+4. Deploy compose: `deploy/compose/compose.prod.yml`
+5. Restore Traefik `acme.json` or re-issue certificates
+6. Verify WHMCS integration (HMAC secret unchanged)
+
+### Database corruption
+
+1. Stop writes (`api`, `worker`)
+2. Restore to last known-good PITR timestamp
+3. Run WHMCS reconciliation dry-run from addon module
+4. Review audit log gap for the corruption window
+
+### Object storage unavailable
+
+- API continues serving metadata; backup restore and file uploads fail
+- Fail over to replica endpoint or restore MinIO volume from snapshot
+- Workers retry S3 operations with exponential backoff
+
+### All game nodes offline
+
+- Customers cannot start servers; running containers on nodes may continue until stopped
+- Edge gateway returns connection errors for join-to-start
+- Restore nodes in priority order; scheduler assigns new provisions when nodes return
+
+### WHMCS host lost
+
+GameCloud remains authoritative for technical state. Reinstall WHMCS module from `integrations/whmcs/`, restore WHMCS database, re-import product mappings from backup export.
+
+## Standby strategy (recommended)
+
+| Component | Standby approach |
+|-----------|------------------|
+| PostgreSQL | Streaming replica or managed HA |
+| Redis | Sentinel or managed Redis |
+| S3 | Cross-region replication |
+| Control plane VMs | IaC templates in `deploy/ansible/` |
+| Traefik certs | Shared `acme.json` on persistent volume |
+
+## Communication
+
+During DR:
+
+1. Status page update (if applicable)
+2. Internal war room with roles: coordinator, DB, network, WHMCS
+3. Customer comms after RTO estimate is known — WHMCS can send mass email
+
+## DR drill checklist (quarterly)
+
+- [ ] Restore PostgreSQL dump to staging environment
+- [ ] Restore sample backup from object storage to test server
+- [ ] Fail over DNS to standby edge IP (dry run)
+- [ ] WHMCS provisioning test order on staging
+- [ ] Document actual RPO/RTO vs targets
+
+## Related
+
+- [Backup and restore](backup-restore.md)
+- [Server migration](server-migration.md)
+- [Incident response](incident-response.md)