Disaster recovery

Recovery objectives and procedures when a HexaHost GameCloud control plane or game node is lost.

Recovery objectives (default targets)

Tier	RPO	RTO	Notes
Control plane (API, DB)	1 h	4 h	With WAL + hourly Redis snapshot
Object storage (backups)	15 min	4 h	With cross-region replication
Single game node	0 (running servers)	30 min	Reschedule + DNS unchanged if edge IP stable
Full region loss	24 h	24 h	Requires warm standby or restore from off-site

Adjust these in your internal SLA and document approved deviations.

Failure scenarios

Control plane host lost

Provision replacement via deploy/ansible/control-plane.yml
Restore PostgreSQL from latest dump + WAL (backup-restore)
Restore MinIO data or repoint S3_* to replica bucket
Deploy compose: deploy/compose/compose.prod.yml
Restore Traefik acme.json or re-issue certificates
Verify WHMCS integration (HMAC secret unchanged)

Database corruption

Stop writes (api, worker)
Restore to last known-good PITR timestamp
Run WHMCS reconciliation dry-run from addon module
Review audit log gap for the corruption window

Object storage unavailable

API continues serving metadata; backup restore and file uploads fail
Fail over to replica endpoint or restore MinIO volume from snapshot
Workers retry S3 operations with exponential backoff

All game nodes offline

Customers cannot start servers; running containers on nodes may continue until stopped
Edge gateway returns connection errors for join-to-start
Restore nodes in priority order; scheduler assigns new provisions when nodes return

WHMCS host lost

GameCloud remains authoritative for technical state. Reinstall WHMCS module from integrations/whmcs/, restore WHMCS database, re-import product mappings from backup export.

Standby strategy (recommended)

Component	Standby approach
PostgreSQL	Streaming replica or managed HA
Redis	Sentinel or managed Redis
S3	Cross-region replication
Control plane VMs	IaC templates in `deploy/ansible/`
Traefik certs	Shared `acme.json` on persistent volume

Communication

During DR:

Status page update (if applicable)
Internal war room with roles: coordinator, DB, network, WHMCS
Customer comms after RTO estimate is known — WHMCS can send mass email

DR drill checklist (quarterly)

Restore PostgreSQL dump to staging environment
Restore sample backup from object storage to test server
Fail over DNS to standby edge IP (dry run)
WHMCS provisioning test order on staging
Document actual RPO/RTO vs targets

2.9 KiB Raw Blame History