81 lines
2.9 KiB
Markdown
81 lines
2.9 KiB
Markdown
# Disaster recovery
|
|
|
|
Recovery objectives and procedures when a HexaHost GameCloud control plane or game node is lost.
|
|
|
|
## Recovery objectives (default targets)
|
|
|
|
| Tier | RPO | RTO | Notes |
|
|
|------|-----|-----|-------|
|
|
| Control plane (API, DB) | 1 h | 4 h | With WAL + hourly Redis snapshot |
|
|
| Object storage (backups) | 15 min | 4 h | With cross-region replication |
|
|
| Single game node | 0 (running servers) | 30 min | Reschedule + DNS unchanged if edge IP stable |
|
|
| Full region loss | 24 h | 24 h | Requires warm standby or restore from off-site |
|
|
|
|
Adjust these in your internal SLA and document approved deviations.
|
|
|
|
## Failure scenarios
|
|
|
|
### Control plane host lost
|
|
|
|
1. Provision replacement via `deploy/ansible/control-plane.yml`
|
|
2. Restore PostgreSQL from latest dump + WAL ([backup-restore](backup-restore.md))
|
|
3. Restore MinIO data or repoint `S3_*` to replica bucket
|
|
4. Deploy compose: `deploy/compose/compose.prod.yml`
|
|
5. Restore Traefik `acme.json` or re-issue certificates
|
|
6. Verify WHMCS integration (HMAC secret unchanged)
|
|
|
|
### Database corruption
|
|
|
|
1. Stop writes (`api`, `worker`)
|
|
2. Restore to last known-good PITR timestamp
|
|
3. Run WHMCS reconciliation dry-run from addon module
|
|
4. Review audit log gap for the corruption window
|
|
|
|
### Object storage unavailable
|
|
|
|
- API continues serving metadata; backup restore and file uploads fail
|
|
- Fail over to replica endpoint or restore MinIO volume from snapshot
|
|
- Workers retry S3 operations with exponential backoff
|
|
|
|
### All game nodes offline
|
|
|
|
- Customers cannot start servers; running containers on nodes may continue until stopped
|
|
- Edge gateway returns connection errors for join-to-start
|
|
- Restore nodes in priority order; scheduler assigns new provisions when nodes return
|
|
|
|
### WHMCS host lost
|
|
|
|
GameCloud remains authoritative for technical state. Reinstall WHMCS module from `integrations/whmcs/`, restore WHMCS database, re-import product mappings from backup export.
|
|
|
|
## Standby strategy (recommended)
|
|
|
|
| Component | Standby approach |
|
|
|-----------|------------------|
|
|
| PostgreSQL | Streaming replica or managed HA |
|
|
| Redis | Sentinel or managed Redis |
|
|
| S3 | Cross-region replication |
|
|
| Control plane VMs | IaC templates in `deploy/ansible/` |
|
|
| Traefik certs | Shared `acme.json` on persistent volume |
|
|
|
|
## Communication
|
|
|
|
During DR:
|
|
|
|
1. Status page update (if applicable)
|
|
2. Internal war room with roles: coordinator, DB, network, WHMCS
|
|
3. Customer comms after RTO estimate is known — WHMCS can send mass email
|
|
|
|
## DR drill checklist (quarterly)
|
|
|
|
- [ ] Restore PostgreSQL dump to staging environment
|
|
- [ ] Restore sample backup from object storage to test server
|
|
- [ ] Fail over DNS to standby edge IP (dry run)
|
|
- [ ] WHMCS provisioning test order on staging
|
|
- [ ] Document actual RPO/RTO vs targets
|
|
|
|
## Related
|
|
|
|
- [Backup and restore](backup-restore.md)
|
|
- [Server migration](server-migration.md)
|
|
- [Incident response](incident-response.md)
|