109 lines
3.7 KiB
Markdown
109 lines
3.7 KiB
Markdown
# Backup and restore
|
|
|
|
This runbook covers backing up and restoring HexaHost GameCloud control plane data. Game server world data is backed up separately via the in-product backup feature (stored in object storage).
|
|
|
|
## Scope
|
|
|
|
| Component | Method | Frequency |
|
|
|-----------|--------|-----------|
|
|
| PostgreSQL | `pg_dump` + WAL archiving | Daily full; continuous WAL (recommended) |
|
|
| Redis | RDB snapshot (`BGSAVE`) | Hourly if non-transient keys matter |
|
|
| Object storage (MinIO/S3) | Replication or `mc mirror` | Continuous or daily |
|
|
| Traefik ACME | Copy `acme.json` | After each cert issue |
|
|
| `.env.prod` / secrets | Encrypted off-site vault | On change |
|
|
| WHMCS mappings | DB export of `mod_hexagamecloud_*` | Weekly |
|
|
|
|
## PostgreSQL backup
|
|
|
|
### Logical dump (single host)
|
|
|
|
```bash
|
|
docker exec hexahost-gamecloud-prod-postgres-1 \
|
|
pg_dump -U gamecloud -Fc gamecloud > "gamecloud-$(date +%F).dump"
|
|
```
|
|
|
|
Store dumps encrypted off-site. Retain at least 30 daily backups for production.
|
|
|
|
### Restore from dump
|
|
|
|
1. Stop API and worker to prevent writes:
|
|
|
|
```bash
|
|
docker compose -f deploy/compose/compose.prod.yml stop api worker
|
|
```
|
|
|
|
2. Drop and recreate database (destructive):
|
|
|
|
```bash
|
|
docker exec -i hexahost-gamecloud-prod-postgres-1 \
|
|
psql -U gamecloud -c "DROP DATABASE gamecloud WITH (FORCE);"
|
|
docker exec -i hexahost-gamecloud-prod-postgres-1 \
|
|
psql -U gamecloud -c "CREATE DATABASE gamecloud OWNER gamecloud;"
|
|
```
|
|
|
|
3. Restore:
|
|
|
|
```bash
|
|
cat gamecloud-2026-07-01.dump | docker exec -i hexahost-gamecloud-prod-postgres-1 \
|
|
pg_restore -U gamecloud -d gamecloud --no-owner --role=gamecloud
|
|
```
|
|
|
|
4. Start services and verify:
|
|
|
|
```bash
|
|
docker compose -f deploy/compose/compose.prod.yml start api worker
|
|
curl -s https://api.example.net/api/v1/health/live
|
|
```
|
|
|
|
### Point-in-time recovery (PITR)
|
|
|
|
For managed PostgreSQL or self-hosted WAL archiving, enable `archive_mode` and restore to a timestamp before the incident. Document your WAL retention in [disaster-recovery](disaster-recovery.md).
|
|
|
|
## Redis backup
|
|
|
|
Redis holds job queues and ephemeral cache. For queue durability during maintenance:
|
|
|
|
```bash
|
|
docker exec hexahost-gamecloud-prod-redis-1 redis-cli BGSAVE
|
|
docker cp hexahost-gamecloud-prod-redis-1:/data/dump.rdb ./redis-$(date +%F).rdb
|
|
```
|
|
|
|
Restore by stopping Redis, replacing `dump.rdb`, and restarting. Expect in-flight jobs to be reprocessed or dead-lettered.
|
|
|
|
## Object storage backup
|
|
|
|
```bash
|
|
mc alias set prod http://minio:9000 "$S3_ACCESS_KEY" "$S3_SECRET_KEY"
|
|
mc mirror prod/gamecloud /backup/gamecloud-$(date +%F)/
|
|
```
|
|
|
|
Verify checksums on a sample of backup objects (`backups/*/manifest.json`).
|
|
|
|
## Game server backups (tenant data)
|
|
|
|
Customer-initiated backups are stored under `s3://gamecloud/backups/{serverId}/`. Restore through the panel or admin API — not via PostgreSQL restore alone.
|
|
|
|
## Pre-restore checklist
|
|
|
|
- [ ] Incident ticket with restore target time documented
|
|
- [ ] WHMCS placed in maintenance mode if billing sync is active
|
|
- [ ] Node agents will reconnect automatically after API restore
|
|
- [ ] DNS records remain valid (no restore needed for RFC2136 provider state in DB)
|
|
|
|
## Verification after restore
|
|
|
|
1. Admin login and random server detail page
|
|
2. WHMCS addon dashboard health (`Addons → HexaHost GameCloud`)
|
|
3. Worker queue processing (`failed` jobs near zero)
|
|
4. Sample backup download from object storage
|
|
|
|
## Automation
|
|
|
|
Integrate dumps with your existing backup tool (restic, Borg, Velero). Ansible playbooks in `deploy/ansible/` provision hosts; schedule backups via cron or systemd timers on `cp-db` or the compose host.
|
|
|
|
## Related
|
|
|
|
- [Disaster recovery](disaster-recovery.md)
|
|
- [Object storage](object-storage.md)
|
|
- [Data retention](../security/data-retention.md)
|