Enhance API with OIDC support, including login and callback endpoints. Update environment variables for OIDC configuration in .env.example. Add new features to the catalog service for listing software families, Minecraft versions, and deployment regions. Implement server management actions such as kill, delete, and update in the servers module. Integrate feature flags for maintenance mode in server operations. Update pnpm-lock.yaml with new dependencies and versions.

docs/operations/backup-restore.md

@@ -0,0 +1,108 @@
+# Backup and restore
+
+This runbook covers backing up and restoring HexaHost GameCloud control plane data. Game server world data is backed up separately via the in-product backup feature (stored in object storage).
+
+## Scope
+
+| Component | Method | Frequency |
+|-----------|--------|-----------|
+| PostgreSQL | `pg_dump` + WAL archiving | Daily full; continuous WAL (recommended) |
+| Redis | RDB snapshot (`BGSAVE`) | Hourly if non-transient keys matter |
+| Object storage (MinIO/S3) | Replication or `mc mirror` | Continuous or daily |
+| Traefik ACME | Copy `acme.json` | After each cert issue |
+| `.env.prod` / secrets | Encrypted off-site vault | On change |
+| WHMCS mappings | DB export of `mod_hexagamecloud_*` | Weekly |
+
+## PostgreSQL backup
+
+### Logical dump (single host)
+
+```bash
+docker exec hexahost-gamecloud-prod-postgres-1 \
+  pg_dump -U gamecloud -Fc gamecloud > "gamecloud-$(date +%F).dump"
+```
+
+Store dumps encrypted off-site. Retain at least 30 daily backups for production.
+
+### Restore from dump
+
+1. Stop API and worker to prevent writes:
+
+   ```bash
+   docker compose -f deploy/compose/compose.prod.yml stop api worker
+   ```
+
+2. Drop and recreate database (destructive):
+
+   ```bash
+   docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     psql -U gamecloud -c "DROP DATABASE gamecloud WITH (FORCE);"
+   docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     psql -U gamecloud -c "CREATE DATABASE gamecloud OWNER gamecloud;"
+   ```
+
+3. Restore:
+
+   ```bash
+   cat gamecloud-2026-07-01.dump | docker exec -i hexahost-gamecloud-prod-postgres-1 \
+     pg_restore -U gamecloud -d gamecloud --no-owner --role=gamecloud
+   ```
+
+4. Start services and verify:
+
+   ```bash
+   docker compose -f deploy/compose/compose.prod.yml start api worker
+   curl -s https://api.example.net/api/v1/health/live
+   ```
+
+### Point-in-time recovery (PITR)
+
+For managed PostgreSQL or self-hosted WAL archiving, enable `archive_mode` and restore to a timestamp before the incident. Document your WAL retention in [disaster-recovery](disaster-recovery.md).
+
+## Redis backup
+
+Redis holds job queues and ephemeral cache. For queue durability during maintenance:
+
+```bash
+docker exec hexahost-gamecloud-prod-redis-1 redis-cli BGSAVE
+docker cp hexahost-gamecloud-prod-redis-1:/data/dump.rdb ./redis-$(date +%F).rdb
+```
+
+Restore by stopping Redis, replacing `dump.rdb`, and restarting. Expect in-flight jobs to be reprocessed or dead-lettered.
+
+## Object storage backup
+
+```bash
+mc alias set prod http://minio:9000 "$S3_ACCESS_KEY" "$S3_SECRET_KEY"
+mc mirror prod/gamecloud /backup/gamecloud-$(date +%F)/
+```
+
+Verify checksums on a sample of backup objects (`backups/*/manifest.json`).
+
+## Game server backups (tenant data)
+
+Customer-initiated backups are stored under `s3://gamecloud/backups/{serverId}/`. Restore through the panel or admin API — not via PostgreSQL restore alone.
+
+## Pre-restore checklist
+
+- [ ] Incident ticket with restore target time documented
+- [ ] WHMCS placed in maintenance mode if billing sync is active
+- [ ] Node agents will reconnect automatically after API restore
+- [ ] DNS records remain valid (no restore needed for RFC2136 provider state in DB)
+
+## Verification after restore
+
+1. Admin login and random server detail page
+2. WHMCS addon dashboard health (`Addons → HexaHost GameCloud`)
+3. Worker queue processing (`failed` jobs near zero)
+4. Sample backup download from object storage
+
+## Automation
+
+Integrate dumps with your existing backup tool (restic, Borg, Velero). Ansible playbooks in `deploy/ansible/` provision hosts; schedule backups via cron or systemd timers on `cp-db` or the compose host.
+
+## Related
+
+- [Disaster recovery](disaster-recovery.md)
+- [Object storage](object-storage.md)
+- [Data retention](../security/data-retention.md)