Phase9

docs/operations/control-plane.md

@@ -0,0 +1,69 @@
+# Control plane operations
+
+The control plane hosts the customer-facing panel, REST API, background worker, PostgreSQL, Redis, object storage, and the edge gateway.
+
+## Recommended topology (production)
+
+| Host | Role |
+|------|------|
+| `cp-01` | API, Web, Worker, Edge Gateway |
+| `cp-db` | PostgreSQL (managed or dedicated) |
+| `cp-cache` | Redis |
+| `cp-storage` | S3-compatible object storage |
+| `node-01`, `node-02` | Game nodes with node-agent |
+
+Phase 9 acceptance target: **one control plane stack and two game nodes**.
+
+## Core services
+
+- **API** (`apps/api`) — public REST + internal integration endpoints
+- **Worker** (`apps/worker`) — queues, idle shutdown, metering, DNS sync
+- **Web** (`apps/web`) — customer panel
+- **Edge gateway** (`apps/edge-gateway`) — join-to-start TCP proxy
+
+## Billing modes
+
+| Mode | `BILLING_PROVIDER` | Commercial system of record |
+|------|--------------------|-----------------------------|
+| Internal credits | `internal` | GameCloud wallet |
+| WHMCS | `whmcs` | WHMCS invoices/subscriptions |
+
+In WHMCS mode, provisioning and lifecycle are driven by `/api/v1/integrations/whmcs/*` with HMAC authentication.
+
+## Secrets
+
+Never commit production secrets. Rotate:
+
+- `SESSION_SECRET`, `ENCRYPTION_KEY`
+- `EDGE_INTERNAL_API_KEY`
+- `WHMCS_API_SECRET` (per installation)
+- `RFC2136_KEY_SECRET`
+- Node enrollment tokens
+
+## Backups
+
+Back up at minimum:
+
+1. PostgreSQL (full daily + WAL)
+2. Redis persistence snapshot (if used for non-transient state)
+3. Object storage bucket replication
+4. `/etc` and deployment configuration on control plane hosts
+
+## Monitoring
+
+See `deploy/monitoring/prometheus.yml` for scrape targets. Alert on:
+
+- API health check failures
+- Worker queue depth / failed jobs
+- Node heartbeat staleness > 90s
+- DNS records in `FAILED` status
+
+## Deployment
+
+Use Ansible playbooks in `deploy/ansible/`:
+
+```bash
+ansible-playbook -i inventories/production site.yml
+```
+
+Import the production compose stack from `deploy/compose` after base OS provisioning.