# Control plane operations

The control plane hosts the customer-facing panel, REST API, background worker, PostgreSQL, Redis, object storage, and the edge gateway.

## Recommended topology (production)

| Host | Role |
|------|------|
| `cp-01` | API, Web, Worker, Edge Gateway |
| `cp-db` | PostgreSQL (managed or dedicated) |
| `cp-cache` | Redis |
| `cp-storage` | S3-compatible object storage |
| `node-01`, `node-02` | Game nodes with node-agent |

Phase 9 acceptance target: **one control plane stack and two game nodes**.

## Core services

- **API** (`apps/api`) — public REST + internal integration endpoints
- **Worker** (`apps/worker`) — queues, idle shutdown, metering, DNS sync
- **Web** (`apps/web`) — customer panel
- **Edge gateway** (`apps/edge-gateway`) — join-to-start TCP proxy

## Billing modes

| Mode | `BILLING_PROVIDER` | Commercial system of record |
|------|--------------------|-----------------------------|
| Internal credits | `internal` | GameCloud wallet |
| WHMCS | `whmcs` | WHMCS invoices/subscriptions |

In WHMCS mode, provisioning and lifecycle are driven by `/api/v1/integrations/whmcs/*` with HMAC authentication.

## Secrets

Never commit production secrets. Rotate:

- `SESSION_SECRET`, `ENCRYPTION_KEY`
- `EDGE_INTERNAL_API_KEY`
- `WHMCS_API_SECRET` (per installation)
- `RFC2136_KEY_SECRET`
- Node enrollment tokens

## Backups

Back up at minimum:

1. PostgreSQL (full daily + WAL)
2. Redis persistence snapshot (if used for non-transient state)
3. Object storage bucket replication
4. `/etc` and deployment configuration on control plane hosts

## Monitoring

See `deploy/monitoring/prometheus.yml` for scrape targets. Alert on:

- API health check failures
- Worker queue depth / failed jobs
- Node heartbeat staleness > 90s
- DNS records in `FAILED` status

## Deployment

Use Ansible playbooks in `deploy/ansible/`:

```bash
ansible-playbook -i inventories/production site.yml
```

Import the production compose stack from `deploy/compose` after base OS provisioning.