Phase9

docs/operations/control-plane.md

@@ -0,0 +1,69 @@
+# Control plane operations
+
+The control plane hosts the customer-facing panel, REST API, background worker, PostgreSQL, Redis, object storage, and the edge gateway.
+
+## Recommended topology (production)
+
+| Host | Role |
+|------|------|
+| `cp-01` | API, Web, Worker, Edge Gateway |
+| `cp-db` | PostgreSQL (managed or dedicated) |
+| `cp-cache` | Redis |
+| `cp-storage` | S3-compatible object storage |
+| `node-01`, `node-02` | Game nodes with node-agent |
+
+Phase 9 acceptance target: **one control plane stack and two game nodes**.
+
+## Core services
+
+- **API** (`apps/api`) — public REST + internal integration endpoints
+- **Worker** (`apps/worker`) — queues, idle shutdown, metering, DNS sync
+- **Web** (`apps/web`) — customer panel
+- **Edge gateway** (`apps/edge-gateway`) — join-to-start TCP proxy
+
+## Billing modes
+
+| Mode | `BILLING_PROVIDER` | Commercial system of record |
+|------|--------------------|-----------------------------|
+| Internal credits | `internal` | GameCloud wallet |
+| WHMCS | `whmcs` | WHMCS invoices/subscriptions |
+
+In WHMCS mode, provisioning and lifecycle are driven by `/api/v1/integrations/whmcs/*` with HMAC authentication.
+
+## Secrets
+
+Never commit production secrets. Rotate:
+
+- `SESSION_SECRET`, `ENCRYPTION_KEY`
+- `EDGE_INTERNAL_API_KEY`
+- `WHMCS_API_SECRET` (per installation)
+- `RFC2136_KEY_SECRET`
+- Node enrollment tokens
+
+## Backups
+
+Back up at minimum:
+
+1. PostgreSQL (full daily + WAL)
+2. Redis persistence snapshot (if used for non-transient state)
+3. Object storage bucket replication
+4. `/etc` and deployment configuration on control plane hosts
+
+## Monitoring
+
+See `deploy/monitoring/prometheus.yml` for scrape targets. Alert on:
+
+- API health check failures
+- Worker queue depth / failed jobs
+- Node heartbeat staleness > 90s
+- DNS records in `FAILED` status
+
+## Deployment
+
+Use Ansible playbooks in `deploy/ansible/`:
+
+```bash
+ansible-playbook -i inventories/production site.yml
+```
+
+Import the production compose stack from `deploy/compose` after base OS provisioning.

docs/operations/dns.md

@@ -0,0 +1,48 @@
+# DNS operations
+
+HexaHost GameCloud stores desired DNS records in PostgreSQL (`server_dns_records`) and synchronizes them through a provider abstraction in `@hexahost/dns`.
+
+## Providers
+
+| Provider | `DNS_PROVIDER` | Use case |
+|----------|----------------|----------|
+| Local metadata | `local` (default) | Development — records exist only in the database |
+| RFC2136 dynamic updates | `rfc2136` | Production with BIND or compatible authoritative DNS |
+
+## Production (RFC2136)
+
+Set on API and worker hosts:
+
+```env
+DNS_PROVIDER=rfc2136
+GAME_BASE_DOMAIN=play.example.net
+EDGE_PUBLIC_HOST=203.0.113.10
+EDGE_LISTEN_PORT=25565
+RFC2136_SERVER=ns1.internal.example.net
+RFC2136_KEY_NAME=gamecloud-updater.
+RFC2136_KEY_SECRET=base64-encoded-secret
+RFC2136_TTL=300
+```
+
+Requirements:
+
+- `nsupdate` (bind9-utils) installed on API/worker hosts
+- TSIG key authorized for A and SRV updates on `*.play.example.net`
+- Edge gateway reachable at `EDGE_PUBLIC_HOST:EDGE_LISTEN_PORT`
+
+## Record layout
+
+For each Java server with join slug `my-server-abc123`:
+
+- `A` record: `my-server-abc123.play.example.net` → edge gateway IP
+- `SRV` record: `_minecraft._tcp.my-server-abc123.play.example.net` → edge hostname/port
+
+## Worker sync
+
+The worker runs a DNS sync tick every 60 seconds (`pushPendingDnsRecords`). Records start as `PENDING` when `DNS_PROVIDER=rfc2136` and move to `ACTIVE` or `FAILED`.
+
+## Troubleshooting
+
+1. Check `server_dns_records.status` and `lastError` in PostgreSQL.
+2. Run `nsupdate -y` manually with the generated script from logs.
+3. Verify TSIG key name ends with `.` if your nameserver expects FQDN form.

docs/operations/game-node.md

@@ -0,0 +1,63 @@
+# Game node operations
+
+Game nodes run Docker workloads for Minecraft servers and connect to the control plane through the node-agent.
+
+## Capacity
+
+Each node exposes:
+
+- `maxServers`, `maxRamMb`, `platformRamMb`
+- Host port pool: `portRangeStart`–`portRangeEnd` (default `25565`–`25664`)
+
+The scheduler places new servers on the least-loaded eligible node.
+
+## Enrollment
+
+1. Create a `GameNode` row in the control plane database.
+2. Generate an enrollment token and distribute it securely to the node host.
+3. Start `node-agent` with matching `NODE_ID` and `NODE_TOKEN`.
+4. Confirm node status becomes `ONLINE` in the admin API.
+
+## Port allocation
+
+Host ports are unique per node (`@@unique([nodeId, hostPort])`). Do not manually assign ports outside the configured range.
+
+## Draining
+
+Before maintenance:
+
+1. Set node status to `DRAINING` via admin API.
+2. Wait for running servers to stop or migrate.
+3. Perform maintenance.
+4. Return node to `ONLINE` or decommission.
+
+## Firewall
+
+Allow:
+
+- Outbound WebSocket to control plane API
+- Inbound TCP on allocated Minecraft host port range
+- Docker bridge traffic locally
+
+Deny:
+
+- Public access to Docker API
+- Unrestricted inbound on management ports
+
+## Ansible
+
+Bootstrap with:
+
+```bash
+ansible-playbook -i inventories/production deploy/ansible/game-node.yml
+```
+
+Copy the `hgc-node-agent` binary to `/usr/local/bin/hgc-node-agent` and provide TLS materials via `NODE_TLS_CERT` / `NODE_TLS_KEY` or CA enrollment.
+
+## Failure scenarios
+
+| Scenario | Action |
+|----------|--------|
+| Node offline | Servers marked `UNKNOWN`; reschedule starts when node returns |
+| Port exhaustion | Provisioning fails with `NO_NODE_AVAILABLE`; add node or expand port range |
+| Disk full | Stop new provisions; prune unused images; expand volume |