Enhance API with OIDC support, including login and callback endpoints. Update environment variables for OIDC configuration in .env.example. Add new features to the catalog service for listing software families, Minecraft versions, and deployment regions. Implement server management actions such as kill, delete, and update in the servers module. Integrate feature flags for maintenance mode in server operations. Update pnpm-lock.yaml with new dependencies and versions.

docs/operations/incident-response.md

@@ -0,0 +1,84 @@
+# Incident response
+
+Operational playbook for security and availability incidents affecting HexaHost GameCloud.
+
+## Severity levels
+
+| Level | Examples | Response time | Examples |
+|-------|----------|---------------|----------|
+| SEV-1 | API down, data breach suspected, all nodes offline | 15 min | Full platform outage |
+| SEV-2 | Single node down, WHMCS provision failures, DNS sync broken | 1 h | Partial customer impact |
+| SEV-3 | Elevatoration, non-critical bug, one customer server stuck | Next business day | Limited blast radius |
+
+## Initial response (all severities)
+
+1. **Acknowledge** — Assign incident commander and scribe
+2. **Triage** — Customer-facing vs internal-only impact
+3. **Communicate** — Internal channel + status page if SEV-1/2
+4. **Preserve evidence** — Do not reboot hosts before log capture if security-related
+
+## Common incidents
+
+### API unavailable (502/503)
+
+```bash
+docker compose -f deploy/compose/compose.prod.yml ps
+docker compose -f deploy/compose/compose.prod.yml logs --tail=200 api
+curl -v https://api.example.net/api/v1/health/live
+```
+
+Check Traefik routing ([traefik](traefik.md)), PostgreSQL connectivity, disk full.
+
+### Worker queue backlog
+
+Inspect Redis queue depth and failed jobs in logs. Scale worker container or restart after fixing root cause (DNS provider down, S3 errors).
+
+### Node heartbeat stale
+
+Nodes not seen > 90s trigger alerts. Verify node-agent systemd status, outbound firewall to `wss://api.example.net/api/v1/nodes/ws`, certificate expiry.
+
+### Suspected credential leak
+
+1. Rotate affected secrets immediately ([secrets](../security/secrets.md)):
+   - `SESSION_SECRET` (invalidates all sessions)
+   - `WHMCS_API_SECRET` per installation
+   - `NODE_TOKEN` for compromised node
+   - `S3_ACCESS_KEY` / database password
+2. Review audit logs and WHMCS module call logs
+3. Enable mTLS if not already active for WHMCS integration
+
+### Tenant isolation concern (SEV-1)
+
+1. Disable affected API endpoints if exploit is active
+2. Capture request IDs from audit log
+3. Engage security lead; reference [threat model](../security/threat-model.md) T21
+
+## WHMCS-specific
+
+Provisioning failures often appear in WHMCS **Utilities → Logs → Module Log**. Cross-check GameCloud API logs with `X-HGC-Integration-Id` header.
+
+Run reconciliation dry-run before live fixes: [reconciliation](../integrations/whmcs/reconciliation.md).
+
+## Post-incident
+
+Within 5 business days:
+
+- [ ] Timeline document (UTC)
+- [ ] Root cause (5 whys)
+- [ ] Action items with owners
+- [ ] Update runbooks if gaps found
+
+## Contacts and escalation
+
+Document internally:
+
+- On-call rotation
+- WHMCS admin contact
+- DNS provider support
+- Hosting provider (game nodes)
+
+## Related
+
+- [Disaster recovery](disaster-recovery.md)
+- [Threat model](../security/threat-model.md)
+- [WHMCS troubleshooting](../integrations/whmcs/troubleshooting.md)