# Game node drain

Gracefully remove a game node from scheduling before maintenance, decommissioning, or migration.

## When to drain

- OS kernel / Docker upgrade on the node
- Hardware maintenance or rack move
- Shrinking cluster capacity
- Replacing a node with new hardware ([server migration](server-migration.md))

Do **not** drain the last node in a region if active servers have no migration target.

## Procedure

### 1. Set node to DRAINING

Via admin API:

```http
PATCH /api/v1/admin/nodes/{nodeId}
Authorization: Bearer <admin-token>
Content-Type: application/json

{ "status": "DRAINING" }
```

The scheduler stops placing new servers on this node. Existing running servers continue until stopped by users or idle shutdown.

### 2. Wait for natural shutdown (optional)

If maintenance window allows, wait for idle shutdown policy to stop inactive servers. Monitor:

```sql
SELECT id, status, "nodeId" FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status = 'RUNNING';
```

### 3. Migrate or stop remaining servers

For each running server:

- **Preferred:** Customer stops server via panel; reprovision on another node on next start (scheduler picks eligible node)
- **Admin:** Force stop via admin API, then start — triggers reschedule if original node is DRAINING
- **Maintenance:** Communicate window to affected customers

### 4. Confirm empty node

```sql
SELECT COUNT(*) FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status IN ('RUNNING', 'STARTING');
```

Expected: `0` before maintenance on Docker host.

### 5. Stop node-agent

```bash
sudo systemctl stop hgc-node-agent
```

Configured in `deploy/systemd/node-agent.service`.

### 6. Perform maintenance

Apply updates via `deploy/ansible/game-node.yml` or manual steps. See [game node operations](game-node.md).

### 7. Return to service or decommission

**Return:**

```http
PATCH /api/v1/admin/nodes/{nodeId}
{ "status": "ONLINE" }
```

```bash
sudo systemctl start hgc-node-agent
```

**Decommission:**

1. Revoke enrollment token in control plane
2. Remove `GameNode` row or mark `OFFLINE` permanently
3. Wipe `/var/lib/hgc-node` data if disks are reused

## Edge cases

| Situation | Action |
|-----------|--------|
| Server stuck STARTING during drain | Admin stop + investigate node-agent logs |
| Port range exhausted on other nodes | Add node or expand range before drain |
| Node loses connectivity mid-drain | Servers marked UNKNOWN; reconcile when agent returns |

## Automation

Future: automated drain API with `--wait-empty` timeout. Until then, use SQL + admin API as above.

## Related

- [Game node operations](game-node.md)
- [Server migration](server-migration.md)
- [Upgrades](upgrades.md)