102 lines
2.6 KiB
Markdown
102 lines
2.6 KiB
Markdown
# Game node drain
|
|
|
|
Gracefully remove a game node from scheduling before maintenance, decommissioning, or migration.
|
|
|
|
## When to drain
|
|
|
|
- OS kernel / Docker upgrade on the node
|
|
- Hardware maintenance or rack move
|
|
- Shrinking cluster capacity
|
|
- Replacing a node with new hardware ([server migration](server-migration.md))
|
|
|
|
Do **not** drain the last node in a region if active servers have no migration target.
|
|
|
|
## Procedure
|
|
|
|
### 1. Set node to DRAINING
|
|
|
|
Via admin API:
|
|
|
|
```http
|
|
PATCH /api/v1/admin/nodes/{nodeId}
|
|
Authorization: Bearer <admin-token>
|
|
Content-Type: application/json
|
|
|
|
{ "status": "DRAINING" }
|
|
```
|
|
|
|
The scheduler stops placing new servers on this node. Existing running servers continue until stopped by users or idle shutdown.
|
|
|
|
### 2. Wait for natural shutdown (optional)
|
|
|
|
If maintenance window allows, wait for idle shutdown policy to stop inactive servers. Monitor:
|
|
|
|
```sql
|
|
SELECT id, status, "nodeId" FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status = 'RUNNING';
|
|
```
|
|
|
|
### 3. Migrate or stop remaining servers
|
|
|
|
For each running server:
|
|
|
|
- **Preferred:** Customer stops server via panel; reprovision on another node on next start (scheduler picks eligible node)
|
|
- **Admin:** Force stop via admin API, then start — triggers reschedule if original node is DRAINING
|
|
- **Maintenance:** Communicate window to affected customers
|
|
|
|
### 4. Confirm empty node
|
|
|
|
```sql
|
|
SELECT COUNT(*) FROM "GameServer" WHERE "nodeId" = '<nodeId>' AND status IN ('RUNNING', 'STARTING');
|
|
```
|
|
|
|
Expected: `0` before maintenance on Docker host.
|
|
|
|
### 5. Stop node-agent
|
|
|
|
```bash
|
|
sudo systemctl stop hgc-node-agent
|
|
```
|
|
|
|
Configured in `deploy/systemd/node-agent.service`.
|
|
|
|
### 6. Perform maintenance
|
|
|
|
Apply updates via `deploy/ansible/game-node.yml` or manual steps. See [game node operations](game-node.md).
|
|
|
|
### 7. Return to service or decommission
|
|
|
|
**Return:**
|
|
|
|
```http
|
|
PATCH /api/v1/admin/nodes/{nodeId}
|
|
{ "status": "ONLINE" }
|
|
```
|
|
|
|
```bash
|
|
sudo systemctl start hgc-node-agent
|
|
```
|
|
|
|
**Decommission:**
|
|
|
|
1. Revoke enrollment token in control plane
|
|
2. Remove `GameNode` row or mark `OFFLINE` permanently
|
|
3. Wipe `/var/lib/hgc-node` data if disks are reused
|
|
|
|
## Edge cases
|
|
|
|
| Situation | Action |
|
|
|-----------|--------|
|
|
| Server stuck STARTING during drain | Admin stop + investigate node-agent logs |
|
|
| Port range exhausted on other nodes | Add node or expand range before drain |
|
|
| Node loses connectivity mid-drain | Servers marked UNKNOWN; reconcile when agent returns |
|
|
|
|
## Automation
|
|
|
|
Future: automated drain API with `--wait-empty` timeout. Until then, use SQL + admin API as above.
|
|
|
|
## Related
|
|
|
|
- [Game node operations](game-node.md)
|
|
- [Server migration](server-migration.md)
|
|
- [Upgrades](upgrades.md)
|