Eric WagonerandClaude Fable 5
92613d7386
Operational: auto-reboot for kernel patches (09:30 UTC), Caddy access logs at /var/lib/caddy (self-rotating; the sandbox denies /var/log), logrotate for the backup log, a restore drill proving a Spaces snapshot boots 56/56 rooms clean, the about page's plain sentence on permanent recording, and /wizwar-pulse to read it all weekly. Rules: safes now smash — attacks aimed at the box accumulate on it and the fifteenth point bursts it (widening, ungated). And SLOW DEATH's bites pause for a victim holding an ABSORB — the card named by the FAQ, not Absorb Spell — each soaking exactly one point, rev-gated at 13 with the instant-bite legacy pinned. Bots weigh the soak against their life. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015RCWSTnb1KYTPyL4GmhGnF
65 lines
3.2 KiB
Markdown
65 lines
3.2 KiB
Markdown
---
|
||
name: wizwar-pulse
|
||
description: The weekly Wiz-War operations pulse — droplet health, error and traffic signals, backup status, and who has been playing. Use when Eric asks how the server is doing, wants an ops check, or says "run the pulse".
|
||
---
|
||
|
||
# The Wiz-War ops pulse
|
||
|
||
One invocation, one screen: how the table is doing. Gather everything,
|
||
then report in five short sections. Lead with anything ALARMING (service
|
||
down, memory near the cap, failed backup, new Sentry issues); if all is
|
||
quiet, say so in one line before the details.
|
||
|
||
## Gather
|
||
|
||
1. **Box health** (one ssh to root@104.236.96.198):
|
||
- `systemctl is-active wizwar caddy` — both must be active
|
||
- `systemctl status wizwar` Memory line — the cap is 700M; worry
|
||
past ~500M steady-state
|
||
- `free -m`, `df -h /`, `uptime` — swap in heavy use, disk past
|
||
~70%, or load near 1.0 (single vCPU) are the resize triggers
|
||
- `journalctl -u wizwar --since "-7 days" | grep -ci "unhandled
|
||
protocol error"` — anything nonzero deserves a look (Sentry has
|
||
the stack traces)
|
||
2. **Sentry** (MCP connector, org `locallygrownnet`, project `wizwar`):
|
||
- `search_issues` for `is:unresolved` — new issues are the headline
|
||
- `get_monitor_details` for cron monitor slug `new-monitor` (named
|
||
wizwar-backup) — last check-in must be ok and recent; the uptime
|
||
monitor alerts on its own but note any incidents
|
||
- New-project caveat: results can lag ingest by minutes; the
|
||
connector reads fine but 403s on create operations
|
||
3. **Backups**: `tail -3 /var/log/wizwar/backup.log` (or its rotation)
|
||
for the last run, plus the cron monitor above for missed runs
|
||
4. **The table itself**:
|
||
- Room count and growth: `ls /var/lib/wizwar/rooms/*.jsonl | wc -l`
|
||
- New human names this week: room ledgers' join lines minus bots
|
||
(Automaton*) and known names, on rooms created in the last 7 days
|
||
(meta line `createdAt`)
|
||
- Unanswered feedback count from /var/lib/wizwar/feedback.jsonl —
|
||
if any, offer to run the reports desk (`/wizwar-reports`)
|
||
- Access-log pulse: `wc -l /var/lib/caddy/access.log` and a count of
|
||
distinct `client_ip`s in the last day, for the growth line
|
||
|
||
## Report
|
||
|
||
Five sections, tight: **Verdict** (one line), **Box**, **Errors &
|
||
monitors**, **Backups**, **The table** (games, new faces, feedback).
|
||
Numbers with their thresholds, not raw dumps. End with any recommended
|
||
action, or "nothing needs you."
|
||
|
||
## Standing knowledge
|
||
|
||
- The service restarts cleanly (ledgers replay); restarts in the journal
|
||
during deploys are normal, not incidents.
|
||
- Room memory is evicted after idle (finished 30 min, others 24 h);
|
||
boot-restores load everything then shed, so post-deploy memory spikes
|
||
settle within the hour.
|
||
- Unattended-upgrades reboots the box at 09:30 UTC when a kernel patch
|
||
requires it; a reboot there is maintenance, not an outage.
|
||
- Caddy access logs live at /var/lib/caddy/access.log (self-rotating,
|
||
10MiB × 5); the systemd sandbox denies /var/log/caddy.
|
||
- Restore drill (2026-09-01): a Spaces snapshot booted 56/56 rooms
|
||
clean. The path: `rclone copy spaces:kestrel-wizwar-backups/wizwar/
|
||
snapshots/<date> ...` on the droplet, then run a local server with
|
||
WIZWAR_DATA_DIR pointed at its rooms dir. Re-drill quarterly-ish.
|