The nightly rollup keeps what the access log forgets; two doors get a limit

Three sysadmin items ahead of the public announcement's arrivals.

deploy/wizwar-rollup.sh writes one JSON line per UTC day to
/var/lib/wizwar/rollup.jsonl at 00:10: yesterday's traffic from Caddy's
access log (requests, human vs bot addresses as counts only, socket
connects, path mix, external referrers), the table's growth (new
rooms, human seats and names, lifetime game totals, reports and
replies), and the box's vitals (service memory and peak, disk, load,
protocol errors, service starts, ledger size). Re-rolling a day
replaces its line. It checks in to a Sentry cron monitor of its own,
whose URL deploy.sh derives from the backup monitor's on first
install, alongside the cron entry. The pulse gains a Trends section
that reads the last week of it. Caddy keeps 30 log files instead of 5
as the raw backing.

Per-address limits on the two doors anyone may use unseated: 12 new
rooms and 6 reports per address per hour, in a sliding window keyed by
the address Caddy forwards. The per-connection cap stays; it reset on
reconnect, which is what a script would do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jm2auWk6RP71CjaAb4FMoG
This commit is contained in:
Eric Wagoner
2026-09-03 11:30:44 -04:00
co-authored by Claude Fable 5.1
parent 3f909b485b
commit ca108f83a6
7 changed files with 205 additions and 6 deletions
+20 -4
View File
@@ -30,7 +30,18 @@ quiet, say so in one line before the details.
connector reads fine but 403s on create operations
3. **Backups**: `tail -3 /var/log/wizwar/backup.log` (or its rotation)
for the last run, plus the cron monitor above for missed runs
4. **The table itself**:
4. **Trends** (`/var/lib/wizwar/rollup.jsonl`, one JSON line per UTC day
written at 00:10 by /usr/local/bin/wizwar-rollup.sh; counts only, no
addresses): the last 7 rows give requests, human vs bot addresses,
socket connects, path mix (/, /clips, /watch, /ws), external
referrers (reddit.com, boardgamegeek.com after the 2026-09-03
announcement), new rooms, human seats and names, reports/replies,
and vitals (memMB, memPeakMB, diskPct, load1, protocolErrors,
serviceStarts, ledgerMB). Report day-over-day direction, not raw
rows. Its Sentry cron monitor is slug `wizwar-rollup` (a missed
check-in there means the rollup itself is broken). Re-roll a day by
hand: `wizwar-rollup.sh YYYY-MM-DD`.
5. **The table itself**:
- Room count and growth: `ls /var/lib/wizwar/rooms/*.jsonl | wc -l`
- New human names this week: room ledgers' join lines minus bots
(Automaton*) and known names, on rooms created in the last 7 days
@@ -42,8 +53,9 @@ quiet, say so in one line before the details.
## Report
Five sections, tight: **Verdict** (one line), **Box**, **Errors &
monitors**, **Backups**, **The table** (games, new faces, feedback).
Six sections, tight: **Verdict** (one line), **Box**, **Errors &
monitors**, **Backups**, **Trends** (the week's direction from the
rollup), **The table** (games, new faces, feedback).
Numbers with their thresholds, not raw dumps. End with any recommended
action, or "nothing needs you."
@@ -57,7 +69,11 @@ action, or "nothing needs you."
- Unattended-upgrades reboots the box at 09:30 UTC when a kernel patch
requires it; a reboot there is maintenance, not an outage.
- Caddy access logs live at /var/lib/caddy/access.log (self-rotating,
10MiB × 5); the systemd sandbox denies /var/log/caddy.
10MiB × 30 since 2026-09-03); the systemd sandbox denies /var/log/caddy.
- Per-address limits (2026-09-03): 12 new rooms and 6 reports per
address per hour, in packages/server/src/ratelimit.ts. A player who
hits one sees "try again in an hour"; a pulse showing many refused
creates from one address is a script, not a friend.
- Restore drill (2026-09-01): a Spaces snapshot booted 56/56 rooms
clean. The path: `rclone copy spaces:kestrel-wizwar-backups/wizwar/
snapshots/<date> ...` on the droplet, then run a local server with