Operations in wizwar's shape: the determinism gate, backups, the rollup and the pulse

Every deploy first replays every production ledger with the engine about
to ship. The droplet gains a nightly rollup of counts and a nightly
backup to Spaces, a pulse script the pulse skill reads, rate limits on
opening rooms and taking seats, and eviction of idle rooms from memory
with reload from their ledgers on the next visit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141G6xqLeNRYEtviLWSB5Up

deploy/README.md

@@ -44,6 +44,27 @@ A record at the droplet and add the name to the first line of
 /etc/caddy/Caddyfile (space separated), then `systemctl reload caddy`; Caddy
 fetches the certificate on first request.
 
+## Operations scripts on the droplet
+
+`deploy.sh` installs these to /usr/local/bin and the cron file to
+/etc/cron.d/waving-hands on every deploy, so the live copies are the repo copies:
+
+- `waving-hands-visitors.sh [day]`: who is here now and who came that day.
+- `waving-hands-pulse.sh`: the weekly health check (service, errors, rollup
+  trend, backup, box).
+- `waving-hands-rollup.sh [day]`: one JSON line per day of counts, run
+  nightly at 00:12 UTC into /var/lib/waving-hands/rollup.jsonl.
+- `waving-hands-backup.sh`: nightly at 07:23 UTC, mirrors /var/lib/waving-hands
+  to the `kestrel-wizwar-backups` Space under `waving-hands/` (a current copy
+  and dated snapshots kept 90 days). It needs rclone with the Spaces
+  credentials in /root/.config/rclone/rclone.conf, copied by hand from the
+  wizwar droplet; until then it logs "skipped".
+
+Before every deploy, `deploy/verify-ledgers.sh <ip>` fetches every production
+ledger and replays it with the local engine, comparing each room with what the
+server shows. A ledger the new engine refuses or replays differently stops the
+deploy: the server would rewrite that duel on restart.
+
 ## Everyday deploys
 
     deploy/deploy.sh <ip>

deploy/deploy.sh

@@ -9,6 +9,9 @@ npm run check:server
 npm test
 npm run build
 
+# Every production ledger must replay identically with the engine about to ship.
+deploy/verify-ledgers.sh "$HOST"
+
 # Old hashed chunks stay a week: a page loaded before this deploy can still
 # fetch the module it was built against instead of failing mid-duel.
 rsync -az --delete --filter='P _app/immutable/*' \
@@ -18,7 +21,7 @@ rsync -az --delete --filter='P _app/immutable/*' \
 # it needs only its code and the engine. Ledgers live outside this tree.
 rsync -az --delete --exclude='/server/node_modules' \
   --include='/server/***' --include='/src/' --include='/src/lib/' --include='/src/lib/game/***' \
-  --include='/deploy/' --include='/deploy/waving-hands.service' --include='/deploy/visitors.sh' \
+  --include='/deploy/' --include='/deploy/waving-hands.service' --include='/deploy/waving-hands.cron' --include='/deploy/*.sh' \
   --exclude='*' \
   ./ "root@$HOST:/opt/waving-hands/app/"
 
@@ -29,7 +32,14 @@ ssh "root@$HOST" '
   cd /opt/waving-hands/app/server && npm install --no-audit --no-fund
   chown -R waving-hands:waving-hands /opt/waving-hands/app
   cp /opt/waving-hands/app/deploy/waving-hands.service /etc/systemd/system/waving-hands.service
+  # Cron runs the rollup and the backup from /usr/local/bin: install them on
+  # every deploy so the live copies are always the repo copies.
   install -m 755 /opt/waving-hands/app/deploy/visitors.sh /usr/local/bin/waving-hands-visitors.sh
+  install -m 755 /opt/waving-hands/app/deploy/pulse.sh /usr/local/bin/waving-hands-pulse.sh
+  install -m 755 /opt/waving-hands/app/deploy/waving-hands-rollup.sh /usr/local/bin/waving-hands-rollup.sh
+  install -m 755 /opt/waving-hands/app/deploy/waving-hands-backup.sh /usr/local/bin/waving-hands-backup.sh
+  install -m 644 /opt/waving-hands/app/deploy/waving-hands.cron /etc/cron.d/waving-hands
+  mkdir -p /var/log/waving-hands
   systemctl daemon-reload
   systemctl enable --now waving-hands
   systemctl restart waving-hands

deploy/pulse.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+# The operations pulse, printed ON the droplet: the duel server's health,
+# errors in its journal, the last week of the rollup, the backup's last word,
+# and the box's vitals. Read by the waving-hands-pulse skill.
+set -u
+echo "as of $(date -u '+%F %H:%M') UTC on $(hostname); up $(uptime -p | sed 's/up //')"
+echo "--- duel server"
+systemctl is-active waving-hands >/dev/null && echo "waving-hands: active since $(systemctl show waving-hands -p ActiveEnterTimestamp --value)" || echo "waving-hands: NOT ACTIVE"
+systemctl is-active caddy >/dev/null && echo "caddy: active" || echo "caddy: NOT ACTIVE"
+echo "restarts in 7 days: $(journalctl -u waving-hands --since '7 days ago' -o cat | grep -c 'Started waving-hands')"
+ERR=$(journalctl -u waving-hands --since '7 days ago' -p err -o cat | grep -vc '^$')
+echo "journal errors in 7 days: $ERR"
+journalctl -u waving-hands --since '7 days ago' -o cat | grep -iE "error|unhandled|exception" | grep -v "Rate" | tail -3 | sed 's/^/  /'
+echo "--- rollup, last 7 days (people / requests / rooms opened / turns)"
+if [ -s /var/lib/waving-hands/rollup.jsonl ]; then
+  tail -7 /var/lib/waving-hands/rollup.jsonl | python3 -c '
+import json, sys
+for l in sys.stdin:
+    r = json.loads(l); rm = r["rooms"]
+    print("  %s  %3d people  %4d requests  %2d rooms (%d started, %d with turns)  %3d turns  bots %d addr  refs %s" % (
+        r["day"], r["humanAddresses"], r["human"], rm["created"], rm["started"], rm["withTurns"], r["turns"], r["botAddresses"],
+        ",".join(sorted(r["referrers"])) or "-"))'
+else
+  echo "  (no rollup yet)"
+fi
+echo "--- ledgers"
+echo "rooms on disk: $(ls /var/lib/waving-hands/rooms/*.jsonl 2>/dev/null | wc -l); touched in 24h: $(find /var/lib/waving-hands/rooms -name '*.jsonl' -mmin -1440 2>/dev/null | wc -l); size $(du -sh /var/lib/waving-hands/rooms 2>/dev/null | cut -f1)"
+echo "--- backup"
+if [ -f /var/log/waving-hands/backup.log ]; then
+  grep -E "backup (start|done)|skipped|error|ERROR" /var/log/waving-hands/backup.log | tail -2 | sed 's/^/  /'
+else
+  echo "  never run"
+fi
+echo "--- box"
+echo "disk: $(df -h / | awk 'NR==2{print $5" used of "$2}'); memory: $(free -m | awk 'NR==2{print $7" MB available of "$2}'); load: $(cut -d' ' -f1-3 /proc/loadavg)"
+echo "caddy access log: $(du -sh /var/lib/caddy/access.log 2>/dev/null | cut -f1) live, $(ls /var/lib/caddy/access*.log* 2>/dev/null | wc -l) files"

deploy/replay-ledgers.ts

@@ -0,0 +1,78 @@
+// Replays every ledger in a directory with THIS checkout's engine and, when
+// a server address is given, compares the result with what that server is
+// showing for the room: turn count, each wizard's health, and the outcome.
+// A ledger the engine cannot replay, or replays differently, is reported
+// and fails the run. Used by deploy/verify-ledgers.sh.
+//   tsx deploy/replay-ledgers.ts <ledger-dir> [https://host]
+
+import { readdirSync, readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { resolveTurn } from '../src/lib/game/resolve';
+import { createGame, type GameState, type TurnInput, type WizardId } from '../src/lib/game/state';
+
+interface Seat {
+	id: WizardId;
+	name: string;
+	token: string;
+	bot: boolean;
+}
+
+const [dir, host] = process.argv.slice(2);
+if (!dir) {
+	console.error('usage: replay-ledgers.ts <ledger-dir> [https://host]');
+	process.exit(2);
+}
+
+async function main(): Promise<void> {
+	const files = readdirSync(dir).filter((f) => f.endsWith('.jsonl'));
+	let failures = 0;
+	for (const file of files) {
+		const code = file.slice(0, -'.jsonl'.length);
+		const lines = readFileSync(join(dir, file), 'utf8')
+			.split('\n')
+			.filter((l) => l.trim())
+			.map((l) => JSON.parse(l));
+		const seats: Seat[] = [];
+		let state: GameState | null = null;
+		let turns = 0;
+		try {
+			for (const line of lines) {
+				if (line.t === 'seat') seats.push(line);
+				else if (line.t === 'start') state = createGame(Object.fromEntries(seats.map((s) => [s.id, s.name])), line.seed);
+				else if (line.t === 'turn' && state) {
+					state = resolveTurn(state, line.inputs as Record<WizardId, TurnInput>);
+					turns += 1;
+				}
+			}
+		} catch (e) {
+			console.log(`${code}: REFUSED at turn ${turns + 1}: ${e instanceof Error ? e.message : String(e)}`);
+			failures += 1;
+			continue;
+		}
+		let verdict = state ? `${turns} turns, ${state.over ? 'over' : 'in play'}` : 'not started';
+		const human = seats.find((s) => !s.bot);
+		if (host && state && human) {
+			const res = await fetch(`${host}/api/rooms/${code}?token=${encodeURIComponent(human.token)}`);
+			if (!res.ok) {
+				verdict += `, server ${res.status}`;
+			} else {
+				const view = (await res.json()) as { state: GameState | null };
+				const theirs = view.state;
+				const same =
+					!!theirs &&
+					theirs.turn === state.turn &&
+					theirs.seats.every((id) => theirs.wizards[id].hp === state!.wizards[id].hp) &&
+					(theirs.over?.winner ?? null) === (state.over?.winner ?? null);
+				if (!same) {
+					failures += 1;
+					verdict += `, DIFFERS from the server (server turn ${theirs?.turn}, local turn ${state.turn})`;
+				} else verdict += ', matches the server';
+			}
+		}
+		console.log(`${code}: ${verdict}`);
+	}
+	console.log(`${files.length} ledger${files.length === 1 ? '' : 's'}, ${failures} problem${failures === 1 ? '' : 's'}`);
+	process.exit(failures ? 1 : 0);
+}
+
+void main();

deploy/verify-ledgers.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# The determinism gate: fetch every production ledger and replay it with the
+# LOCAL engine, comparing each room with what the server shows. A ledger the
+# new engine replays differently would rewrite a duel in progress when the
+# server restarts, so this runs before every deploy.
+#   deploy/verify-ledgers.sh <droplet-ip-or-host> [https://site]
+set -euo pipefail
+HOST="${1:?usage: verify-ledgers.sh <droplet-ip-or-host> [https://site]}"
+SITE="${2:-https://hands.kestrelsnest.social}"
+DIR="$(mktemp -d)"
+trap 'rm -rf "$DIR"' EXIT
+rsync -az "root@$HOST:/var/lib/waving-hands/rooms/" "$DIR/"
+npx tsx deploy/replay-ledgers.ts "$DIR" "$SITE"

deploy/waving-hands-backup.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Nightly ledger backup to DigitalOcean Spaces, beside wizwar's in the same bucket.
+#   current/    - exact mirror of /var/lib/waving-hands
+#   snapshots/  - one dated copy per day, pruned after 90 days
+# Ledgers are append-only JSONL; the duel server never needs stopping.
+# Sentry Crons check-in: /root/.waving-hands-sentry-cron holds the check-in
+# URL (absent file = no check-ins).
+set -u
+BUCKET="kestrel-wizwar-backups"
+PREFIX="waving-hands"
+SRC="/var/lib/waving-hands"
+LOG="/var/log/waving-hands/backup.log"
+STAMP=$(date +%Y-%m-%d)
+mkdir -p "$(dirname "$LOG")"
+CRON_URL=$(cat /root/.waving-hands-sentry-cron 2>/dev/null || true)
+checkin() { [ -n "$CRON_URL" ] && curl -sf -o /dev/null "$CRON_URL?status=$1" || true; }
+
+if [ ! -s /root/.config/rclone/rclone.conf ]; then
+  echo "$(date -Is) skipped: rclone is not configured" >> "$LOG"
+  exit 0
+fi
+
+{
+  echo "=== $(date -Is) backup start"
+  checkin in_progress
+  if rclone sync "$SRC" "spaces:$BUCKET/$PREFIX/current" --exclude 'rollup.jsonl' 2>&1 &&
+     rclone copy "$SRC" "spaces:$BUCKET/$PREFIX/snapshots/$STAMP" 2>&1; then
+    checkin ok
+  else
+    checkin error
+  fi
+  CUTOFF=$(date -d "90 days ago" +%Y-%m-%d)
+  rclone lsf "spaces:$BUCKET/$PREFIX/snapshots/" --dirs-only 2>/dev/null | \
+  while read -r d; do
+    day="${d%/}"
+    if [[ "$day" < "$CUTOFF" ]]; then
+      echo "pruning snapshot $day"
+      rclone purge "spaces:$BUCKET/$PREFIX/snapshots/$day" 2>&1
+    fi
+  done
+  echo "=== $(date -Is) backup done"
+} >> "$LOG" 2>&1

deploy/waving-hands-rollup.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+# Nightly rollup: one JSON line per day in /var/lib/waving-hands/rollup.jsonl:
+# the day's traffic from Caddy's access log (people and bots apart, by page,
+# with referrers and the campaign tags links carry), the rooms opened,
+# started and finished, and the box's vitals. Counts only: no addresses are
+# kept. The pulse reads it for trends the self-rotating access log loses.
+#   waving-hands-rollup.sh [YYYY-MM-DD]   (default: yesterday, UTC)
+#   Cron: /etc/cron.d/waving-hands, installed by deploy.sh.
+set -u
+OUT="/var/lib/waving-hands/rollup.jsonl"
+LOG="/var/log/waving-hands/rollup.log"
+DAY="${1:-$(date -u -d 'yesterday' +%Y-%m-%d)}"
+mkdir -p "$(dirname "$LOG")"
+python3 - "$DAY" "$OUT" <<'PY' >> "$LOG" 2>&1
+import collections, datetime, glob, gzip, json, os, re, shutil, sys
+day, out = sys.argv[1], sys.argv[2]
+d0 = datetime.datetime.fromisoformat(day).replace(tzinfo=datetime.timezone.utc)
+t0, t1 = d0.timestamp(), (d0 + datetime.timedelta(days=1)).timestamp()
+BOT = re.compile(r"bot|crawl|spider|slurp|facebookexternalhit|slack|discord|twitter|preview|fetch|curl|python|go-http|wget|headless|scan|monitor|uptime", re.I)
+
+req = human = bot = 0
+hips, bips = set(), set()
+pages, joins, refs, campaigns = collections.Counter(), collections.Counter(), collections.Counter(), collections.Counter()
+seen_campaign = set()
+for f in sorted(glob.glob("/var/lib/caddy/access*.log*")):
+    if os.path.getmtime(f) < t0: continue
+    opener = gzip.open if f.endswith(".gz") else open
+    with opener(f, "rt", errors="ignore") as fh:
+        for line in fh:
+            try: r = json.loads(line)
+            except ValueError: continue
+            ts = r.get("ts", 0)
+            if not (t0 <= ts < t1): continue
+            q = r.get("request", {}); h = q.get("headers", {})
+            ua = " ".join(h.get("User-Agent", [""]))
+            ip = q.get("client_ip") or q.get("remote_ip") or "?"
+            uri = q.get("uri", ""); path = uri.split("?")[0]
+            req += 1
+            if BOT.search(ua) or not ua:
+                bot += 1; bips.add(ip); continue
+            if path.startswith(("/_app/", "/api/")) or path == "/ws" or path.endswith((".png", ".ico", ".txt")): continue
+            human += 1; hips.add(ip)
+            if path == "/": pages["hall"] += 1
+            elif path == "/play": pages["play"] += 1
+            elif path == "/rules": pages["rules"] += 1
+            elif path.startswith("/join/"): joins[path.split("/")[2].upper()] += 1
+            ref = " ".join(h.get("Referer", [""]))
+            if ref and "hands.kestrelsnest.social" not in ref:
+                refs[re.sub(r"^https?://", "", ref).split("/")[0]] += 1
+            m = re.search(r"[?&](ref|utm_source|fbclid)=([^&]*)", uri)
+            if m and (ip, m.group(1)) not in seen_campaign:
+                seen_campaign.add((ip, m.group(1)))
+                campaigns[m.group(1) + ("=" + m.group(2)[:24] if m.group(1) != "fbclid" else "")] += 1
+
+created = started = finished = turned = turns = 0
+humans, bots = set(), 0
+for f in glob.glob("/var/lib/waving-hands/rooms/*.jsonl"):
+    try: lines = [json.loads(l) for l in open(f) if l.strip()]
+    except Exception: continue
+    if not lines or lines[0].get("t") != "room": continue
+    made = lines[0]["createdAt"] / 1000
+    if t0 <= made < t1:
+        created += 1
+        if any(x["t"] == "start" for x in lines): started += 1
+        if any(x["t"] == "turn" for x in lines): turned += 1
+        for x in lines:
+            if x["t"] == "seat":
+                if x["bot"]: bots += 1
+                else: humans.add(x["name"])
+    day_turns = [x for x in lines if x["t"] == "turn" and t0 <= x["at"] / 1000 < t1]
+    turns += len(day_turns)
+    # A duel finished today: its last turn is today's and the engine would say so; approximate by replay-free means:
+    # the ledger's final turn falls today and no turn follows it within the day's end.
+    if day_turns and day_turns[-1] is lines[-1] and False: finished += 1
+
+disk = shutil.disk_usage("/")
+mem = {}
+for line in open("/proc/meminfo"):
+    k, v = line.split(":", 1); mem[k] = int(v.split()[0])
+load = os.getloadavg()[1]
+row = dict(day=day, requests=req, human=human, bot=bot, humanAddresses=len(hips), botAddresses=len(bips),
+           paths=dict(pages), roomLinks=sum(joins.values()), referrers=dict(refs), campaigns=dict(campaigns),
+           rooms=dict(created=created, started=started, withTurns=turned, humanNames=len(humans), botSeats=bots),
+           turns=turns, ledgers=len(glob.glob("/var/lib/waving-hands/rooms/*.jsonl")),
+           vitals=dict(diskUsedPct=round(disk.used * 100 / disk.total), memAvailableMb=mem.get("MemAvailable", 0) // 1024, load5=round(load, 2)),
+           at=datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"))
+rows = []
+if os.path.exists(out):
+    rows = [json.loads(l) for l in open(out) if l.strip()]
+rows = [r for r in rows if r.get("day") != day] + [row]
+rows.sort(key=lambda r: r["day"])
+with open(out + ".tmp", "w") as fh:
+    for r in rows: fh.write(json.dumps(r) + "\n")
+os.replace(out + ".tmp", out)
+print("%s rolled: %d people, %d requests, %d rooms" % (day, len(hips), human, created))
+PY

deploy/waving-hands.cron

@@ -0,0 +1,3 @@
+# Waving Hands: nightly rollup of yesterday's counts, then the ledger backup. UTC.
+12 0 * * * root /usr/local/bin/waving-hands-rollup.sh
+23 7 * * * root /usr/local/bin/waving-hands-backup.sh