From 60b3c0e60a69abc5c6b0191911db192141646dc8 Mon Sep 17 00:00:00 2001 From: Eric Wagoner Date: Tue, 29 Sep 2026 12:25:38 -0400 Subject: [PATCH] A deploy waits for the game server to answer and fails with its journal if it does not; a report id is checked whole; the backup's notes say what it keeps Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Jm2auWk6RP71CjaAb4FMoG --- template/deploy/README.md | 12 +++++++----- template/deploy/__SLUG__-backup.sh | 4 ++-- template/deploy/deploy.sh | 14 ++++++++++++-- template/deploy/report-reply.sh | 2 +- 4 files changed, 22 insertions(+), 10 deletions(-) diff --git a/template/deploy/README.md b/template/deploy/README.md index 01eaeea..1d2bab3 100644 --- a/template/deploy/README.md +++ b/template/deploy/README.md @@ -57,7 +57,7 @@ deploy, so the live copies are the repo copies: nightly at 00:12 UTC into /var/lib/__SLUG__/rollup.jsonl. - `__SLUG__-backup.sh`: nightly at 07:23 UTC, mirrors /var/lib/__SLUG__ to the `kestrel-wizwar-backups` Space under `__SLUG__/` (a current copy - and dated snapshots kept 90 days). It needs rclone with the Spaces + less the rollup, and dated snapshots kept 90 days). It needs rclone with the Spaces credentials in /root/.config/rclone/rclone.conf, copied by hand from the wizwar droplet; until then it logs "skipped". @@ -84,14 +84,16 @@ deploy: the server would rewrite that game on restart. Runs the type-checks, the tests and the build locally, rsyncs `build/` to the droplet keeping the previous week's hashed assets, rsyncs the server and -engine sources, installs dependencies, and restarts the game server. A room -is replayed from its ledger when the server comes back, which takes a few -seconds; Caddy holds requests that land in the gap. +engine sources, installs dependencies, and restarts the game server, then +waits up to twenty seconds for it to answer HTTP and fails with the tail of +its journal if it does not. A room is replayed from its ledger when the +server comes back, which takes a few seconds; Caddy holds requests that +land in the gap. ## Operations - Logs: `ssh root@ journalctl -u __SLUG__ -f` for the game server, - `journalctl -u caddy -f` and /var/lib/caddy/access.log for the web side. + `journalctl -u caddy -f` and /var/lib/caddy/__SLUG__-access.log for the web side. - Restart: `ssh root@ systemctl restart __SLUG__` - Rooms: `/var/lib/__SLUG__/rooms/.jsonl`, one ledger per game; the nightly backup keeps them in the Space. diff --git a/template/deploy/__SLUG__-backup.sh b/template/deploy/__SLUG__-backup.sh index 5f934a0..f0b6d34 100755 --- a/template/deploy/__SLUG__-backup.sh +++ b/template/deploy/__SLUG__-backup.sh @@ -1,13 +1,13 @@ #!/usr/bin/env bash # Nightly ledger backup to DigitalOcean Spaces, one prefix per game in the shared bucket. -# current/ - exact mirror of /var/lib/__SLUG__ +# current/ - mirror of /var/lib/__SLUG__, less the rollup (rebuilt from logs) # snapshots/ - one dated copy per day, pruned after 90 days # Ledgers are append-only JSONL; the game server never needs stopping. # Sentry Crons check-in: /root/.__SLUG__-sentry-cron holds the check-in # URL (absent file = no check-ins). SCHEDULE must match /etc/cron.d/__SLUG__. set -u SCHEDULE="23 7 * * *" -BUCKET="kestrel-wizwar-backups" +BUCKET="kestrel-wizwar-backups" # the hall's shared Space, named for its first tenant PREFIX="__SLUG__" SRC="/var/lib/__SLUG__" LOG="/var/log/__SLUG__/backup.log" diff --git a/template/deploy/deploy.sh b/template/deploy/deploy.sh index 556f2cc..bb754e1 100755 --- a/template/deploy/deploy.sh +++ b/template/deploy/deploy.sh @@ -49,7 +49,17 @@ ssh "root@$HOST" ' systemctl enable --now __SLUG__ systemctl restart __SLUG__ systemctl reload caddy - sleep 1 - systemctl --no-pager -l status __SLUG__ | head -3 + # "active" means only that the process started: wait for it to answer HTTP, + # so a server that dies loading its code fails the deploy instead of looping. + for i in $(seq 20); do + if curl -s -o /dev/null http://127.0.0.1:__PORT__/api/rooms/ZZZZ; then + systemctl --no-pager -l status __SLUG__ | head -3 + exit 0 + fi + sleep 1 + done + echo "the game server is not answering:" >&2 + journalctl -u __SLUG__ -n 25 --no-pager >&2 + exit 1 ' echo "deployed." diff --git a/template/deploy/report-reply.sh b/template/deploy/report-reply.sh index 18d3a67..92fd965 100755 --- a/template/deploy/report-reply.sh +++ b/template/deploy/report-reply.sh @@ -8,7 +8,7 @@ STATUS="${3:?usage: report-reply.sh }" shift 3 TEXT="$*" case "$STATUS" in resolved|by-design|open) ;; *) echo "status must be resolved, by-design, or open" >&2; exit 1;; esac -case "$REPORT_ID" in *[!0-9a-f]*|"") echo "a report id is eight hex characters" >&2; exit 1;; esac +[[ "$REPORT_ID" =~ ^[0-9a-f]{8}$ ]] || { echo "a report id is eight hex characters" >&2; exit 1; } LINE=$(REPORT_ID="$REPORT_ID" STATUS="$STATUS" TEXT="$TEXT" python3 -c ' import json, os, datetime print(json.dumps({"reportId": os.environ["REPORT_ID"], "at": datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z"), "status": os.environ["STATUS"], "text": os.environ["TEXT"]}))')