A deploy waits for the game server to answer and fails with its journal if it does not; a report id is checked whole; the backup's notes say what it keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jm2auWk6RP71CjaAb4FMoG

template/deploy/README.md

@@ -57,7 +57,7 @@ deploy, so the live copies are the repo copies:
   nightly at 00:12 UTC into /var/lib/__SLUG__/rollup.jsonl.
 - `__SLUG__-backup.sh`: nightly at 07:23 UTC, mirrors /var/lib/__SLUG__
   to the `kestrel-wizwar-backups` Space under `__SLUG__/` (a current copy
-  and dated snapshots kept 90 days). It needs rclone with the Spaces
+  less the rollup, and dated snapshots kept 90 days). It needs rclone with the Spaces
   credentials in /root/.config/rclone/rclone.conf, copied by hand from the
   wizwar droplet; until then it logs "skipped".
 
@@ -84,14 +84,16 @@ deploy: the server would rewrite that game on restart.
 
 Runs the type-checks, the tests and the build locally, rsyncs `build/` to
 the droplet keeping the previous week's hashed assets, rsyncs the server and
-engine sources, installs dependencies, and restarts the game server. A room
+engine sources, installs dependencies, and restarts the game server, then
-is replayed from its ledger when the server comes back, which takes a few
+waits up to twenty seconds for it to answer HTTP and fails with the tail of
-seconds; Caddy holds requests that land in the gap.
+its journal if it does not. A room is replayed from its ledger when the
+server comes back, which takes a few seconds; Caddy holds requests that
+land in the gap.
 
 ## Operations
 
 - Logs: `ssh root@<ip> journalctl -u __SLUG__ -f` for the game server,
-  `journalctl -u caddy -f` and /var/lib/caddy/access.log for the web side.
+  `journalctl -u caddy -f` and /var/lib/caddy/__SLUG__-access.log for the web side.
 - Restart: `ssh root@<ip> systemctl restart __SLUG__`
 - Rooms: `/var/lib/__SLUG__/rooms/<CODE>.jsonl`, one ledger per game; the
   nightly backup keeps them in the Space.

template/deploy/__SLUG__-backup.sh

@@ -1,13 +1,13 @@
 #!/usr/bin/env bash
 # Nightly ledger backup to DigitalOcean Spaces, one prefix per game in the shared bucket.
-#   current/    - exact mirror of /var/lib/__SLUG__
+#   current/    - mirror of /var/lib/__SLUG__, less the rollup (rebuilt from logs)
 #   snapshots/  - one dated copy per day, pruned after 90 days
 # Ledgers are append-only JSONL; the game server never needs stopping.
 # Sentry Crons check-in: /root/.__SLUG__-sentry-cron holds the check-in
 # URL (absent file = no check-ins). SCHEDULE must match /etc/cron.d/__SLUG__.
 set -u
 SCHEDULE="23 7 * * *"
-BUCKET="kestrel-wizwar-backups"
+BUCKET="kestrel-wizwar-backups" # the hall's shared Space, named for its first tenant
 PREFIX="__SLUG__"
 SRC="/var/lib/__SLUG__"
 LOG="/var/log/__SLUG__/backup.log"

template/deploy/deploy.sh

@@ -49,7 +49,17 @@ ssh "root@$HOST" '
   systemctl enable --now __SLUG__
   systemctl restart __SLUG__
   systemctl reload caddy
-  sleep 1
+  # "active" means only that the process started: wait for it to answer HTTP,
-  systemctl --no-pager -l status __SLUG__ | head -3
+  # so a server that dies loading its code fails the deploy instead of looping.
+  for i in $(seq 20); do
+    if curl -s -o /dev/null http://127.0.0.1:__PORT__/api/rooms/ZZZZ; then
+      systemctl --no-pager -l status __SLUG__ | head -3
+      exit 0
+    fi
+    sleep 1
+  done
+  echo "the game server is not answering:" >&2
+  journalctl -u __SLUG__ -n 25 --no-pager >&2
+  exit 1
 '
 echo "deployed."

template/deploy/report-reply.sh

@@ -8,7 +8,7 @@ STATUS="${3:?usage: report-reply.sh <host> <reportId> <status> <text...>}"
 shift 3
 TEXT="$*"
 case "$STATUS" in resolved|by-design|open) ;; *) echo "status must be resolved, by-design, or open" >&2; exit 1;; esac
-case "$REPORT_ID" in *[!0-9a-f]*|"") echo "a report id is eight hex characters" >&2; exit 1;; esac
+[[ "$REPORT_ID" =~ ^[0-9a-f]{8}$ ]] || { echo "a report id is eight hex characters" >&2; exit 1; }
 LINE=$(REPORT_ID="$REPORT_ID" STATUS="$STATUS" TEXT="$TEXT" python3 -c '
 import json, os, datetime
 print(json.dumps({"reportId": os.environ["REPORT_ID"], "at": datetime.datetime.now(datetime.timezone.utc).isoformat().replace("+00:00", "Z"), "status": os.environ["STATUS"], "text": os.environ["TEXT"]}))')