The hall keeps the night on the shared host: one job runs every game's rollup and backup and checks in with Sentry once for each

Eight cron monitors, two per game, watched eight scripts that run on the
same box at the same two minutes. hall-nightly.sh now runs the hall's own
rollup and every game's rollup at 00:12, and every game's backup at 07:23,
finding the games by their scripts in /usr/local/bin, and checks in once
for each job to the gamehall project's hall-rollup and hall-backup
monitors. The template's rollup and backup say by their exit status
whether they managed, the skipped backup included, and its deploy leaves
the game's own cron file off a host the hall schedules.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

template/deploy/README.md

@@ -46,9 +46,11 @@ A fresh host, when one is needed: `doctl compute droplet create hall --region ny
 
 ## Operations scripts on the droplet
 
-`deploy.sh` installs these to /usr/local/bin, the `traffic.py` they share to
-/usr/local/lib/__SLUG__, and the cron file to /etc/cron.d/__SLUG__ on every
-deploy, so the live copies are the repo copies:
+`deploy.sh` installs these to /usr/local/bin and the `traffic.py` they share
+to /usr/local/lib/__SLUG__ on every deploy, so the live copies are the repo
+copies. The cron file goes to /etc/cron.d/__SLUG__ only on a box of the game's
+own: on the hall's shared host, `hall-nightly.sh` (in the kit's `gamehall/ops`)
+runs every game's rollup and backup from /etc/cron.d/gamehall.
 
 - `__SLUG__-visitors.sh [day]`: who is here now and who came that day.
 - `__SLUG__-pulse.sh`: the weekly health check (service, errors, rollup
@@ -62,10 +64,13 @@ deploy, so the live copies are the repo copies:
   wizwar droplet; until then it logs "skipped".
 
 Errors from the game server go to Sentry, project `__SLUG__` in the
-locallygrownnet organisation; the DSN is in the unit file. The nightly rollup
-and backup check in with Sentry Crons when /root/.__SLUG__-sentry-cron-rollup
-and /root/.__SLUG__-sentry-cron hold their check-in URLs (the ingest
-URL with the project's cron path and public key), so a missed night is noticed.
+locallygrownnet organisation; the DSN is in the unit file. On the shared host
+the hall's nightly job checks in with Sentry Crons once for every game's
+rollup and once for every backup (monitors `hall-rollup` and `hall-backup` in
+the `gamehall` project), and each game's script reports by its exit status.
+A game on its own box checks in for itself when /root/.__SLUG__-sentry-cron-rollup
+and /root/.__SLUG__-sentry-cron hold check-in URLs (the ingest URL with the
+project's cron path and public key); absent, it sends nothing.
 
 To have every new issue, regression and reappearance posted to Slack the way
 wizwar's are, run `deploy/sentry-slack-alert.sh [#channel]` once from your own

template/deploy/__SLUG__-backup.sh

@@ -5,6 +5,8 @@
 # Ledgers are append-only JSONL; the game server never needs stopping.
 # Sentry Crons check-in: /root/.__SLUG__-sentry-cron holds the check-in
 # URL (absent file = no check-ins). SCHEDULE must match /etc/cron.d/__SLUG__.
+# On a host the hall shares, hall-nightly.sh runs this and checks in once
+# for every game, so the file is absent and the exit status is the word.
 set -u
 SCHEDULE="23 7 * * *"
 BUCKET="kestrel-wizwar-backups" # the hall's shared Space, named for its first tenant
@@ -27,9 +29,10 @@ checkin() {
 
 if [ ! -s /root/.config/rclone/rclone.conf ]; then
   echo "$(date -Is) skipped: rclone is not configured" >> "$LOG"
-  exit 0
+  exit 1
 fi
 
+status=0
 {
   echo "=== $(date -Is) backup start"
   if rclone sync "$SRC" "spaces:$BUCKET/$PREFIX/current" --exclude 'rollup.jsonl' 2>&1 &&
@@ -37,6 +40,7 @@ fi
     checkin ok
   else
     checkin error
+    status=1
   fi
   CUTOFF=$(date -d "90 days ago" +%Y-%m-%d)
   rclone lsf "spaces:$BUCKET/$PREFIX/snapshots/" --dirs-only 2>/dev/null | \
@@ -49,3 +53,4 @@ fi
   done
   echo "=== $(date -Is) backup done"
 } >> "$LOG" 2>&1
+exit $status

template/deploy/__SLUG__-rollup.sh

@@ -8,7 +8,9 @@
 #   Cron: /etc/cron.d/__SLUG__, installed by deploy.sh; SCHEDULE below
 #   must match it, because the Sentry monitor is told the same shape.
 #   Sentry Crons check-in: /root/.__SLUG__-sentry-cron-rollup holds the
-#   check-in URL (absent = no check-ins).
+#   check-in URL (absent = no check-ins). On a host the hall shares,
+#   hall-nightly.sh runs this and checks in once for every game, so the
+#   file is absent and the exit status is the word.
 set -u
 OUT="/var/lib/__SLUG__/rollup.jsonl"
 LOG="/var/log/__SLUG__/rollup.log"
@@ -78,4 +80,4 @@ with open(out + ".tmp", "w") as fh:
 os.replace(out + ".tmp", out)
 print("%s rolled: %d people, %d requests, %d rooms" % (day, len(t["addresses"]), t["human"], created))
 PY
-then checkin ok; else checkin error; fi
+then checkin ok; else checkin error; exit 1; fi

template/deploy/deploy.sh

@@ -43,7 +43,11 @@ ssh "root@$HOST" '
   install -m 755 /opt/__SLUG__/app/deploy/__SLUG__-rollup.sh /usr/local/bin/__SLUG__-rollup.sh
   install -m 755 /opt/__SLUG__/app/deploy/__SLUG__-backup.sh /usr/local/bin/__SLUG__-backup.sh
   install -m 644 -D /opt/__SLUG__/app/deploy/traffic.py /usr/local/lib/__SLUG__/traffic.py
-  install -m 644 /opt/__SLUG__/app/deploy/__SLUG__.cron /etc/cron.d/__SLUG__
+  # On a host the hall shares, its hall-nightly.sh runs the rollup and backup
+  # of every game and checks in with Sentry once for all; a game on a box of
+  # its own schedules its own.
+  if [ -f /etc/cron.d/gamehall ]; then rm -f /etc/cron.d/__SLUG__
+  else install -m 644 /opt/__SLUG__/app/deploy/__SLUG__.cron /etc/cron.d/__SLUG__; fi
   mkdir -p /var/log/__SLUG__
   systemctl daemon-reload
   systemctl enable --now __SLUG__