Sentry: errors from the duel server, and cron check-ins for the rollup and backup

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141G6xqLeNRYEtviLWSB5Up

deploy/README.md

@@ -60,6 +60,12 @@ fetches the certificate on first request.
   credentials in /root/.config/rclone/rclone.conf, copied by hand from the
   wizwar droplet; until then it logs "skipped".
 
+Errors from the duel server go to Sentry, project `waving-hands` in the
+locallygrownnet organisation; the DSN is in the unit file. The nightly rollup
+and backup check in with Sentry Crons when /root/.waving-hands-sentry-cron-rollup
+and /root/.waving-hands-sentry-cron hold their check-in URLs (the ingest
+URL with the project's cron path and public key), so a missed night is noticed.
+
 Before every deploy, `deploy/verify-ledgers.sh <ip>` fetches every production
 ledger and replays it with the local engine, comparing each room with what the
 server shows. A ledger the new engine refuses or replays differently stops the

deploy/waving-hands-backup.sh

@@ -4,8 +4,9 @@
 #   snapshots/  - one dated copy per day, pruned after 90 days
 # Ledgers are append-only JSONL; the duel server never needs stopping.
 # Sentry Crons check-in: /root/.waving-hands-sentry-cron holds the check-in
-# URL (absent file = no check-ins).
+# URL (absent file = no check-ins). SCHEDULE must match /etc/cron.d/waving-hands.
 set -u
+SCHEDULE="23 7 * * *"
 BUCKET="kestrel-wizwar-backups"
 PREFIX="waving-hands"
 SRC="/var/lib/waving-hands"
@@ -13,7 +14,11 @@ LOG="/var/log/waving-hands/backup.log"
 STAMP=$(date +%Y-%m-%d)
 mkdir -p "$(dirname "$LOG")"
 CRON_URL=$(cat /root/.waving-hands-sentry-cron 2>/dev/null || true)
-checkin() { [ -n "$CRON_URL" ] && curl -sf -o /dev/null "$CRON_URL?status=$1" || true; }
+checkin() {
+  [ -n "$CRON_URL" ] && curl -sf -o /dev/null -X POST -H "Content-Type: application/json" \
+    -d "{\"status\":\"$1\",\"monitor_config\":{\"schedule\":{\"type\":\"crontab\",\"value\":\"$SCHEDULE\"},\"checkin_margin\":30,\"max_runtime\":10,\"timezone\":\"UTC\"}}" \
+    "$CRON_URL" || true
+}
 
 if [ ! -s /root/.config/rclone/rclone.conf ]; then
   echo "$(date -Is) skipped: rclone is not configured" >> "$LOG"

deploy/waving-hands-rollup.sh

@@ -5,13 +5,25 @@
 # started and finished, and the box's vitals. Counts only: no addresses are
 # kept. The pulse reads it for trends the self-rotating access log loses.
 #   waving-hands-rollup.sh [YYYY-MM-DD]   (default: yesterday, UTC)
-#   Cron: /etc/cron.d/waving-hands, installed by deploy.sh.
+#   Cron: /etc/cron.d/waving-hands, installed by deploy.sh; SCHEDULE below
+#   must match it, because the Sentry monitor is told the same shape.
+#   Sentry Crons check-in: /root/.waving-hands-sentry-cron-rollup holds the
+#   check-in URL (absent = no check-ins).
 set -u
 OUT="/var/lib/waving-hands/rollup.jsonl"
 LOG="/var/log/waving-hands/rollup.log"
 DAY="${1:-$(date -u -d 'yesterday' +%Y-%m-%d)}"
+SCHEDULE="12 0 * * *"
+CRON_URL=$(cat /root/.waving-hands-sentry-cron-rollup 2>/dev/null || true)
+# The check-in carries the schedule: Sentry keeps a monitor only once told its shape.
+checkin() {
+  [ -n "$CRON_URL" ] && curl -sf -o /dev/null -X POST -H "Content-Type: application/json" \
+    -d "{\"status\":\"$1\",\"monitor_config\":{\"schedule\":{\"type\":\"crontab\",\"value\":\"$SCHEDULE\"},\"checkin_margin\":30,\"max_runtime\":10,\"timezone\":\"UTC\"}}" \
+    "$CRON_URL" || true
+}
 mkdir -p "$(dirname "$LOG")"
-python3 - "$DAY" "$OUT" <<'PY' >> "$LOG" 2>&1
+checkin in_progress
+if python3 - "$DAY" "$OUT" <<'PY' >> "$LOG" 2>&1
 import collections, datetime, glob, gzip, json, os, re, shutil, sys
 day, out = sys.argv[1], sys.argv[2]
 d0 = datetime.datetime.fromisoformat(day).replace(tzinfo=datetime.timezone.utc)
@@ -94,3 +106,4 @@ with open(out + ".tmp", "w") as fh:
 os.replace(out + ".tmp", out)
 print("%s rolled: %d people, %d requests, %d rooms" % (day, len(hips), human, created))
 PY
+then checkin ok; else checkin error; fi

deploy/waving-hands.service

@@ -10,6 +10,7 @@ Environment=PORT=8788
 # Caddy terminates TLS; the plaintext port must not face the internet.
 Environment=HOST=127.0.0.1
 Environment=WH_DATA_DIR=/var/lib/waving-hands/rooms
+Environment=SENTRY_DSN=https://e11c1f168789d9d7706bfc5eea4f373e@o4509525984149504.ingest.us.sentry.io/4512133232787456
 ExecStart=/opt/waving-hands/app/server/node_modules/.bin/tsx src/index.ts
 Restart=always
 RestartSec=3