waving-hands/deploy/README.md

# Deploying Waving Hands

Every game is a room on a small Node process that keeps it as an
append-only ledger of moves in /var/lib/waving-hands/rooms (the unit names it with WH_DATA_DIR). Production is the
shared Kestrel's Hall host: Caddy terminates TLS with automatic
certificates, serves the static build from /opt/waving-hands/build, and proxies
/api and /ws to the game server on port 8788, which runs as the
`waving-hands` user under systemd from /opt/waving-hands/app.

## The shared host

The games of Kestrel's Hall share one droplet, `hall` (nyc3, s-1vcpu-1gb,
$6 a month, tag `kestrels-hall`, IP 159.203.115.252). Each game keeps its
own Caddy site file in /etc/caddy/sites/<slug>.caddy, its own port,
service user, /opt/<slug> and /var/lib/<slug> trees, access log
(/var/lib/caddy/<slug>-access.log), cron file and ops scripts;
/etc/caddy/Caddyfile only says `import sites/*.caddy`. A game server is one
small Node process capped at 300 MB by its unit file, and a room is a few
kilobytes of ledger, so the host carries several. Each game needs a port
of its own on the host: wizwar 8787, waving-hands 8788, hnefatafl 8789,
dip 8790.

## Current production

- Host: 159.203.115.252
- URLs: https://hands.kestrelsnest.social (A record at Hover, where kestrelsnest.social's
  DNS lives) and https://hands.159.203.115.252.sslip.io (always works, zero DNS).
- Everyday deploy: `SITE=https://hands.159.203.115.252.sslip.io deploy/deploy.sh 159.203.115.252`
  (this Mac's resolver caches the old *.kestrelsnest.social wildcard, so the gate reads the sslip name).
- Moved here from its own droplet (159.203.98.71) on 2026-09-25.
- Players' reports: `deploy/pull-reports.sh` mirrors /var/lib/waving-hands/feedback.jsonl
  and the screenshots to ~/Desktop/waving-hands-reports with a digest;
  `deploy/report-reply.sh 159.203.115.252 <id> <status> "text"` answers one.

## Adding this game to the host

1. `scp deploy/setup-droplet.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> \
    "bash /root/setup-droplet.sh 'hands.kestrelsnest.social, hands.<ip>.sslip.io'"`
   It installs Caddy and Node if the host lacks them, writes this game's
   site file, and touches nothing of another game's. Point the A record at
   the host first, or leave the real name out until it is; running it again
   with the new line is how a name is added later.
2. `deploy/deploy.sh <ip>`

A fresh host, when one is needed: `doctl compute droplet create hall --region nyc3 \
--size s-1vcpu-1gb --image ubuntu-24-04-x64 --ssh-keys <your-key-ids> \
--tag-name kestrels-hall --wait`, then step 1.

## Operations scripts on the droplet

`deploy.sh` installs these to /usr/local/bin and the cron file to
/etc/cron.d/waving-hands on every deploy, so the live copies are the repo copies:

- `waving-hands-visitors.sh [day]`: who is here now and who came that day.
- `waving-hands-pulse.sh`: the weekly health check (service, errors, rollup
  trend, backup, box).
- `waving-hands-rollup.sh [day]`: one JSON line per day of counts, run
  nightly at 00:12 UTC into /var/lib/waving-hands/rollup.jsonl.
- `waving-hands-backup.sh`: nightly at 07:23 UTC, mirrors /var/lib/waving-hands
  to the `kestrel-wizwar-backups` Space under `waving-hands/` (a current copy
  and dated snapshots kept 90 days). It needs rclone with the Spaces
  credentials in /root/.config/rclone/rclone.conf, copied by hand from the
  wizwar droplet; until then it logs "skipped".

Errors from the duel server go to Sentry, project `waving-hands` in the
locallygrownnet organisation; the DSN is in the unit file. The nightly rollup
and backup check in with Sentry Crons when /root/.waving-hands-sentry-cron-rollup
and /root/.waving-hands-sentry-cron hold their check-in URLs (the ingest
URL with the project's cron path and public key), so a missed night is noticed.

To have every new issue, regression and reappearance posted to Slack the way
wizwar's are, run `deploy/sentry-slack-alert.sh [#channel]` once from your own
shell with `SENTRY_TOKEN` (an org auth token with alerts:write) and
`SLACK_CHANNEL_ID` set; the token never leaves the shell.

Before every deploy, `deploy/verify-ledgers.sh <ip>` fetches every production
ledger and replays it with the local engine, comparing each room with what the
server shows. A ledger the new engine refuses or replays differently stops the
deploy: the server would rewrite that duel on restart.

## Everyday deploys

    deploy/deploy.sh <ip>

Runs the type-checks, the tests and the build locally, rsyncs `build/` to
the droplet keeping the previous week's hashed assets, rsyncs the server and
engine sources, installs dependencies, and restarts the duel server. A
single-player game is in the player's own browser and loses nothing. A room
is replayed from its ledger when the server comes back, which takes a few
seconds; Caddy holds requests that land in the gap.

## Operations

- Logs: `ssh root@<ip> journalctl -u waving-hands -f` for the duel server,
  `journalctl -u caddy -f` and /var/lib/caddy/access.log for the web side.
- Restart: `ssh root@<ip> systemctl restart waving-hands`
- Who has been playing: `ssh root@<ip> waving-hands-visitors.sh [YYYY-MM-DD]`
  prints today's visitors from Caddy's log (people and bots apart, by page),
  every room opened today with how far it got, and the names seated, marking
  those seen for the first time.
- Rooms: `/var/lib/waving-hands/rooms/<CODE>.jsonl`, one ledger per duel.
  Copy that directory to back them up; single-player games are in players'
  browsers.