# Deploying __NAME__

Every game is a room on a small Node process that keeps it as an
append-only ledger of moves in /var/lib/__SLUG__/rooms. Production is the
shared Kestrel's Hall host: Caddy terminates TLS with automatic
certificates, serves the static build from /opt/__SLUG__/build, and proxies
/api and /ws to the game server on port __PORT__, which runs as the
`__SLUG__` user under systemd from /opt/__SLUG__/app.

## The shared host

The games of Kestrel's Hall share one droplet, `hall` (nyc3, s-1vcpu-1gb,
$6 a month, tag `kestrels-hall`, IP 159.203.115.252). Each game keeps its
own Caddy site file in /etc/caddy/sites/<slug>.caddy, its own port,
service user, /opt/<slug> and /var/lib/<slug> trees, access log
(/var/lib/caddy/<slug>-access.log), cron file and ops scripts;
/etc/caddy/Caddyfile only says `import sites/*.caddy`. A game server is one
small Node process capped at 300 MB by its unit file, and a room is a few
kilobytes of ledger, so the host carries several. Each game needs a port
of its own on the host: wizwar 8787, waving-hands 8788, hnefatafl 8789,
dip 8790.

## Current production

- Host: __IP__
- URLs: https://__DOMAIN__ (A record at Hover, where kestrelsnest.social's
  DNS lives) and https://__SLUG__.__IP__.sslip.io (always works, zero DNS).
- Everyday deploy: `deploy/deploy.sh __IP__`
- Players' reports: `deploy/pull-reports.sh` mirrors /var/lib/__SLUG__/feedback.jsonl
  and the screenshots to ~/Desktop/__SLUG__-reports with a digest;
  `deploy/report-reply.sh __IP__ <id> <status> "text"` answers one.

## Adding this game to the host

1. `scp deploy/setup-droplet.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> \
    "bash /root/setup-droplet.sh '__DOMAIN__, __SLUG__.<ip>.sslip.io'"`
   It installs Caddy and Node if the host lacks them, writes this game's
   site file, and touches nothing of another game's. Point the A record at
   the host first, or leave the real name out until it is; running it again
   with the new line is how a name is added later.
2. `deploy/deploy.sh <ip>`

A fresh host, when one is needed: `doctl compute droplet create hall --region nyc3 \
--size s-1vcpu-1gb --image ubuntu-24-04-x64 --ssh-keys <your-key-ids> \
--tag-name kestrels-hall --wait`, then step 1.

## Operations scripts on the droplet

`deploy.sh` installs these to /usr/local/bin, the `traffic.py` they share to
/usr/local/lib/__SLUG__, and the cron file to /etc/cron.d/__SLUG__ on every
deploy, so the live copies are the repo copies:

- `__SLUG__-visitors.sh [day]`: who is here now and who came that day.
- `__SLUG__-pulse.sh`: the weekly health check (service, errors, rollup
  trend, backup, box).
- `__SLUG__-rollup.sh [day]`: one JSON line per day of counts, run
  nightly at 00:12 UTC into /var/lib/__SLUG__/rollup.jsonl.
- `__SLUG__-backup.sh`: nightly at 07:23 UTC, mirrors /var/lib/__SLUG__
  to the `kestrel-wizwar-backups` Space under `__SLUG__/` (a current copy
  and dated snapshots kept 90 days). It needs rclone with the Spaces
  credentials in /root/.config/rclone/rclone.conf, copied by hand from the
  wizwar droplet; until then it logs "skipped".

Errors from the game server go to Sentry, project `__SLUG__` in the
locallygrownnet organisation; the DSN is in the unit file. The nightly rollup
and backup check in with Sentry Crons when /root/.__SLUG__-sentry-cron-rollup
and /root/.__SLUG__-sentry-cron hold their check-in URLs (the ingest
URL with the project's cron path and public key), so a missed night is noticed.

To have every new issue, regression and reappearance posted to Slack the way
wizwar's are, run `deploy/sentry-slack-alert.sh [#channel]` once from your own
shell with `SENTRY_TOKEN` (an org auth token with alerts:write) and
`SLACK_CHANNEL_ID` (the channel's Slack ID) set; the token never leaves the
shell.

Before every deploy, `deploy/verify-ledgers.sh <ip>` fetches every production
ledger and replays it with the local engine, comparing each room with what the
server shows. A ledger the new engine refuses or replays differently stops the
deploy: the server would rewrite that game on restart.

## Everyday deploys

    deploy/deploy.sh <ip>

Runs the type-checks, the tests and the build locally, rsyncs `build/` to
the droplet keeping the previous week's hashed assets, rsyncs the server and
engine sources, installs dependencies, and restarts the game server. A room
is replayed from its ledger when the server comes back, which takes a few
seconds; Caddy holds requests that land in the gap.

## Operations

- Logs: `ssh root@<ip> journalctl -u __SLUG__ -f` for the game server,
  `journalctl -u caddy -f` and /var/lib/caddy/access.log for the web side.
- Restart: `ssh root@<ip> systemctl restart __SLUG__`
- Rooms: `/var/lib/__SLUG__/rooms/<CODE>.jsonl`, one ledger per game; the
  nightly backup keeps them in the Space.