# Deploying __NAME__

Every game, against a housecarl or a friend, is a room on a small Node
process that keeps it as an append-only ledger of moves in
/var/lib/__SLUG__/rooms. Production is one DigitalOcean droplet: Caddy
terminates TLS with automatic certificates, serves the static build from
/opt/__SLUG__/build, and proxies /api and /ws to the game server on port
__PORT__, which runs as the `__SLUG__` user under systemd from
/opt/__SLUG__/app.

## Sizing

The smallest droplet, `s-1vcpu-512mb-10gb` ($4 a month), carries both Caddy
and the game server comfortably: the server is one small Node process capped
at 300 MB by its unit file, and a room is a few kilobytes of ledger.

## Current production

- Droplet: `__SLUG__` (nyc3, s-1vcpu-512mb-10gb, tag `__SLUG__`), IP __IP__
- URLs: https://__DOMAIN__ (A record at Hover, where kestrelsnest.social's
  DNS lives) and https://__SLUG__.__IP__.sslip.io (always works, zero DNS).
- Everyday deploy: `deploy/deploy.sh __IP__`
- Players' reports: `deploy/pull-reports.sh` mirrors /var/lib/__SLUG__/feedback.jsonl
  and the screenshots to ~/Desktop/__SLUG__-reports with a digest;
  `deploy/report-reply.sh __IP__ <id> <status> "text"` answers one.

## New droplet from scratch

1. `doctl compute droplet create __SLUG__ --region nyc3 \
    --size s-1vcpu-512mb-10gb --image ubuntu-24-04-x64 \
    --ssh-keys <your-key-ids> --tag-name __SLUG__ --wait`
2. `scp deploy/setup-droplet.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> \
    "bash /root/setup-droplet.sh '__DOMAIN__, __SLUG__.<ip>.sslip.io'"`
   (point the A record at the new IP first, or leave the real name out until it is).
3. `scp deploy/setup-server.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> "bash /root/setup-server.sh"`
4. `deploy/deploy.sh <ip>`

The sslip.io hostname works with no DNS at all. To add a real name, point an
A record at the droplet and add the name to the first line of
/etc/caddy/Caddyfile (space separated), then `systemctl reload caddy`; Caddy
fetches the certificate on first request.

## Operations scripts on the droplet

`deploy.sh` installs these to /usr/local/bin, the `traffic.py` they share to
/usr/local/lib/__SLUG__, and the cron file to /etc/cron.d/__SLUG__ on every
deploy, so the live copies are the repo copies:

- `__SLUG__-visitors.sh [day]`: who is here now and who came that day.
- `__SLUG__-pulse.sh`: the weekly health check (service, errors, rollup
  trend, backup, box).
- `__SLUG__-rollup.sh [day]`: one JSON line per day of counts, run
  nightly at 00:12 UTC into /var/lib/__SLUG__/rollup.jsonl.
- `__SLUG__-backup.sh`: nightly at 07:23 UTC, mirrors /var/lib/__SLUG__
  to the `kestrel-wizwar-backups` Space under `__SLUG__/` (a current copy
  and dated snapshots kept 90 days). It needs rclone with the Spaces
  credentials in /root/.config/rclone/rclone.conf, copied by hand from the
  wizwar droplet; until then it logs "skipped".

Errors from the game server go to Sentry, project `__SLUG__` in the
locallygrownnet organisation; the DSN is in the unit file. The nightly rollup
and backup check in with Sentry Crons when /root/.__SLUG__-sentry-cron-rollup
and /root/.__SLUG__-sentry-cron hold their check-in URLs (the ingest
URL with the project's cron path and public key), so a missed night is noticed.

To have every new issue, regression and reappearance posted to Slack the way
wizwar's are, run `deploy/sentry-slack-alert.sh [#channel]` once from your own
shell with `SENTRY_TOKEN` (an org auth token with alerts:write) and
`SLACK_CHANNEL_ID` (the channel's Slack ID) set; the token never leaves the
shell.

Before every deploy, `deploy/verify-ledgers.sh <ip>` fetches every production
ledger and replays it with the local engine, comparing each room with what the
server shows. A ledger the new engine refuses or replays differently stops the
deploy: the server would rewrite that game on restart.

## Everyday deploys

    deploy/deploy.sh <ip>

Runs the type-checks, the tests and the build locally, rsyncs `build/` to
the droplet keeping the previous week's hashed assets, rsyncs the server and
engine sources, installs dependencies, and restarts the game server. A room
is replayed from its ledger when the server comes back, which takes a few
seconds; Caddy holds requests that land in the gap.

## Operations

- Logs: `ssh root@<ip> journalctl -u __SLUG__ -f` for the game server,
  `journalctl -u caddy -f` and /var/lib/caddy/access.log for the web side.
- Restart: `ssh root@<ip> systemctl restart __SLUG__`
- Rooms: `/var/lib/__SLUG__/rooms/<CODE>.jsonl`, one ledger per game; the
  nightly backup keeps them in the Space.