Waving Hands moves to the shared hall host

The kit's shared setup, its own Caddy site file and access log (the
visitors, rollup and pulse read it), the gate taking a SITE override,
and the host's address everywhere. The old droplet forwards to hall
until the A record moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jm2auWk6RP71CjaAb4FMoG
This commit is contained in:
Eric Wagoner
2026-09-25 17:04:35 -04:00
co-authored by Claude Opus 5.5
parent e1833e4e3c
commit f0d06db108
12 changed files with 142 additions and 133 deletions
+1 -1
View File
@@ -7,7 +7,7 @@ description: The weekly Waving Hands operations pulse — duel server health, er
One command on the droplet prints everything; read it as a short report. One command on the droplet prints everything; read it as a short report.
ssh root@159.203.98.71 waving-hands-pulse.sh ssh root@159.203.115.252 waving-hands-pulse.sh
- **duel server**: whether the service and Caddy are up, restarts and - **duel server**: whether the service and Caddy are up, restarts and
journal errors in the last seven days with the last three error lines. journal errors in the last seven days with the last three error lines.
+5 -5
View File
@@ -7,7 +7,7 @@ description: Work the Waving Hands reports desk — fetch players' bug reports,
Players file reports from the Report button in a room's masthead (seated Players file reports from the Report button in a room's masthead (seated
or from the gallery). Each lands in `/var/lib/waving-hands/feedback.jsonl` or from the gallery). Each lands in `/var/lib/waving-hands/feedback.jsonl`
on the droplet (159.203.98.71) pinned with `roomId`, `turn` (the turn on the droplet (159.203.115.252) pinned with `roomId`, `turn` (the turn
being written when it was filed) and `seq` (the ledger's length then), being written when it was filed) and `seq` (the ledger's length then),
enough to replay the duel to the moment. Replies live in the same file enough to replay the duel to the moment. Replies live in the same file
and appear under the report in the player's hall. This desk fetches, and appear under the report in the player's hall. This desk fetches,
@@ -34,7 +34,7 @@ Every report and every player's answer rings a Sentry issue (project
waving-hands, fingerprinted per report or per line); the desk's own waving-hands, fingerprinted per report or per line); the desk's own
replies ring nothing. replies ring nothing.
Fetch: `ssh root@159.203.98.71 'cat /var/lib/waving-hands/feedback.jsonl'` Fetch: `ssh root@159.203.115.252 'cat /var/lib/waving-hands/feedback.jsonl'`
## a) Fetch and b) display ## a) Fetch and b) display
@@ -47,7 +47,7 @@ to hear his players' voices; never compress their words into a table.
## c) Process an unanswered report ## c) Process an unanswered report
1. **Replay to the pin.** `scp root@159.203.98.71:/var/lib/waving-hands/rooms/<roomId>.jsonl <scratchpad>/` 1. **Replay to the pin.** `scp root@159.203.115.252:/var/lib/waving-hands/rooms/<roomId>.jsonl <scratchpad>/`
and replay it with the engine as `deploy/replay-ledgers.ts` does: and replay it with the engine as `deploy/replay-ledgers.ts` does:
`createGame(names, start.seed)` then `resolveTurn(state, line.inputs)` `createGame(names, start.seed)` then `resolveTurn(state, line.inputs)`
per turn line, printing `state.log` around the pinned turn. The per turn line, printing `state.log` around the pinned turn. The
@@ -70,8 +70,8 @@ to hear his players' voices; never compress their words into a table.
passing `rules` to `createGame`). Old ledgers then replay as their passing `rules` to `createGame`). Old ledgers then replay as their
players saw them; new duels get the fix. players saw them; new duels get the fix.
Either way add a test in `src/lib/game/engine.test.ts` pinning the Either way add a test in `src/lib/game/engine.test.ts` pinning the
corrected behaviour. Deploy with `deploy/deploy.sh 159.203.98.71`. corrected behaviour. Deploy with `deploy/deploy.sh 159.203.115.252`.
5. **Reply.** `bash deploy/report-reply.sh 159.203.98.71 <reportId> <status> "text"`. 5. **Reply.** `bash deploy/report-reply.sh 159.203.115.252 <reportId> <status> "text"`.
Pass the id as ONE clean argument. Write to the PLAYER: name what you Pass the id as ONE clean argument. Write to the PLAYER: name what you
replayed, cite the rule, and say plainly what was wrong or why nothing replayed, cite the rule, and say plainly what was wrong or why nothing
was. The desk's voice is warm and specific. was. The desk's voice is warm and specific.
@@ -10,7 +10,7 @@ host glancing over the hall.
## Gather ## Gather
ssh root@159.203.98.71 waving-hands-visitors.sh [YYYY-MM-DD] ssh root@159.203.115.252 waving-hands-visitors.sh [YYYY-MM-DD]
Default is today, UTC; Eric is in US Eastern, so an evening at his desk Default is today, UTC; Eric is in US Eastern, so an evening at his desk
spills into the next UTC day. Run yesterday too when the hour is early. spills into the next UTC day. Run yesterday too when the hour is early.
+20 -20
View File
@@ -1,42 +1,42 @@
__HOST__ __HOST__ {
root * /opt/waving-hands/build
encode gzip zstd
root * /opt/waving-hands/build header {
encode gzip zstd
header {
Strict-Transport-Security "max-age=31536000" Strict-Transport-Security "max-age=31536000"
X-Content-Type-Options "nosniff" X-Content-Type-Options "nosniff"
X-Frame-Options "DENY" X-Frame-Options "DENY"
Referrer-Policy "no-referrer" Referrer-Policy "no-referrer"
} }
log { log {
output file /var/lib/caddy/access.log { output file /var/lib/caddy/waving-hands-access.log {
roll_size 10MiB roll_size 10MiB
roll_keep 30 roll_keep 30
} }
} }
# The duel server answers the API and the websocket; a deploy restarts it for # The game server answers the API and the websocket; a deploy restarts it for
# a few seconds, and the proxy holds requests that land in that gap. # a few seconds, and the proxy holds requests that land in that gap.
@duel path /api/* /ws @game path /api/* /ws
handle @duel { handle @game {
reverse_proxy localhost:8788 { reverse_proxy localhost:8788 {
lb_try_duration 30s lb_try_duration 30s
lb_try_interval 250ms lb_try_interval 250ms
} }
} }
# Everything else is the static site. SvelteKit writes each route as # Everything else is the static site. SvelteKit writes each route as
# <route>.html, so /rules is tried as /rules.html before falling back to the # <route>.html, so /rules is tried as /rules.html before falling back to the
# app shell, which serves the rooms. Hashed assets under _app/immutable are # app shell, which serves the rooms. Hashed assets under _app/immutable are
# cached for a year; every other response is revalidated so a deploy shows # cached for a year; every other response is revalidated so a deploy shows
# up on the next load. The two header matchers are disjoint. # up on the next load.
handle { handle {
@immutable path /_app/immutable/* @immutable path /_app/immutable/*
header @immutable Cache-Control "public, max-age=31536000, immutable" header @immutable Cache-Control "public, max-age=31536000, immutable"
@mutable not path /_app/immutable/* @mutable not path /_app/immutable/*
header @mutable Cache-Control "no-cache" header @mutable Cache-Control "no-cache"
try_files {path} {path}.html /index.html try_files {path} {path}.html /index.html
file_server file_server
}
} }
+35 -36
View File
@@ -1,51 +1,50 @@
# Deploying Waving Hands # Deploying Waving Hands
The single-player game runs entirely in the browser. Duels between people Every game is a room on a small Node process that keeps it as an
go through a small Node process that keeps each room as an append-only append-only ledger of moves in /var/lib/waving-hands/rooms (the unit names it with WH_DATA_DIR). Production is the
ledger of moves in /var/lib/waving-hands/rooms. Production is one shared Kestrel's Hall host: Caddy terminates TLS with automatic
DigitalOcean droplet: Caddy terminates TLS with automatic certificates, certificates, serves the static build from /opt/waving-hands/build, and proxies
serves the static build from /opt/waving-hands/build, and proxies /api and /api and /ws to the game server on port 8788, which runs as the
/ws to the duel server on port 8788, which runs as the `waving-hands` user `waving-hands` user under systemd from /opt/waving-hands/app.
under systemd from /opt/waving-hands/app.
## Sizing ## The shared host
The smallest droplet, `s-1vcpu-512mb-10gb` ($4 a month), carries both Caddy The games of Kestrel's Hall share one droplet, `hall` (nyc3, s-1vcpu-1gb,
and the duel server comfortably: the server is one small Node process capped $6 a month, tag `kestrels-hall`, IP 159.203.115.252). Each game keeps its
at 300 MB by its unit file, and a room is a few kilobytes of ledger. own Caddy site file in /etc/caddy/sites/<slug>.caddy, its own port,
service user, /opt/<slug> and /var/lib/<slug> trees, access log
The zero-cost alternative is a second site block in an existing Caddy (/var/lib/caddy/<slug>-access.log), cron file and ops scripts;
server's configuration pointing at a second directory; the deploy script /etc/caddy/Caddyfile only says `import sites/*.caddy`. A game server is one
works unchanged against that host. A droplet of its own keeps this site's small Node process capped at 300 MB by its unit file, and a room is a few
uptime and upgrades independent of anything else. kilobytes of ledger, so the host carries several. Each game needs a port
of its own on the host: wizwar 8787, waving-hands 8788, hnefatafl 8789,
dip 8790.
## Current production ## Current production
- Droplet: `waving-hands` (nyc3, s-1vcpu-512mb-10gb, tag `waving-hands`), IP 159.203.98.71 - Host: 159.203.115.252
- URLs: https://hands.kestrelsnest.social (A record at Hover, where - URLs: https://hands.kestrelsnest.social (A record at Hover, where kestrelsnest.social's
kestrelsnest.social's DNS lives) and https://waving-hands.159.203.98.71.sslip.io DNS lives) and https://hands.159.203.115.252.sslip.io (always works, zero DNS).
(always works, zero DNS). Caddy serves both from the first line of - Everyday deploy: `SITE=https://hands.159.203.115.252.sslip.io deploy/deploy.sh 159.203.115.252`
/etc/caddy/Caddyfile. (this Mac's resolver caches the old *.kestrelsnest.social wildcard, so the gate reads the sslip name).
- Everyday deploy: `deploy/deploy.sh 159.203.98.71` - Moved here from its own droplet (159.203.98.71) on 2026-09-25.
- Players' reports: `deploy/pull-reports.sh` mirrors /var/lib/waving-hands/feedback.jsonl - Players' reports: `deploy/pull-reports.sh` mirrors /var/lib/waving-hands/feedback.jsonl
and the screenshots to ~/Desktop/waving-hands-reports with a digest; and the screenshots to ~/Desktop/waving-hands-reports with a digest;
`deploy/report-reply.sh 159.203.98.71 <id> <status> "text"` answers one. `deploy/report-reply.sh 159.203.115.252 <id> <status> "text"` answers one.
## New droplet from scratch ## Adding this game to the host
1. `doctl compute droplet create waving-hands --region nyc3 \ 1. `scp deploy/setup-droplet.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> \
--size s-1vcpu-512mb-10gb --image ubuntu-24-04-x64 \ "bash /root/setup-droplet.sh 'hands.kestrelsnest.social, hands.<ip>.sslip.io'"`
--ssh-keys <your-key-ids> --tag-name waving-hands --wait` It installs Caddy and Node if the host lacks them, writes this game's
2. `scp deploy/setup-droplet.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> \ site file, and touches nothing of another game's. Point the A record at
"bash /root/setup-droplet.sh 'hands.kestrelsnest.social, waving-hands.<ip>.sslip.io'"` the host first, or leave the real name out until it is; running it again
(point the A record at the new IP first, or leave the real name out until it is). with the new line is how a name is added later.
3. `scp deploy/setup-server.sh deploy/Caddyfile.tmpl root@<ip>:/root/ && ssh root@<ip> "bash /root/setup-server.sh"` 2. `deploy/deploy.sh <ip>`
4. `deploy/deploy.sh <ip>`
The sslip.io hostname works with no DNS at all. To add a real name, point an A fresh host, when one is needed: `doctl compute droplet create hall --region nyc3 \
A record at the droplet and add the name to the first line of --size s-1vcpu-1gb --image ubuntu-24-04-x64 --ssh-keys <your-key-ids> \
/etc/caddy/Caddyfile (space separated), then `systemctl reload caddy`; Caddy --tag-name kestrels-hall --wait`, then step 1.
fetches the certificate on first request.
## Operations scripts on the droplet ## Operations scripts on the droplet
+1 -1
View File
@@ -3,7 +3,7 @@
# to a local folder, and write a digest beside them for reading. # to a local folder, and write a digest beside them for reading.
# deploy/pull-reports.sh [host] [folder] (default ~/Desktop/waving-hands-reports) # deploy/pull-reports.sh [host] [folder] (default ~/Desktop/waving-hands-reports)
set -euo pipefail set -euo pipefail
HOST="${1:-159.203.98.71}" HOST="${1:-159.203.115.252}"
OUT="${2:-$HOME/Desktop/waving-hands-reports}" OUT="${2:-$HOME/Desktop/waving-hands-reports}"
mkdir -p "$OUT/images" mkdir -p "$OUT/images"
scp -q "root@$HOST:/var/lib/waving-hands/feedback.jsonl" "$OUT/feedback.jsonl" 2>/dev/null || : > "$OUT/feedback.jsonl" scp -q "root@$HOST:/var/lib/waving-hands/feedback.jsonl" "$OUT/feedback.jsonl" 2>/dev/null || : > "$OUT/feedback.jsonl"
+1 -1
View File
@@ -33,4 +33,4 @@ else
fi fi
echo "--- box" echo "--- box"
echo "disk: $(df -h / | awk 'NR==2{print $5" used of "$2}'); memory: $(free -m | awk 'NR==2{print $7" MB available of "$2}'); load: $(cut -d' ' -f1-3 /proc/loadavg)" echo "disk: $(df -h / | awk 'NR==2{print $5" used of "$2}'); memory: $(free -m | awk 'NR==2{print $7" MB available of "$2}'); load: $(cut -d' ' -f1-3 /proc/loadavg)"
echo "caddy access log: $(du -sh /var/lib/caddy/access.log 2>/dev/null | cut -f1) live, $(ls /var/lib/caddy/access*.log* 2>/dev/null | wc -l) files" echo "caddy access log: $(du -sh /var/lib/caddy/waving-hands-access.log 2>/dev/null | cut -f1) live, $(ls /var/lib/caddy/waving-hands-access*.log* 2>/dev/null | wc -l) files"
+56 -21
View File
@@ -1,34 +1,69 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# One-time droplet setup. Copy this script and Caddyfile.tmpl to the droplet # Prepare a host for Waving Hands. The host may be shared with other games of
# and run ON the droplet as root: # Kestrel's Hall: each keeps its own Caddy site file in /etc/caddy/sites, its
# bash setup-droplet.sh 'hands.kestrelsnest.social, waving-hands.<droplet-ip>.sslip.io' # own port, service user, data directory and access log, and this script
# The argument is the Caddy site address line: one name, or several # touches nothing that belongs to another game. Copy it and Caddyfile.tmpl
# separated by commas. Every name must already resolve to this droplet. # to the host and run ON it as root; safe to run again.
# Waving Hands is a static site: Caddy serves the built files and terminates # bash setup-droplet.sh 'hands.kestrelsnest.social, waving-hands.<host-ip>.sslip.io'
# TLS. There is no application process to install or supervise. # The argument is this game's Caddy site address line: one name, or several
# separated by commas. Every name must already resolve to this host.
set -euo pipefail set -euo pipefail
HOST="${1:?usage: setup-droplet.sh <hostname>}" HOST="${1:?usage: setup-droplet.sh <site address line>}"
apt-get update -q apt-get update -q
apt-get install -qy curl rsync apt-get install -qy curl rsync
# Caddy (auto-HTTPS) # Caddy (auto-HTTPS), once per host.
apt-get install -qy debian-keyring debian-archive-keyring apt-transport-https if ! command -v caddy >/dev/null; then
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \ apt-get install -qy debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \
| gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \
| tee /etc/apt/sources.list.d/caddy-stable.list | tee /etc/apt/sources.list.d/caddy-stable.list
apt-get update -q && apt-get install -qy caddy apt-get update -q && apt-get install -qy caddy
fi
# Site directory, readable by Caddy. # Node 22, once per host; every game runs its server on it.
mkdir -p /opt/waving-hands/build if ! command -v node >/dev/null || [[ "$(node -v)" != v22* ]]; then
chown -R root:caddy /opt/waving-hands curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
chmod -R g+rX /opt/waving-hands apt-get install -qy nodejs
fi
# The host's Caddyfile only gathers the games' site files. A Caddyfile that
# is neither Caddy's stock one nor this is a single-game host from before the
# shared layout: stop rather than overwrite it.
ROOT_CADDY='# Kestrel'"'"'s Hall: every game on this host brings its own site file.
import sites/*.caddy'
mkdir -p /etc/caddy/sites
if ! grep -q '^import sites/\*.caddy' /etc/caddy/Caddyfile 2>/dev/null; then
if [ -s /etc/caddy/Caddyfile ] && ! grep -q 'root \* /usr/share/caddy' /etc/caddy/Caddyfile; then
echo "/etc/caddy/Caddyfile holds another site; move it into /etc/caddy/sites/ first" >&2
exit 1
fi
printf '%s\n' "$ROOT_CADDY" > /etc/caddy/Caddyfile
fi
# This game: its user, its directories, its site.
# The service user shares the slug. A slug that is already a system group
# (Ubuntu ships "dip", "games", "audio" and others) cannot also be a user:
# stop and say so, and choose another user name in this script and the unit.
if ! id -u waving-hands &>/dev/null; then
if getent group waving-hands >/dev/null; then
echo "a system group is already called waving-hands; pick another service user name" >&2
exit 1
fi
useradd -r -m -d /opt/waving-hands-home waving-hands
fi
mkdir -p /opt/waving-hands/build /opt/waving-hands/app /var/lib/waving-hands/rooms /var/log/waving-hands
chown -R root:caddy /opt/waving-hands/build
chmod -R g+rX /opt/waving-hands/build
chown -R waving-hands:waving-hands /var/lib/waving-hands
sed "s|__HOST__|$HOST|" "$(dirname "$0")/Caddyfile.tmpl" > /etc/caddy/sites/waving-hands.caddy
# Caddy: the site and the duel server's paths, from the template beside this script.
sed "s|__HOST__|$HOST|" "$(dirname "$0")/Caddyfile.tmpl" > /etc/caddy/Caddyfile
caddy validate --config /etc/caddy/Caddyfile caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy # Validation as root touches the access logs; Caddy runs as caddy and must own them.
chown -R caddy:caddy /var/lib/caddy
systemctl reload-or-restart caddy
# Firewall: ssh + web only. # Firewall: ssh + web only.
ufw allow OpenSSH ufw allow OpenSSH
@@ -36,4 +71,4 @@ ufw allow 80/tcp
ufw allow 443/tcp ufw allow 443/tcp
ufw --force enable ufw --force enable
echo "droplet ready: now run deploy/deploy.sh <ip> from your machine" echo "host ready for waving-hands: now run deploy/deploy.sh <ip> from your machine"
-25
View File
@@ -1,25 +0,0 @@
#!/usr/bin/env bash
# Adds the duel server to a droplet that setup-droplet.sh already prepared.
# Copy this script and Caddyfile.tmpl to the droplet and run ON the droplet
# as root; safe to run again. Installs Node 22, creates the service user and
# data directory, and teaches Caddy to hand /api and /ws to the server while
# it keeps serving the static site itself.
set -euo pipefail
if ! command -v node >/dev/null || [[ "$(node -v)" != v22* ]]; then
curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt-get install -qy nodejs
fi
id -u waving-hands &>/dev/null || useradd -r -m -d /opt/waving-hands-home waving-hands
mkdir -p /opt/waving-hands/app /var/lib/waving-hands/rooms
chown -R waving-hands:waving-hands /var/lib/waving-hands
# Caddy: rewrite the site's configuration from the template, keeping its
# address line, so the duel server's paths are routed before the files.
HOST_LINE=$(head -1 /etc/caddy/Caddyfile)
sed "s|__HOST__|$HOST_LINE|" "$(dirname "$0")/Caddyfile.tmpl" > /etc/caddy/Caddyfile
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
echo "server prerequisites ready: now run deploy/deploy.sh <ip> from your machine"
+1 -1
View File
@@ -6,7 +6,7 @@
# deploy/verify-ledgers.sh <droplet-ip-or-host> [https://site] # deploy/verify-ledgers.sh <droplet-ip-or-host> [https://site]
set -euo pipefail set -euo pipefail
HOST="${1:?usage: verify-ledgers.sh <droplet-ip-or-host> [https://site]}" HOST="${1:?usage: verify-ledgers.sh <droplet-ip-or-host> [https://site]}"
SITE="${2:-https://hands.kestrelsnest.social}" SITE="${2:-${SITE:-https://hands.kestrelsnest.social}}"
DIR="$(mktemp -d)" DIR="$(mktemp -d)"
trap 'rm -rf "$DIR"' EXIT trap 'rm -rf "$DIR"' EXIT
rsync -az "root@$HOST:/var/lib/waving-hands/rooms/" "$DIR/" rsync -az "root@$HOST:/var/lib/waving-hands/rooms/" "$DIR/"
+1 -1
View File
@@ -30,7 +30,7 @@ PAGES = {"/": "hall", "/play": "play", "/rules": "rules"}
addresses, bots = set(), set() addresses, bots = set(), set()
pages, joins, referrers, campaigns, agents = Counter(), Counter(), Counter(), Counter(), Counter() pages, joins, referrers, campaigns, agents = Counter(), Counter(), Counter(), Counter(), Counter()
requests = 0 requests = 0
for path in sorted(glob.glob("/var/lib/caddy/access.log*")): for path in sorted(glob.glob("/var/lib/caddy/waving-hands-access*.log*")):
try: try:
with open(path) as f: with open(path) as f:
for line in f: for line in f:
+1 -1
View File
@@ -34,7 +34,7 @@ req = human = bot = 0
hips, bips = set(), set() hips, bips = set(), set()
pages, joins, refs, campaigns = collections.Counter(), collections.Counter(), collections.Counter(), collections.Counter() pages, joins, refs, campaigns = collections.Counter(), collections.Counter(), collections.Counter(), collections.Counter()
seen_campaign = set() seen_campaign = set()
for f in sorted(glob.glob("/var/lib/caddy/access*.log*")): for f in sorted(glob.glob("/var/lib/caddy/waving-hands-access*.log*")):
if os.path.getmtime(f) < t0: continue if os.path.getmtime(f) < t0: continue
opener = gzip.open if f.endswith(".gz") else open opener = gzip.open if f.endswith(".gz") else open
with opener(f, "rt", errors="ignore") as fh: with opener(f, "rt", errors="ignore") as fh: