Eric WagonerandClaude Fable 5
0358079da4
The Anthropic key was the last hard gate for other users. extract's
VisionFn seam gains a second factory speaking the chat-completions
format — OpenAI, OpenRouter, or a local runtime (Ollama, LM Studio,
llama.cpp, vLLM) via config.toml: vision_provider, vision_base_url,
and vision_key_env ("" = keyless local endpoint, no Authorization
header sent). Anthropic stays the default. load_config rejects unknown
providers loudly, the pipeline page's credentials warning follows the
configured provider (a keyless local endpoint warns about nothing),
and config.toml + README document the local-model trade honestly:
weaker spine reading means a longer proofread pass, which the shaky-
read workflow absorbs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
181 lines
15 KiB
Markdown
181 lines
15 KiB
Markdown
# bggpipe — Shelf-to-BoardGameGeek Collection Pipeline
|
|
|
|
<img src="assets/logo-full.jpeg" alt="the bggpipe piper — a bagpiper whose bag is a board game box" width="220" align="right">
|
|
|
|
Photograph your board game shelves. End up with your whole collection — including which *edition* of each game you own — cataloged on [BoardGameGeek](https://boardgamegeek.com).
|
|
|
|
```
|
|
photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
|
|
→ [3 review] → matches.csv (approved) → [4 diff] → to_add.csv
|
|
→ [5 upload] → upload_log.csv
|
|
→ [6 enrich] → games.json
|
|
```
|
|
|
|
> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
|
|
|
|
## Why this exists
|
|
|
|
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done.
|
|
|
|
## How it works
|
|
|
|
1. **extract** — Shelf photos go to the Anthropic API (Claude vision), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
|
|
2. **resolve** — Titles are matched to BGG game IDs via the [XML API2](https://boardgamegeek.com/wiki/page/BGG_XML_API2) (exact + fuzzy matching, popularity tiebreaks), then edition cues are matched against BGG's version list for each game. Anything uncertain is flagged rather than guessed.
|
|
3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
|
|
4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
|
|
5. **upload** — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
|
|
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in `data/games.json`, the seed data for a future web frontend.
|
|
|
|
Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit.
|
|
|
|
## What it looks like
|
|
|
|
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) are banners, not surprises.
|
|
|
|
|
|
|
|
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
|
|
|
|
|
|
|
|
**Titles** — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games.
|
|
|
|
|
|
|
|
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto). Here: everything resolvable is reviewed, 106 titles parked awaiting the API token.
|
|
|
|
|
|
|
|
**Queue** — exactly what upload will do before it does it, and a permanent log of every attempt ever made.
|
|
|
|
|
|
|
|
**Library** — the enriched collection browser (board games and RPGs), waiting for real BGG data to fill its shelves.
|
|
|
|
|
|
|
|
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
|
|
|
|
|
|
|
|
**And on a phone** (`--lan`) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.
|
|
|
|
<p>
|
|
<img src="docs/screenshots/08-phone-menu.jpeg" alt="Phone view: the hamburger menu open over the Photos page, showing all seven pages with attention badges" width="24%">
|
|
<img src="docs/screenshots/09-phone-camera.jpeg" alt="Phone view: tapping the photo drop zone offers iOS's Photo Library / Take Photo / Choose Files sheet" width="24%">
|
|
<img src="docs/screenshots/10-phone-titles.jpeg" alt="Phone view: the Titles page as stacked cards with status chips, edit and split buttons, and the shaky-reads filter" width="24%">
|
|
<img src="docs/screenshots/11-phone-piper.jpeg" alt="Phone view: the Help page's credits card with Juniper's full piper artwork and the BGG trademark attribution" width="24%">
|
|
</p>
|
|
|
|
## Requirements
|
|
|
|
- macOS or Linux, Python 3.12+, [uv](https://docs.astral.sh/uv/)
|
|
- A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint via `config.toml` (`vision_provider = "openai-compatible"` + `vision_base_url`): OpenAI, OpenRouter, or a free local runtime like [Ollama](https://ollama.com/) with a vision-capable model. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline.
|
|
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared.
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
git clone https://git.kestrelsnest.social/eric/bggpipe.git
|
|
cd bggpipe
|
|
uv sync # installs Python deps
|
|
uv run bggpipe init # guided setup: folders, credentials, browser download
|
|
```
|
|
|
|
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? `cp .env.example .env`, fill it in, and run `uv run playwright install chromium` yourself.
|
|
|
|
Secrets live in environment variables only, never in config files, code, or logs. `.env` is gitignored. If you use [direnv](https://direnv.net/), the committed `.envrc` loads `.env` automatically after a one-time `direnv allow`; otherwise export the variables yourself (e.g. `set -a; source .env; set +a`).
|
|
|
|
| Variable | Used by | What it is |
|
|
|---|---|---|
|
|
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
|
|
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
|
|
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
|
|
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
|
|
|
|
Non-secret knobs (`photos_dir`, `data_dir`, the vision model, the rate limit) live in `config.toml`. From here you can drive everything from the browser:
|
|
|
|
```sh
|
|
uv run bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
|
|
```
|
|
|
|
### From your phone
|
|
|
|
The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:
|
|
|
|
```sh
|
|
uv run bggpipe web --lan # localhost + your network, behind an access key
|
|
```
|
|
|
|
Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (`data/.lan_key`; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.
|
|
|
|
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
|
|
|
|
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
|
|
|
```sh
|
|
uv run bggpipe extract # photos → titles.json (+ retake prompts)
|
|
uv run bggpipe resolve # titles → BGG ids/versions in matches.csv
|
|
uv run bggpipe review --web # review UI only
|
|
uv run bggpipe diff # compare against your BGG collection
|
|
uv run bggpipe upload --dry-run # ALWAYS inspect this first
|
|
uv run bggpipe upload --limit 1 # then one game, then small batches
|
|
uv run bggpipe enrich # full metadata → data/games.json
|
|
```
|
|
|
|
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. `review` without `--web` runs in the terminal instead. `upload` also has `--retry-failed`, `--verify` (re-fetches your collection and cross-checks the log), and runs a **headed** browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
|
|
|
|
### RPGs on your shelves
|
|
|
|
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them.
|
|
|
|
### Fixing what the model gets wrong
|
|
|
|
Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, and **remove** lines that aren't games at all. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store. The in-app **Help** page documents all of it.
|
|
|
|
### Taking good shelf photos
|
|
|
|
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again.
|
|
|
|
## Bring your own shelves
|
|
|
|
This repo doubles as its author's live pipeline, so `data/` ships with his real artifacts — extracted titles, matches, and 2018 collection snapshots. Before running against *your* shelves, clear the data:
|
|
|
|
```sh
|
|
rm data/*.csv data/*.json data/*.xml data/STUB_DATA.marker
|
|
rm -rf data/bgg_cache data/extract_raw
|
|
```
|
|
|
|
Two of those files deserve a word:
|
|
|
|
- **`data/STUB_DATA.marker`** — the committed CSVs were resolved from *hand-written stub fixtures* (the author's BGG application is still awaiting approval), so every version id in them is a synthetic placeholder. The upload stage refuses to run while this marker exists, precisely so nobody (including a fresh clone) can push placeholder data to a real BGG account. Starting fresh with your own token, you'll never see it again.
|
|
- **`data/collection_snapshot_*.xml`** — with `BGG_API_TOKEN` set, `diff` fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
|
|
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
|
|
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
|
|
|
|
No token yet? `extract` works immediately (it only needs the Anthropic key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
|
|
|
|
## Development
|
|
|
|
```sh
|
|
uv run pytest # offline test suite (stub fixtures, no network)
|
|
uv run pytest --run-live # + a read-only live-API smoke test (needs token)
|
|
uv run ruff check src tests # lint
|
|
uv run bggpipe review --web --dev # review UI with code hot-reload
|
|
```
|
|
|
|
The review UI live-follows the data files — run `extract` or `resolve` in another terminal and the page updates itself. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
|
|
|
|
## A note on being a good BGG citizen
|
|
|
|
This tool is **not affiliated with or supported by BoardGameGeek**. It uses only the sanctioned XML API2 for reads (with your own registered application token, per BGG's current policy) and drives the regular website for writes, deliberately slowly (one request every couple of seconds, slower for uploads). Please keep it that way: BGG is a community resource running on community goodwill. You are responsible for your own account — review the dry-run output before a real upload.
|
|
|
|
## License
|
|
|
|
MIT — see [LICENSE](LICENSE).
|
|
|
|
BoardGameGeek and BGG are trademarks of BoardGameGeek, LLC. bggpipe is an
|
|
independent project, not affiliated with or endorsed by BoardGameGeek.
|
|
|
|
Mascot art by Juniper, used with pride.
|