Eric Wagoner
74fc4fe847
When the board-game search (and truncation heads) runs dry, resolve falls back to type=rpgitem — the geekdo database is shared, so the same API, token, cache, and classification machinery apply. Matched rpgitems flow through review and enrich normally but diff routes them to a local_only bucket, structurally outside to_add/to_update: their collections live on RPGGeek, beyond this pipeline's write scope. The library page gains an All/Board games/RPGs filter and an "RPG · local only" badge; the catalog tags them too. Fixture generators write blanket empty rpgitem stubs for every known query (the fallback fires for every unmatched title) with real synthetic entries for Alice Is Missing. Data: both Alice rows re-resolved from unmatched to auto rpgitem matches. First diff since the audit reworks also lands their real-data consequences: Dungeon! gains its TSR edition update on a versionless copy the old claim ordering missed, to_add rows carry unioned reshoot provenance, and the Herbaceous typo row's survivor is now the correctly-spelled title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
10 KiB
bggpipe — Shelf-to-BoardGameGeek Collection Pipeline
Photograph your board game shelves. End up with your whole collection — including which edition of each game you own — cataloged on BoardGameGeek.
photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
→ [3 review] → matches.csv (approved) → [4 diff] → to_add.csv
→ [5 upload] → upload_log.csv
→ [6 enrich] → games.json
Status: working, not yet battle-tested. All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with
--dry-run, then--limit 1. (This repo also carries its author's in-progress pipeline data; see Bring your own shelves.)
Why this exists
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done.
How it works
- extract — Shelf photos go to the Anthropic API (Claude vision), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
- resolve — Titles are matched to BGG game IDs via the XML API2 (exact + fuzzy matching, popularity tiebreaks), then edition cues are matched against BGG's version list for each game. Anything uncertain is flagged rather than guessed.
- review — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
- diff — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
- upload — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
- enrich — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in
data/games.json, the seed data for a future web frontend.
Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit.
Requirements
- macOS or Linux, Python 3.12+, uv
- An Anthropic API key (vision extraction)
- A BoardGameGeek account and a registered BGG application — as of BGG's 2025 API policy, the XML API requires a Bearer token from a registered app. Register a free non-commercial application at boardgamegeek.com/applications (approval can take a week or more, so apply on day one), then create a token. Each user needs their own; tokens must not be shared.
Quick start
git clone https://git.kestrelsnest.social/eric/bggpipe.git
cd bggpipe
uv sync # installs Python deps
uv run bggpipe init # guided setup: folders, credentials, browser download
The init wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a .env it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? cp .env.example .env, fill it in, and run uv run playwright install chromium yourself.
Secrets live in environment variables only, never in config files, code, or logs. .env is gitignored. If you use direnv, the committed .envrc loads .env automatically after a one-time direnv allow; otherwise export the variables yourself (e.g. set -a; source .env; set +a).
| Variable | Used by | What it is |
|---|---|---|
ANTHROPIC_API_KEY |
extract | Anthropic API key |
BGG_API_TOKEN |
resolve, diff, enrich | Bearer token from your registered BGG application |
BGG_USERNAME |
diff, upload, enrich | Your BGG username (public, but kept in .env so it lives in one place) |
BGG_PASSWORD |
upload (website login) | Your BGG password |
Non-secret knobs (photos_dir, data_dir, the vision model, the rate limit) live in config.toml. From here you can drive everything from the browser:
uv run bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
Six pages in one local app: Pipeline (run stages, watch live output), Photos (drag-and-drop upload, gallery, reshoot tickets), Review (keyboard-first match and edition decisions), Catalog (every extracted title and its status), Queue (exactly what upload will do, plus its full log), and Library (your enriched collection, browsable once real BGG data lands). The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
uv run bggpipe extract # photos → titles.json (+ retake prompts)
uv run bggpipe resolve # titles → BGG ids/versions in matches.csv
uv run bggpipe review --web # review UI only
uv run bggpipe diff # compare against your BGG collection
uv run bggpipe upload --dry-run # ALWAYS inspect this first
uv run bggpipe upload --limit 1 # then one game, then small batches
uv run bggpipe enrich # full metadata → data/games.json
Each stage skips work it has already done; --force/--refresh flags redo it. review without --web runs in the terminal instead. upload also has --retry-failed, --verify (re-fetches your collection and cross-checks the log), and runs a headed browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
RPGs on your shelves
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay local only — they're never uploaded, since your BGG collection can't hold them.
Taking good shelf photos
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in unidentified.json and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in photos/, and run extract again.
Bring your own shelves
This repo doubles as its author's live pipeline, so data/ ships with his real artifacts — extracted titles, matches, and 2018 collection snapshots. Before running against your shelves, clear the data:
rm data/*.csv data/*.json data/*.xml data/STUB_DATA.marker
rm -rf data/bgg_cache data/extract_raw
Two of those files deserve a word:
data/STUB_DATA.marker— the committed CSVs were resolved from hand-written stub fixtures (the author's BGG application is still awaiting approval), so every version id in them is a synthetic placeholder. The upload stage refuses to run while this marker exists, precisely so nobody (including a fresh clone) can push placeholder data to a real BGG account. Starting fresh with your own token, you'll never see it again.data/collection_snapshot_*.xml— withBGG_API_TOKENset,difffetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs asdata/collection_snapshot_base.xmlanddata/collection_snapshot_expansions.xml(if you get a "queued" message, refresh after a few seconds):https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion
No token yet? extract works immediately (it only needs the Anthropic key), and resolve does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
Development
uv run pytest # offline test suite (stub fixtures, no network)
uv run pytest --run-live # + a read-only live-API smoke test (needs token)
uv run ruff check src tests # lint
uv run bggpipe review --web --dev # review UI with code hot-reload
The review UI live-follows the data files — run extract or resolve in another terminal and the page updates itself. Architecture and contributor guidance: CLAUDE.md and the full spec; BGG automation notes: docs/bgg-upload-flow.md.
A note on being a good BGG citizen
This tool is not affiliated with or supported by BoardGameGeek. It uses only the sanctioned XML API2 for reads (with your own registered application token, per BGG's current policy) and drives the regular website for writes, deliberately slowly (one request every couple of seconds, slower for uploads). Please keep it that way: BGG is a community resource running on community goodwill. You are responsible for your own account — review the dry-run output before a real upload.
License
MIT — see LICENSE.
BoardGameGeek and BGG are trademarks of BoardGameGeek, LLC. bggpipe is an independent project, not affiliated with or endorsed by BoardGameGeek.
Mascot art by Juniper, used with pride.