bggpipe/README.md

bggpipe — Shelf-to-BoardGameGeek Collection Pipeline

Photograph your board game shelves. End up with your whole collection — including which edition of each game you own — cataloged on BoardGameGeek.

photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
        → [3 review] → matches.csv (approved) → [4 diff] → to_add.csv
        → [5 upload] → upload_log.csv
        → [6 enrich] → games.json

Status: battle-tested end to end. The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection, with every upload browser flow verified against the live site (docs/bgg-upload-flow.md records what the site actually does, selector by selector). Still sensible on a first run: --dry-run, then --limit 1. (This repo also carries its author's pipeline data; see Bring your own shelves.)

Why this exists

BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done.

How it works

1. extract — Shelf photos go to a vision model (Claude by default; any OpenAI-compatible endpoint or a local Ollama model works), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
2. resolve — Titles are matched to BGG game IDs via the XML API2 (exact + fuzzy matching, popularity tiebreaks), then edition cues are matched against BGG's version list for each game. Anything uncertain is flagged rather than guessed.
3. review — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
4. diff — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
5. upload — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
6. enrich — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in data/games.json, the seed data for a future web frontend.

Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit.

What it looks like

The Pipeline page — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.

Photos — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.

Titles — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games.

Review — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.

Queue — exactly what upload will do before it does it, and a permanent log of every attempt ever made.

Library — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.

A game's detail page — BGG's stats, chips, and description joined with what only the pipeline knows: your edition, and the shelf photo it was read from.

Help — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.

And on a phone (--lan) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.

Requirements

• Python 3.12+ and uv, on macOS, Linux, or Windows. (Development happens on macOS; Windows is untested but nothing is platform-specific. One caveat: the owner-only file permissions bggpipe sets on .env and browser session state are POSIX-only — on Windows, keep those files in a directory protected by your account.)
• A vision model for extraction — an Anthropic API key by default, or any OpenAI-compatible endpoint: OpenAI, OpenRouter, or a free local runtime like Ollama with a vision-capable model. config.toml carries a [vision.<provider>] block for each; vision_provider picks one. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline.
• A BoardGameGeek account and a registered BGG application — as of BGG's 2025 API policy, the XML API requires a Bearer token from a registered app. Register a free non-commercial application at boardgamegeek.com/applications (approval can take a week or more, so apply on day one), then create a token. Each user needs their own; tokens must not be shared.

Quick start

uv tool install git+https://git.kestrelsnest.social/eric/bggpipe
mkdir shelves && cd shelves   # any directory of your own — NOT a clone of this repo
bggpipe init                  # guided setup: folders, credentials, browser download

Your photos and every pipeline artifact live in the directory where you run it, and uv tool upgrade bggpipe picks up fixes without going anywhere near your data (why this matters).

The init wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a .env it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? Copy .env.example beside your data, fill it in, and run playwright install chromium yourself.

Secrets live in environment variables only, never in config files, code, or logs, and .env is gitignored. Every bggpipe command loads .env from the working directory by itself — real environment variables always win over the file, so direnv users and CI overrides keep working unchanged.

Variable	Used by	What it is
ANTHROPIC_API_KEY	extract	Anthropic API key
BGG_API_TOKEN	resolve, diff, enrich	Bearer token from your registered BGG application
BGG_USERNAME	diff, upload, enrich	Your BGG username (public, but kept in .env so it lives in one place)
BGG_PASSWORD	upload (website login)	Your BGG password

Non-secret knobs (photos_dir, data_dir, the per-provider vision blocks, the rate limit) live in config.toml. From here you can drive everything from the browser:

bggpipe web    # opens http://127.0.0.1:8377/ — the whole app in the browser

(Working from a development clone instead? Prefix commands with uv run.)

From your phone

The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:

bggpipe web --lan   # localhost + your network, behind an access key

Startup prints a pairing link (?k=...) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (data/.lan_key; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.

To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique shelf-<timestamp> names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).

Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic fixtures ever regenerate). Prefer the terminal? Every stage is also a command, and the two interfaces share all state:

bggpipe extract              # photos → titles.json (+ retake prompts)
bggpipe resolve              # titles → BGG ids/versions in matches.csv
bggpipe review --web         # review UI only
bggpipe diff                 # compare against your BGG collection
bggpipe upload --dry-run     # ALWAYS inspect this first
bggpipe upload --limit 1     # then one game, then small batches
bggpipe enrich               # full metadata → data/games.json

Each stage skips work it has already done; --force/--refresh flags redo it. review without --web runs in the terminal instead. upload also has --retry-failed, --verify (re-fetches your collection and cross-checks the log), and runs a headed browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.

RPGs on your shelves

Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay local only — they're never uploaded, since your BGG collection can't hold them.

Fixing what the model gets wrong

Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you edit a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), split a line into per-photo copies when one title is actually several boxes, and remove lines that aren't games at all. Every one of these is durable: the decision lands in a small committed store (data/title_edits.json, data/title_splits.json, data/title_removals.json) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store. The in-app Help page documents all of it.

Taking good shelf photos

Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in unidentified.json and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in photos/, and run extract again.

Bring your own shelves

The Quick start install — as a tool, run in a directory of your own — is the only supported way to use bggpipe on your collection. Everything the pipeline produces (photos, matches, review decisions, your upload log) lives where you run it, and a bug fix is uv tool upgrade bggpipe, which by construction cannot touch your data.

Why not clone and run? This repo doubles as its author's live pipeline: data/ ships with their real artifacts, committed and updated often. Run inside a clone and your data lands at git-tracked paths — the next git pull will refuse to merge, and the usual remedies (git reset --hard, git checkout ., git stash, git clean -fdx) would destroy your review decisions, hand-written games, upload log, and photos. bggpipe detects this arrangement and warns at init and on the web dashboard; don't ignore it. Clone only to develop (see Development), and keep your own pipeline runs elsewhere.

Two files of the author's data deserve a word:

• data/STUB_DATA.marker — normally absent. It appears only if the synthetic stub fixtures (from scripts/write_stub_fixtures.py) regenerate the CSVs, and the upload stage refuses to run while it exists — so placeholder data can never reach a real BGG account. The committed CSVs are real API data.
• data/collection_snapshot_*.xml — with BGG_API_TOKEN set, diff fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as data/collection_snapshot_base.xml and data/collection_snapshot_expansions.xml (if you get a "queued" message, refresh after a few seconds):
  • https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1
  • https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion

No token yet? extract works immediately (it only needs the Anthropic key), and resolve does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.

Development

git clone https://git.kestrelsnest.social/eric/bggpipe.git
cd bggpipe
uv sync                       # installs Python deps
uv run pytest                 # offline test suite (recorded fixtures, no network)
uv run pytest --run-live      # + a read-only live-API smoke test (needs token)
uv run ruff check src tests   # lint
uv run bggpipe web --dev      # the app with code hot-reload

Run your own pipeline from a different directory (see Bring your own shelves) — the clone's data/ is the author's live data, and the app will warn if you point it at git-tracked paths.

The review UI live-follows the data files — run extract or resolve in another terminal and the page updates itself. Architecture and contributor guidance: CLAUDE.md and the full spec; BGG automation notes: docs/bgg-upload-flow.md.

A note on being a good BGG citizen

This tool is not affiliated with or supported by BoardGameGeek. It uses only the sanctioned XML API2 for reads (with your own registered application token, per BGG's current policy) and drives the regular website for writes, deliberately slowly (one request every couple of seconds, slower for uploads). Please keep it that way: BGG is a community resource running on community goodwill. You are responsible for your own account — review the dry-run output before a real upload.

License

MIT — see LICENSE.

BoardGameGeek and BGG are trademarks of BoardGameGeek, LLC. bggpipe is an independent project, not affiliated with or endorsed by BoardGameGeek.

Mascot art by Juniper, used with pride.