Eric WagonerandClaude Fable 5
f677c7ce65
Eric wanted the library on his blog; the spec always called games.json "the seed for a future web frontend." The new export stage renders it as self-contained static pages — an index with search, one page per game with facts, chips, the owner's edition and the description — that drop into any static host (Hugo's static/ folder included). No server, no build step, no external requests from the published pages. Public pages carry obligations a localhost app doesn't. Cover art is downloaded once from BGG's CDN instead of hotlinked (0.3s between fetches — a guest, not a crawler; part-file writes so a failure never leaves a truncated image; re-runs skip what exists, so the export is idempotent and resumable like every stage). The footer shows a Powered-by-BGG badge per BGG's public-app policy — text by default, upgraded to the official logo when the owner saves it from their registered-application page as data/powered-by-bgg.png — plus the trademark attribution. And one privacy rule, tested: shelf photos are never exported; they picture the inside of the owner's home. Covers and hand-added local art only, per Eric's explicit choice. Slugs are deterministic and collision-stable (two editions of one game get -2 suffixes in sorted-key order) so re-exports keep every URL. Descriptions un-double-encode BGG's entities. First real run: 136 pages, 254 covers, 64MB, live on the blog's static directory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
93 lines
8.9 KiB
Markdown
93 lines
8.9 KiB
Markdown
# bggpipe — Shelf-to-BoardGameGeek Collection Pipeline
|
|
|
|
|
|
|
|
|
|
<img src="assets/logo-full.jpeg" alt="the bggpipe piper — a bagpiper whose bag is a board game box" width="220" align="right">
|
|
|
|
Photograph your board game shelves. End up with your whole collection — including which *edition* of each game you own — cataloged on [BoardGameGeek](https://boardgamegeek.com).
|
|
|
|
```
|
|
photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
|
|
→ [3 review] → matches.csv (approved) → [4 diff] → to_add.csv
|
|
→ [5 upload] → upload_log.csv
|
|
→ [6 enrich] → games.json
|
|
```
|
|
|
|
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection. The result is public — [the author's collection on BGG](https://boardgamegeek.com/collection/user/ewagoner) is what this pipeline built. Still sensible on a first run: `--dry-run`, then `--limit 1`.
|
|
|
|
## Why this exists
|
|
|
|
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. I wanted to point a camera at my shelves instead. This pipeline replaces the typing with: take photos, run a command, settle a handful of ambiguous matches in a review step, done — and every judgment call along the way is yours, made in a review UI, never guessed by the machine.
|
|
|
|
## How it works
|
|
|
|
1. **extract** — Shelf photos go to a vision model (Claude by default; any OpenAI-compatible endpoint or a local Ollama model works), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
|
|
2. **resolve** — Titles are matched to BGG game IDs via the [XML API2](https://boardgamegeek.com/wiki/page/BGG_XML_API2) (exact + fuzzy matching, popularity tiebreaks), then edition cues are matched against BGG's version list for each game. Anything uncertain is flagged rather than guessed.
|
|
3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
|
|
4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
|
|
5. **upload** — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
|
|
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork, version details) lands in `data/games.json`, feeding a browsable library of your shelves — which `bggpipe export` can publish as [static pages on your own site](docs/guide.md#publishing-your-library-on-your-own-site).
|
|
|
|
Everything runs locally, every stage survives being killed mid-run, and all artifacts are flat CSV/JSON files you can inspect and edit. You can drive it from the terminal or from a local web app:
|
|
|
|
|
|
|
|
**➔ [See the full tour](docs/tour.md)** — all seven pages, the game-detail view, and the phone experience.
|
|
|
|
## Requirements
|
|
|
|
- Python 3.12+ and [uv](https://docs.astral.sh/uv/), on macOS, Linux, or Windows. (Development happens on macOS; Windows is untested but nothing is platform-specific. One caveat: the owner-only file permissions bggpipe sets on `.env` and browser session state are POSIX-only — on Windows, keep those files in a directory protected by your account.)
|
|
- A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint including a free local [Ollama](https://ollama.com/) model. **Cost expectation:** extracting the author's whole collection — 65 shelf photos, 136 games — cost under a dollar with the default model (Claude Sonnet), one-time. Re-runs are free: every photo's read is cached, and only new or replaced photos go back to the model.
|
|
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared. ([You can start before it arrives.](docs/guide.md#running-before-your-bgg-token-arrives))
|
|
|
|
### Why it asks for your BGG password — and where your credentials go
|
|
|
|
BGG has no write API: the **only** way to add games to a collection is the website itself. So the upload stage signs into boardgamegeek.com in a real browser window, on your machine, and clicks the same buttons you would — you can literally watch it work (the browser is visible by default). That's the whole reason the password is needed, and it's used for exactly that login and nothing else.
|
|
|
|
Everything stays on your computer. There is no bggpipe server, no telemetry, no analytics, and no account with me — I never see your credentials, your collection, or anything else, and the code is right here to check. Credentials live in a local `.env` file (owner-only permissions, never written to logs) and are sent only to boardgamegeek.com itself. The only other service the pipeline ever contacts is the vision provider *you* configure, which receives your shelf photos and nothing more — and with a local Ollama model, even those never leave the house.
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
uv tool install git+https://git.kestrelsnest.social/eric/bggpipe
|
|
mkdir shelves && cd shelves # any directory of your own — NOT a clone of this repo
|
|
bggpipe init # guided setup: folders, credentials, browser download
|
|
bggpipe web # the whole app at http://127.0.0.1:8377/
|
|
```
|
|
|
|
Drop shelf photos on the Photos page (or straight [from your phone's camera](docs/guide.md#from-your-phone)) and follow the pipeline left to right. Your photos and every pipeline artifact live in the directory where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data — which is also why running inside a clone of this repo is the one unsupported setup: this repo carries its author's live pipeline data at git-tracked paths, and [a `git pull` on top of yours could destroy it](docs/guide.md#keeping-your-data-safe-from-git). bggpipe warns if it catches you doing this.
|
|
|
|
**➔ [The user's guide](docs/guide.md)** — credentials, every stage and flag, phone pairing, RPG handling, fixing misreads, uploading safely, and running before your token arrives.
|
|
|
|
## Development
|
|
|
|
```sh
|
|
git clone https://git.kestrelsnest.social/eric/bggpipe.git
|
|
cd bggpipe
|
|
uv sync # installs Python deps
|
|
uv run pytest # offline test suite (recorded fixtures, no network)
|
|
uv run pytest --run-live # + a read-only live-API smoke test (needs token)
|
|
uv run ruff check src tests # lint
|
|
uv run bggpipe web --dev # the app with code hot-reload
|
|
```
|
|
|
|
Run your own *pipeline* from a different directory — the clone's `data/` is the author's live data. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [design contract](docs/spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
|
|
|
|
## Questions, bugs, ideas
|
|
|
|
Email [eric@ericwagoner.com](mailto:eric@ericwagoner.com), or find me on Mastodon at [@eric@toots.kestrelsnest.social](https://toots.kestrelsnest.social/@eric) or Bluesky at [@kestrelsnest.social](https://bsky.app/profile/kestrelsnest.social) — bug reports, confusions, and "it worked, here's my collection" notes all welcome. (This Gitea instance doesn't take public registrations, so there's no issue tracker to file into yet; if enough people show up, one will materialize.)
|
|
|
|
## A note on being a good BGG citizen
|
|
|
|
This tool is **not affiliated with or supported by BoardGameGeek**. It uses only the sanctioned XML API2 for reads (with your own registered application token, per BGG's current policy) and drives the regular website for writes, deliberately slowly (one request every couple of seconds, slower for uploads). Please keep it that way: BGG is a community resource running on community goodwill. You are responsible for your own account — review the dry-run output before a real upload.
|
|
|
|
## License
|
|
|
|
MIT — see [LICENSE](LICENSE).
|
|
|
|
BoardGameGeek and BGG are trademarks of BoardGameGeek, LLC. bggpipe is an
|
|
independent project, not affiliated with or endorsed by BoardGameGeek.
|
|
|
|
Mascot art by Juniper, used with pride.
|