The README stops being four documents wearing one trench coat

Eric's read on the first-visitor experience: 190 lines of pitch,
manual, gallery, and contributor doc is intimidating when the visitor
only needs the first 40. Split three ways:

README.md is now the front door — what it is, why it exists (told in
first person now, since it IS a personal itch scratched), how the six
stages work, requirements, quick start, one hero screenshot, and the
development/citizenship/license notes. Sixty percent shorter.

docs/tour.md carries the full gallery: all seven pages, the game
detail view, and the phone set, captions intact.

docs/guide.md is the complete user's guide: credentials and config,
the stages and their flags, phone pairing, photo technique, curation
stores, RPG handling, upload safety (including the collection-export
lag), the no-token-yet path, and the keep-data-out-of-git rationale.

Every relative link and README→guide anchor machine-verified to
resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
This commit is contained in:
Eric Wagoner
2026-08-06 10:03:48 -04:00
co-authored by Claude Fable 5
parent 1df784e253
commit 6041150c8d
3 changed files with 163 additions and 121 deletions
+11 -121
View File
@@ -11,11 +11,11 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
→ [6 enrich] → games.json → [6 enrich] → games.json
``` ```
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection, with every upload browser flow verified against the live site (`docs/bgg-upload-flow.md` records what the site actually does, selector by selector). Still sensible on a first run: `--dry-run`, then `--limit 1`. (This repo also carries its author's pipeline data; see [Bring your own shelves](#bring-your-own-shelves).) > **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection. Still sensible on a first run: `--dry-run`, then `--limit 1`.
## Why this exists ## Why this exists
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done. BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. I wanted to point a camera at my shelves instead. This pipeline replaces the typing with: take photos, run a command, settle a handful of ambiguous matches in a review step, done — and every judgment call along the way is yours, made in a review UI, never guessed by the machine.
## How it works ## How it works
@@ -24,58 +24,19 @@ BGG has no bulk import and no write API. Cataloging a few hundred games by hand
3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection. 3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own). 4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
5. **upload** — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included. 5. **upload** — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in `data/games.json`, the seed data for a future web frontend. 6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork, version details) lands in `data/games.json`, feeding a browsable library of your shelves.
Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit. Everything runs locally, every stage survives being killed mid-run, and all artifacts are flat CSV/JSON files you can inspect and edit. You can drive it from the terminal or from a local web app:
## What it looks like
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
![Pipeline dashboard: six stage cards with live counts and Run buttons, showing a completed pipeline — 136 titles extracted, 115 resolved, 62 games added, 136 in the library](docs/screenshots/01.png) ![Pipeline dashboard: six stage cards with live counts and Run buttons, showing a completed pipeline — 136 titles extracted, 115 resolved, 62 games added, 136 in the library](docs/screenshots/01.png)
**Photos**drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow. **➔ [See the full tour](docs/tour.md)** — all seven pages, the game-detail view, and the phone experience.
![Photos page: drag-and-drop upload zone, reshoot tickets with photo thumbnails and shelf-location descriptions, and the gallery of all shelf photos with per-photo title counts](docs/screenshots/02.png)
**Titles** — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games.
![Titles page: 123 extracted titles with AUTO / AWAITING BGG status chips, source photo links, edit and split-into-copies actions, and a shaky-reads filter](docs/screenshots/03.png)
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.
![Review page in its done state: 123 extracted, 17 recognized, 2 with versions, 0 rejected, with 106 titles waiting on the BGG API token](docs/screenshots/04.png)
**Queue** — exactly what upload will do before it does it, and a permanent log of every attempt ever made.
![Upload queue: six new collection entries with game, version, and source photos, plus empty version-updates and upload-log sections](docs/screenshots/05.png)
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
![Library page filled with 136 games as art cards — search, kind filters, player-count filter and sort controls above; RPG entries carry local-only chips](docs/screenshots/06.png)
**A game's detail page** — BGG's stats, chips, and description joined with what only the pipeline knows: *your* edition, and the shelf photo it was read from.
![Game detail page for Britannia: box art beside player counts, playing time, weight, rank, designers and mechanics as chips, with a Your Edition card naming the owned printing](docs/screenshots/10-game-detail.png)
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
![Help page documenting the shelves-to-collection flow, what each page is for, and the proofreading and review checkpoints](docs/screenshots/07.png)
**And on a phone** (`--lan`) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.
<p>
<img src="docs/screenshots/08-phone-menu.jpeg" alt="Phone view: the hamburger menu open over the Photos page, showing all seven pages with attention badges" width="24%">
<img src="docs/screenshots/09-phone-camera.jpeg" alt="Phone view: tapping the photo drop zone offers iOS's Photo Library / Take Photo / Choose Files sheet" width="24%">
<img src="docs/screenshots/10-phone-titles.jpeg" alt="Phone view: the Titles page as stacked cards with status chips, edit and split buttons, and the shaky-reads filter" width="24%">
<img src="docs/screenshots/11-phone-piper.jpeg" alt="Phone view: the Help page's credits card with Juniper's full piper artwork and the BGG trademark attribution" width="24%">
</p>
## Requirements ## Requirements
- Python 3.12+ and [uv](https://docs.astral.sh/uv/), on macOS, Linux, or Windows. (Development happens on macOS; Windows is untested but nothing is platform-specific. One caveat: the owner-only file permissions bggpipe sets on `.env` and browser session state are POSIX-only — on Windows, keep those files in a directory protected by your account.) - Python 3.12+ and [uv](https://docs.astral.sh/uv/), on macOS, Linux, or Windows. (Development happens on macOS; Windows is untested but nothing is platform-specific. One caveat: the owner-only file permissions bggpipe sets on `.env` and browser session state are POSIX-only — on Windows, keep those files in a directory protected by your account.)
- A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint: OpenAI, OpenRouter, or a free local runtime like [Ollama](https://ollama.com/) with a vision-capable model. `config.toml` carries a `[vision.<provider>]` block for each; `vision_provider` picks one. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline. - A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint including a free local [Ollama](https://ollama.com/) model.
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared. - A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared. ([You can start before it arrives.](docs/guide.md#running-before-your-bgg-token-arrives))
## Quick start ## Quick start
@@ -83,81 +44,12 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
uv tool install git+https://git.kestrelsnest.social/eric/bggpipe uv tool install git+https://git.kestrelsnest.social/eric/bggpipe
mkdir shelves && cd shelves # any directory of your own — NOT a clone of this repo mkdir shelves && cd shelves # any directory of your own — NOT a clone of this repo
bggpipe init # guided setup: folders, credentials, browser download bggpipe init # guided setup: folders, credentials, browser download
bggpipe web # the whole app at http://127.0.0.1:8377/
``` ```
Your photos and every pipeline artifact live in the directory where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data ([why this matters](#bring-your-own-shelves)). Drop shelf photos on the Photos page (or straight [from your phone's camera](docs/guide.md#from-your-phone)) and follow the pipeline left to right. Your photos and every pipeline artifact live in the directory where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data — which is also why running inside a clone of this repo is the one unsupported setup: this repo carries its author's live pipeline data at git-tracked paths, and [a `git pull` on top of yours could destroy it](docs/guide.md#keeping-your-data-safe-from-git). bggpipe warns if it catches you doing this.
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? Copy [.env.example](.env.example) beside your data, fill it in, and run `playwright install chromium` yourself. **➔ [The user's guide](docs/guide.md)** — credentials, every stage and flag, phone pairing, RPG handling, fixing misreads, uploading safely, and running before your token arrives.
Secrets live in environment variables only, never in config files, code, or logs, and `.env` is gitignored. Every `bggpipe` command loads `.env` from the working directory by itself — real environment variables always win over the file, so [direnv](https://direnv.net/) users and CI overrides keep working unchanged.
| Variable | Used by | What it is |
|---|---|---|
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
Non-secret knobs (`photos_dir`, `data_dir`, the per-provider vision blocks, the rate limit) live in `config.toml`. From here you can drive everything from the browser:
```sh
bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
```
(Working from a development clone instead? Prefix commands with `uv run`.)
### From your phone
The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:
```sh
bggpipe web --lan # localhost + your network, behind an access key
```
Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (`data/.lan_key`; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic fixtures ever regenerate). Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
```sh
bggpipe extract # photos → titles.json (+ retake prompts)
bggpipe resolve # titles → BGG ids/versions in matches.csv
bggpipe review --web # review UI only
bggpipe diff # compare against your BGG collection
bggpipe upload --dry-run # ALWAYS inspect this first
bggpipe upload --limit 1 # then one game, then small batches
bggpipe enrich # full metadata → data/games.json
```
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. `review` without `--web` runs in the terminal instead. `upload` also has `--retry-failed`, `--verify` (re-fetches your collection and cross-checks the log), and runs a **headed** browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
### RPGs on your shelves
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them.
### Fixing what the model gets wrong
Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, and **remove** lines that aren't games at all. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store. The in-app **Help** page documents all of it.
### Taking good shelf photos
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again.
## Bring your own shelves
The [Quick start](#quick-start) install — as a tool, run in a directory of your own — is the only supported way to use bggpipe on your collection. Everything the pipeline produces (photos, matches, review decisions, your upload log) lives where you run it, and a bug fix is `uv tool upgrade bggpipe`, which by construction cannot touch your data.
**Why not clone and run?** This repo doubles as its author's live pipeline: `data/` ships with their real artifacts, committed and updated often. Run inside a clone and *your* data lands at git-tracked paths — the next `git pull` will refuse to merge, and the usual remedies (`git reset --hard`, `git checkout .`, `git stash`, `git clean -fdx`) would destroy your review decisions, hand-written games, upload log, and photos. bggpipe detects this arrangement and warns at `init` and on the web dashboard; don't ignore it. Clone only to develop (see [Development](#development)), and keep your own pipeline runs elsewhere.
Two files of the author's data deserve a word:
- **`data/STUB_DATA.marker`** — normally absent. It appears only if the synthetic stub fixtures (from `scripts/write_stub_fixtures.py`) regenerate the CSVs, and the upload stage refuses to run while it exists — so placeholder data can never reach a real BGG account. The committed CSVs are real API data.
- **`data/collection_snapshot_*.xml`** — with `BGG_API_TOKEN` set, `diff` fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
No token yet? `extract` works immediately (it only needs the Anthropic key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
## Development ## Development
@@ -171,9 +63,7 @@ uv run ruff check src tests # lint
uv run bggpipe web --dev # the app with code hot-reload uv run bggpipe web --dev # the app with code hot-reload
``` ```
Run your own *pipeline* from a different directory (see [Bring your own shelves](#bring-your-own-shelves)) — the clone's `data/` is the author's live data, and the app will warn if you point it at git-tracked paths. Run your own *pipeline* from a different directory — the clone's `data/` is the author's live data. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
The review UI live-follows the data files — run `extract` or `resolve` in another terminal and the page updates itself. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
## A note on being a good BGG citizen ## A note on being a good BGG citizen
+108
View File
@@ -0,0 +1,108 @@
# The bggpipe user's guide
Everything past the [README](../README.md)'s quick start: credentials, every stage and its flags, the web app, phones, RPGs, fixing the model's mistakes, and running without a BGG token. Screenshots of everything described here: the [tour](tour.md).
## Contents
- [Credentials and configuration](#credentials-and-configuration)
- [The web app](#the-web-app)
- [From your phone](#from-your-phone)
- [The stages, from the terminal](#the-stages-from-the-terminal)
- [Taking good shelf photos](#taking-good-shelf-photos)
- [Fixing what the model gets wrong](#fixing-what-the-model-gets-wrong)
- [RPGs on your shelves](#rpgs-on-your-shelves)
- [Uploading safely](#uploading-safely)
- [Running before your BGG token arrives](#running-before-your-bgg-token-arrives)
- [Keeping your data safe from git](#keeping-your-data-safe-from-git)
## Credentials and configuration
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? Copy [.env.example](../.env.example) beside your data, fill it in, and run `playwright install chromium` yourself.
Secrets live in environment variables only, never in config files, code, or logs, and `.env` is gitignored. Every `bggpipe` command loads `.env` from the working directory by itself — real environment variables always win over the file, so [direnv](https://direnv.net/) users and CI overrides keep working unchanged.
| Variable | Used by | What it is |
|---|---|---|
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
Non-secret knobs live in `config.toml`: `photos_dir`, `data_dir`, the BGG rate limit, and the vision setup — a `[vision.<provider>]` block per provider ("anthropic" or any OpenAI-compatible endpoint, including a local [Ollama](https://ollama.com/)), with `vision_provider` picking one. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline.
## The web app
```sh
bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
```
Seven pages — Pipeline, Photos, Titles, Review, Queue, Library, and Help — all [pictured in the tour](tour.md). Stage runs execute one at a time in the background with live output; every decision saves immediately; the pages live-follow the data files, so a stage run in another terminal shows up without a refresh. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic test fixtures ever regenerate). The in-app **Help** page documents every status chip and keyboard shortcut.
## From your phone
The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:
```sh
bggpipe web --lan # localhost + your network, behind an access key
```
Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (`data/.lan_key`; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
## The stages, from the terminal
Every stage is also a command, and the two interfaces share all state:
```sh
bggpipe extract # photos → titles.json (+ retake prompts)
bggpipe resolve # titles → BGG ids/versions in matches.csv
bggpipe review --web # review UI only
bggpipe diff # compare against your BGG collection
bggpipe upload --dry-run # ALWAYS inspect this first
bggpipe upload --limit 1 # then one game, then small batches
bggpipe enrich # full metadata → data/games.json
```
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit. `review` without `--web` runs in the terminal instead.
## Taking good shelf photos
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the Photos page's "reshoot" tickets: photograph those boxes up close, drop the new photo in, and run `extract` again.
## Fixing what the model gets wrong
Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, **remove** lines that aren't games at all, and **add** a game no photo caught. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`, `data/title_additions.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store.
Games BGG doesn't have at all can be kept as **local** library citizens: their detail page in the Library takes hand-written facts (players, playtime, publisher, notes) and a cover photo of your own, stored in `data/local_games.json` and `data/local_art/` — the only source such a game will ever have.
## RPGs on your shelves
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched (designers, publishers, genres from RPGGeek), and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them. When the automatic search can't reach the right database (BGG has board games named "Dungeons & Dragons" too), every Review card has explicit **search BGG** / **search RPGGeek** buttons.
## Uploading safely
`upload` drives a real logged-in browser session against your real account, so it is deliberately careful:
- `--dry-run` logs what would happen without touching the site — always read it first, then `--limit 1`, then small batches.
- The browser runs **headed** by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
- Requests are slow on purpose (seconds between actions, per BGG's API policy); the Queue page shows exactly what will run before it runs, and `upload_log.csv` keeps a permanent record of every attempt.
- `--retry-failed` re-attempts failures; `--verify` re-fetches your collection and cross-checks the log. Note that BGG's collection export can lag the website by hours — freshly-landed work may look missing to `diff`/`--verify` until it catches up.
- Review decisions outrank the queue: re-deciding a match after `diff` retires its queued job automatically.
## Running before your BGG token arrives
BGG application approval can take a week or more. Until then: `extract` works immediately (it only needs the vision key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
`diff` normally fetches your collection live, but there's a logged-in-browser exemption that needs no token: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
## Keeping your data safe from git
The [README's quick start](../README.md#quick-start) — installed as a tool, run in a directory of your own — is the only supported way to use bggpipe on your collection. Everything the pipeline produces lives where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data.
**Why not clone and run?** The source repo doubles as its author's live pipeline: `data/` ships with their real artifacts, committed and updated often. Run inside a clone and *your* data lands at git-tracked paths — the next `git pull` will refuse to merge, and the usual remedies (`git reset --hard`, `git checkout .`, `git stash`, `git clean -fdx`) would destroy your review decisions, hand-written games, upload log, and photos. bggpipe detects this arrangement and warns at `init` and on the web dashboard; don't ignore it.
One committed file of the author's data deserves a word: `data/STUB_DATA.marker` is normally absent. It appears only if the synthetic stub fixtures (from `scripts/write_stub_fixtures.py`) regenerate the CSVs, and the upload stage refuses to run while it exists — so placeholder data can never reach a real BGG account.
+44
View File
@@ -0,0 +1,44 @@
# A tour of bggpipe
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help. Every screenshot below is the real app on the author's real shelves. (Back to the [README](../README.md) · how to use it all: the [user's guide](guide.md).)
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
![Pipeline dashboard: six stage cards with live counts and Run buttons, showing a completed pipeline — 136 titles extracted, 115 resolved, 62 games added, 136 in the library](screenshots/01.png)
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
![Photos page: drag-and-drop upload zone, reshoot tickets with photo thumbnails and shelf-location descriptions, and the gallery of all shelf photos with per-photo title counts](screenshots/02.png)
**Titles** — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games.
![Titles page: 123 extracted titles with AUTO / AWAITING BGG status chips, source photo links, edit and split-into-copies actions, and a shaky-reads filter](screenshots/03.png)
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.
![Review page in its done state: 123 extracted, 17 recognized, 2 with versions, 0 rejected, with 106 titles waiting on the BGG API token](screenshots/04.png)
**Queue** — exactly what upload will do before it does it, and a permanent log of every attempt ever made.
![Upload queue: six new collection entries with game, version, and source photos, plus empty version-updates and upload-log sections](screenshots/05.png)
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
![Library page filled with 136 games as art cards — search, kind filters, player-count filter and sort controls above; RPG entries carry local-only chips](screenshots/06.png)
**A game's detail page** — BGG's stats, chips, and description joined with what only the pipeline knows: *your* edition, and the shelf photo it was read from.
![Game detail page for Britannia: box art beside player counts, playing time, weight, rank, designers and mechanics as chips, with a Your Edition card naming the owned printing](screenshots/10-game-detail.png)
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
![Help page documenting the shelves-to-collection flow, what each page is for, and the proofreading and review checkpoints](screenshots/07.png)
**And on a phone** (`--lan`) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.
<p>
<img src="screenshots/08-phone-menu.jpeg" alt="Phone view: the hamburger menu open over the Photos page, showing all seven pages with attention badges" width="24%">
<img src="screenshots/09-phone-camera.jpeg" alt="Phone view: tapping the photo drop zone offers iOS's Photo Library / Take Photo / Choose Files sheet" width="24%">
<img src="screenshots/10-phone-titles.jpeg" alt="Phone view: the Titles page as stacked cards with status chips, edit and split buttons, and the shaky-reads filter" width="24%">
<img src="screenshots/11-phone-piper.jpeg" alt="Phone view: the Help page's credits card with Juniper's full piper artwork and the BGG trademark attribution" width="24%">
</p>