The README stops being four documents wearing one trench coat
Eric's read on the first-visitor experience: 190 lines of pitch, manual, gallery, and contributor doc is intimidating when the visitor only needs the first 40. Split three ways: README.md is now the front door — what it is, why it exists (told in first person now, since it IS a personal itch scratched), how the six stages work, requirements, quick start, one hero screenshot, and the development/citizenship/license notes. Sixty percent shorter. docs/tour.md carries the full gallery: all seven pages, the game detail view, and the phone set, captions intact. docs/guide.md is the complete user's guide: credentials and config, the stages and their flags, phone pairing, photo technique, curation stores, RPG handling, upload safety (including the collection-export lag), the no-token-yet path, and the keep-data-out-of-git rationale. Every relative link and README→guide anchor machine-verified to resolve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
This commit is contained in:
Eric Wagoner
co-authored by
Claude Fable 5
Claude Fable 5
parent
1df784e253
commit
6041150c8d
@@ -11,11 +11,11 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
|
||||
→ [6 enrich] → games.json
|
||||
```
|
||||
|
||||
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection, with every upload browser flow verified against the live site (`docs/bgg-upload-flow.md` records what the site actually does, selector by selector). Still sensible on a first run: `--dry-run`, then `--limit 1`. (This repo also carries its author's pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
|
||||
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection. Still sensible on a first run: `--dry-run`, then `--limit 1`.
|
||||
|
||||
## Why this exists
|
||||
|
||||
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done.
|
||||
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. I wanted to point a camera at my shelves instead. This pipeline replaces the typing with: take photos, run a command, settle a handful of ambiguous matches in a review step, done — and every judgment call along the way is yours, made in a review UI, never guessed by the machine.
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -24,58 +24,19 @@ BGG has no bulk import and no write API. Cataloging a few hundred games by hand
|
||||
3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
|
||||
4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
|
||||
5. **upload** — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
|
||||
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in `data/games.json`, the seed data for a future web frontend.
|
||||
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork, version details) lands in `data/games.json`, feeding a browsable library of your shelves.
|
||||
|
||||
Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit.
|
||||
|
||||
## What it looks like
|
||||
|
||||
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
|
||||
Everything runs locally, every stage survives being killed mid-run, and all artifacts are flat CSV/JSON files you can inspect and edit. You can drive it from the terminal or from a local web app:
|
||||
|
||||
|
||||
|
||||
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
|
||||
|
||||
|
||||
|
||||
**Titles** — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games.
|
||||
|
||||
|
||||
|
||||
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.
|
||||
|
||||
|
||||
|
||||
**Queue** — exactly what upload will do before it does it, and a permanent log of every attempt ever made.
|
||||
|
||||
|
||||
|
||||
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
|
||||
|
||||
|
||||
|
||||
**A game's detail page** — BGG's stats, chips, and description joined with what only the pipeline knows: *your* edition, and the shelf photo it was read from.
|
||||
|
||||
|
||||
|
||||
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
|
||||
|
||||
|
||||
|
||||
**And on a phone** (`--lan`) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.
|
||||
|
||||
<p>
|
||||
<img src="docs/screenshots/08-phone-menu.jpeg" alt="Phone view: the hamburger menu open over the Photos page, showing all seven pages with attention badges" width="24%">
|
||||
<img src="docs/screenshots/09-phone-camera.jpeg" alt="Phone view: tapping the photo drop zone offers iOS's Photo Library / Take Photo / Choose Files sheet" width="24%">
|
||||
<img src="docs/screenshots/10-phone-titles.jpeg" alt="Phone view: the Titles page as stacked cards with status chips, edit and split buttons, and the shaky-reads filter" width="24%">
|
||||
<img src="docs/screenshots/11-phone-piper.jpeg" alt="Phone view: the Help page's credits card with Juniper's full piper artwork and the BGG trademark attribution" width="24%">
|
||||
</p>
|
||||
**➔ [See the full tour](docs/tour.md)** — all seven pages, the game-detail view, and the phone experience.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.12+ and [uv](https://docs.astral.sh/uv/), on macOS, Linux, or Windows. (Development happens on macOS; Windows is untested but nothing is platform-specific. One caveat: the owner-only file permissions bggpipe sets on `.env` and browser session state are POSIX-only — on Windows, keep those files in a directory protected by your account.)
|
||||
- A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint: OpenAI, OpenRouter, or a free local runtime like [Ollama](https://ollama.com/) with a vision-capable model. `config.toml` carries a `[vision.<provider>]` block for each; `vision_provider` picks one. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline.
|
||||
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared.
|
||||
- A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint including a free local [Ollama](https://ollama.com/) model.
|
||||
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared. ([You can start before it arrives.](docs/guide.md#running-before-your-bgg-token-arrives))
|
||||
|
||||
## Quick start
|
||||
|
||||
@@ -83,81 +44,12 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
|
||||
uv tool install git+https://git.kestrelsnest.social/eric/bggpipe
|
||||
mkdir shelves && cd shelves # any directory of your own — NOT a clone of this repo
|
||||
bggpipe init # guided setup: folders, credentials, browser download
|
||||
bggpipe web # the whole app at http://127.0.0.1:8377/
|
||||
```
|
||||
|
||||
Your photos and every pipeline artifact live in the directory where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data ([why this matters](#bring-your-own-shelves)).
|
||||
Drop shelf photos on the Photos page (or straight [from your phone's camera](docs/guide.md#from-your-phone)) and follow the pipeline left to right. Your photos and every pipeline artifact live in the directory where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data — which is also why running inside a clone of this repo is the one unsupported setup: this repo carries its author's live pipeline data at git-tracked paths, and [a `git pull` on top of yours could destroy it](docs/guide.md#keeping-your-data-safe-from-git). bggpipe warns if it catches you doing this.
|
||||
|
||||
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? Copy [.env.example](.env.example) beside your data, fill it in, and run `playwright install chromium` yourself.
|
||||
|
||||
Secrets live in environment variables only, never in config files, code, or logs, and `.env` is gitignored. Every `bggpipe` command loads `.env` from the working directory by itself — real environment variables always win over the file, so [direnv](https://direnv.net/) users and CI overrides keep working unchanged.
|
||||
|
||||
| Variable | Used by | What it is |
|
||||
|---|---|---|
|
||||
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
|
||||
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
|
||||
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
|
||||
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
|
||||
|
||||
Non-secret knobs (`photos_dir`, `data_dir`, the per-provider vision blocks, the rate limit) live in `config.toml`. From here you can drive everything from the browser:
|
||||
|
||||
```sh
|
||||
bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
|
||||
```
|
||||
|
||||
(Working from a development clone instead? Prefix commands with `uv run`.)
|
||||
|
||||
### From your phone
|
||||
|
||||
The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:
|
||||
|
||||
```sh
|
||||
bggpipe web --lan # localhost + your network, behind an access key
|
||||
```
|
||||
|
||||
Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (`data/.lan_key`; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.
|
||||
|
||||
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
|
||||
|
||||
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic fixtures ever regenerate). Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
||||
|
||||
```sh
|
||||
bggpipe extract # photos → titles.json (+ retake prompts)
|
||||
bggpipe resolve # titles → BGG ids/versions in matches.csv
|
||||
bggpipe review --web # review UI only
|
||||
bggpipe diff # compare against your BGG collection
|
||||
bggpipe upload --dry-run # ALWAYS inspect this first
|
||||
bggpipe upload --limit 1 # then one game, then small batches
|
||||
bggpipe enrich # full metadata → data/games.json
|
||||
```
|
||||
|
||||
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. `review` without `--web` runs in the terminal instead. `upload` also has `--retry-failed`, `--verify` (re-fetches your collection and cross-checks the log), and runs a **headed** browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
|
||||
|
||||
### RPGs on your shelves
|
||||
|
||||
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them.
|
||||
|
||||
### Fixing what the model gets wrong
|
||||
|
||||
Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, and **remove** lines that aren't games at all. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store. The in-app **Help** page documents all of it.
|
||||
|
||||
### Taking good shelf photos
|
||||
|
||||
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again.
|
||||
|
||||
## Bring your own shelves
|
||||
|
||||
The [Quick start](#quick-start) install — as a tool, run in a directory of your own — is the only supported way to use bggpipe on your collection. Everything the pipeline produces (photos, matches, review decisions, your upload log) lives where you run it, and a bug fix is `uv tool upgrade bggpipe`, which by construction cannot touch your data.
|
||||
|
||||
**Why not clone and run?** This repo doubles as its author's live pipeline: `data/` ships with their real artifacts, committed and updated often. Run inside a clone and *your* data lands at git-tracked paths — the next `git pull` will refuse to merge, and the usual remedies (`git reset --hard`, `git checkout .`, `git stash`, `git clean -fdx`) would destroy your review decisions, hand-written games, upload log, and photos. bggpipe detects this arrangement and warns at `init` and on the web dashboard; don't ignore it. Clone only to develop (see [Development](#development)), and keep your own pipeline runs elsewhere.
|
||||
|
||||
Two files of the author's data deserve a word:
|
||||
|
||||
- **`data/STUB_DATA.marker`** — normally absent. It appears only if the synthetic stub fixtures (from `scripts/write_stub_fixtures.py`) regenerate the CSVs, and the upload stage refuses to run while it exists — so placeholder data can never reach a real BGG account. The committed CSVs are real API data.
|
||||
- **`data/collection_snapshot_*.xml`** — with `BGG_API_TOKEN` set, `diff` fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
|
||||
|
||||
No token yet? `extract` works immediately (it only needs the Anthropic key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
|
||||
**➔ [The user's guide](docs/guide.md)** — credentials, every stage and flag, phone pairing, RPG handling, fixing misreads, uploading safely, and running before your token arrives.
|
||||
|
||||
## Development
|
||||
|
||||
@@ -171,9 +63,7 @@ uv run ruff check src tests # lint
|
||||
uv run bggpipe web --dev # the app with code hot-reload
|
||||
```
|
||||
|
||||
Run your own *pipeline* from a different directory (see [Bring your own shelves](#bring-your-own-shelves)) — the clone's `data/` is the author's live data, and the app will warn if you point it at git-tracked paths.
|
||||
|
||||
The review UI live-follows the data files — run `extract` or `resolve` in another terminal and the page updates itself. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
|
||||
Run your own *pipeline* from a different directory — the clone's `data/` is the author's live data. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
|
||||
|
||||
## A note on being a good BGG citizen
|
||||
|
||||
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
# The bggpipe user's guide
|
||||
|
||||
Everything past the [README](../README.md)'s quick start: credentials, every stage and its flags, the web app, phones, RPGs, fixing the model's mistakes, and running without a BGG token. Screenshots of everything described here: the [tour](tour.md).
|
||||
|
||||
## Contents
|
||||
|
||||
- [Credentials and configuration](#credentials-and-configuration)
|
||||
- [The web app](#the-web-app)
|
||||
- [From your phone](#from-your-phone)
|
||||
- [The stages, from the terminal](#the-stages-from-the-terminal)
|
||||
- [Taking good shelf photos](#taking-good-shelf-photos)
|
||||
- [Fixing what the model gets wrong](#fixing-what-the-model-gets-wrong)
|
||||
- [RPGs on your shelves](#rpgs-on-your-shelves)
|
||||
- [Uploading safely](#uploading-safely)
|
||||
- [Running before your BGG token arrives](#running-before-your-bgg-token-arrives)
|
||||
- [Keeping your data safe from git](#keeping-your-data-safe-from-git)
|
||||
|
||||
## Credentials and configuration
|
||||
|
||||
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? Copy [.env.example](../.env.example) beside your data, fill it in, and run `playwright install chromium` yourself.
|
||||
|
||||
Secrets live in environment variables only, never in config files, code, or logs, and `.env` is gitignored. Every `bggpipe` command loads `.env` from the working directory by itself — real environment variables always win over the file, so [direnv](https://direnv.net/) users and CI overrides keep working unchanged.
|
||||
|
||||
| Variable | Used by | What it is |
|
||||
|---|---|---|
|
||||
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
|
||||
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
|
||||
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
|
||||
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
|
||||
|
||||
Non-secret knobs live in `config.toml`: `photos_dir`, `data_dir`, the BGG rate limit, and the vision setup — a `[vision.<provider>]` block per provider ("anthropic" or any OpenAI-compatible endpoint, including a local [Ollama](https://ollama.com/)), with `vision_provider` picking one. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline.
|
||||
|
||||
## The web app
|
||||
|
||||
```sh
|
||||
bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
|
||||
```
|
||||
|
||||
Seven pages — Pipeline, Photos, Titles, Review, Queue, Library, and Help — all [pictured in the tour](tour.md). Stage runs execute one at a time in the background with live output; every decision saves immediately; the pages live-follow the data files, so a stage run in another terminal shows up without a refresh. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic test fixtures ever regenerate). The in-app **Help** page documents every status chip and keyboard shortcut.
|
||||
|
||||
## From your phone
|
||||
|
||||
The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:
|
||||
|
||||
```sh
|
||||
bggpipe web --lan # localhost + your network, behind an access key
|
||||
```
|
||||
|
||||
Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (`data/.lan_key`; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.
|
||||
|
||||
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
|
||||
|
||||
## The stages, from the terminal
|
||||
|
||||
Every stage is also a command, and the two interfaces share all state:
|
||||
|
||||
```sh
|
||||
bggpipe extract # photos → titles.json (+ retake prompts)
|
||||
bggpipe resolve # titles → BGG ids/versions in matches.csv
|
||||
bggpipe review --web # review UI only
|
||||
bggpipe diff # compare against your BGG collection
|
||||
bggpipe upload --dry-run # ALWAYS inspect this first
|
||||
bggpipe upload --limit 1 # then one game, then small batches
|
||||
bggpipe enrich # full metadata → data/games.json
|
||||
```
|
||||
|
||||
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit. `review` without `--web` runs in the terminal instead.
|
||||
|
||||
## Taking good shelf photos
|
||||
|
||||
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the Photos page's "reshoot" tickets: photograph those boxes up close, drop the new photo in, and run `extract` again.
|
||||
|
||||
## Fixing what the model gets wrong
|
||||
|
||||
Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, **remove** lines that aren't games at all, and **add** a game no photo caught. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`, `data/title_additions.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store.
|
||||
|
||||
Games BGG doesn't have at all can be kept as **local** library citizens: their detail page in the Library takes hand-written facts (players, playtime, publisher, notes) and a cover photo of your own, stored in `data/local_games.json` and `data/local_art/` — the only source such a game will ever have.
|
||||
|
||||
## RPGs on your shelves
|
||||
|
||||
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched (designers, publishers, genres from RPGGeek), and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them. When the automatic search can't reach the right database (BGG has board games named "Dungeons & Dragons" too), every Review card has explicit **search BGG** / **search RPGGeek** buttons.
|
||||
|
||||
## Uploading safely
|
||||
|
||||
`upload` drives a real logged-in browser session against your real account, so it is deliberately careful:
|
||||
|
||||
- `--dry-run` logs what would happen without touching the site — always read it first, then `--limit 1`, then small batches.
|
||||
- The browser runs **headed** by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
|
||||
- Requests are slow on purpose (seconds between actions, per BGG's API policy); the Queue page shows exactly what will run before it runs, and `upload_log.csv` keeps a permanent record of every attempt.
|
||||
- `--retry-failed` re-attempts failures; `--verify` re-fetches your collection and cross-checks the log. Note that BGG's collection export can lag the website by hours — freshly-landed work may look missing to `diff`/`--verify` until it catches up.
|
||||
- Review decisions outrank the queue: re-deciding a match after `diff` retires its queued job automatically.
|
||||
|
||||
## Running before your BGG token arrives
|
||||
|
||||
BGG application approval can take a week or more. Until then: `extract` works immediately (it only needs the vision key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
|
||||
|
||||
`diff` normally fetches your collection live, but there's a logged-in-browser exemption that needs no token: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
|
||||
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
|
||||
|
||||
## Keeping your data safe from git
|
||||
|
||||
The [README's quick start](../README.md#quick-start) — installed as a tool, run in a directory of your own — is the only supported way to use bggpipe on your collection. Everything the pipeline produces lives where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data.
|
||||
|
||||
**Why not clone and run?** The source repo doubles as its author's live pipeline: `data/` ships with their real artifacts, committed and updated often. Run inside a clone and *your* data lands at git-tracked paths — the next `git pull` will refuse to merge, and the usual remedies (`git reset --hard`, `git checkout .`, `git stash`, `git clean -fdx`) would destroy your review decisions, hand-written games, upload log, and photos. bggpipe detects this arrangement and warns at `init` and on the web dashboard; don't ignore it.
|
||||
|
||||
One committed file of the author's data deserves a word: `data/STUB_DATA.marker` is normally absent. It appears only if the synthetic stub fixtures (from `scripts/write_stub_fixtures.py`) regenerate the CSVs, and the upload stage refuses to run while it exists — so placeholder data can never reach a real BGG account.
|
||||
@@ -0,0 +1,44 @@
|
||||
# A tour of bggpipe
|
||||
|
||||
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help. Every screenshot below is the real app on the author's real shelves. (Back to the [README](../README.md) · how to use it all: the [user's guide](guide.md).)
|
||||
|
||||
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
|
||||
|
||||
|
||||
|
||||
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
|
||||
|
||||
|
||||
|
||||
**Titles** — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games.
|
||||
|
||||
|
||||
|
||||
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.
|
||||
|
||||
|
||||
|
||||
**Queue** — exactly what upload will do before it does it, and a permanent log of every attempt ever made.
|
||||
|
||||
|
||||
|
||||
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
|
||||
|
||||
|
||||
|
||||
**A game's detail page** — BGG's stats, chips, and description joined with what only the pipeline knows: *your* edition, and the shelf photo it was read from.
|
||||
|
||||
|
||||
|
||||
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
|
||||
|
||||
|
||||
|
||||
**And on a phone** (`--lan`) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.
|
||||
|
||||
<p>
|
||||
<img src="screenshots/08-phone-menu.jpeg" alt="Phone view: the hamburger menu open over the Photos page, showing all seven pages with attention badges" width="24%">
|
||||
<img src="screenshots/09-phone-camera.jpeg" alt="Phone view: tapping the photo drop zone offers iOS's Photo Library / Take Photo / Choose Files sheet" width="24%">
|
||||
<img src="screenshots/10-phone-titles.jpeg" alt="Phone view: the Titles page as stacked cards with status chips, edit and split buttons, and the shaky-reads filter" width="24%">
|
||||
<img src="screenshots/11-phone-piper.jpeg" alt="Phone view: the Help page's credits card with Juniper's full piper artwork and the BGG trademark attribution" width="24%">
|
||||
</p>
|
||||
Reference in New Issue
Block a user