Docs catch up with reality: the stub era is over and upload is proven

The README still warned "not yet battle-tested / browser flows await
their first real run" — they ran, against a live account: 62 adds and
36 version updates landed, every flow verified. Status blurb, stage
table, spec's update-mode paragraph (now describing the verified
collection-cell route instead of asking for manual verification), and
the CLAUDE.md token/fixture notes all now describe the present.

Screenshots 01 and 06 retaken: the Pipeline card shows a completed
run's real numbers instead of stub-lock banners, and the Library shows
136 games as art cards instead of the empty state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
This commit is contained in:
Eric Wagoner
2026-08-06 00:10:57 -04:00
co-authored by Claude Fable 5
parent 9663f29702
commit 22fa17b5ee
5 changed files with 12 additions and 12 deletions
+3 -3
View File
@@ -12,7 +12,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (real API data since 2026-08-05) |
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | built; browser flows unverified until real data exists (`--dry-run` works now) |
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | working — all browser flows verified live 2026-08-06 (62 adds + 36 version updates landed) |
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
@@ -29,14 +29,14 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
- `src/bggpipe/``cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`); `templates/shell.html` + `templates/pages/*` + `static/app.{css,js}` are the web UI (the stylesheet is the design system — tokens derive from the mascot art), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `jobs.py` (single-slot background stage runner for the web UI), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`, `fsio.py` (atomic writes), `init_wizard.py` (first-run setup).
- `scripts/``write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
- `tests/fixtures/bgg_cache/`stub XML fixtures the offline tests run against.
- `tests/fixtures/bgg_cache/`recorded real XML fixtures the offline tests run against.
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).
## Hard rules (from spec — never violate)
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 24 s randomized delay between games.
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. Until Eric's registration at boardgamegeek.com/applications is approved, tests run on the stub fixtures in `tests/fixtures/bgg_cache/`; re-record them with `scripts/record_fixtures.py` once the token exists.
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. The token is ACTIVE and `tests/fixtures/bgg_cache/` holds real recorded responses; add fixtures for new tests with `scripts/record_fixtures.py`.
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
+7 -7
View File
@@ -11,7 +11,7 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
→ [6 enrich] → games.json
```
> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection, with every upload browser flow verified against the live site (`docs/bgg-upload-flow.md` records what the site actually does, selector by selector). Still sensible on a first run: `--dry-run`, then `--limit 1`. (This repo also carries its author's pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
## Why this exists
@@ -30,9 +30,9 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
## What it looks like
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) are banners, not surprises.
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
![Pipeline dashboard: six stage cards with live counts, Run buttons, and warning banners for the missing BGG token and the stub-data upload lock](docs/screenshots/01.png)
![Pipeline dashboard: six stage cards with live counts and Run buttons, showing a completed pipeline — 136 titles extracted, 115 resolved, 62 games added, 136 in the library](docs/screenshots/01.png)
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
@@ -42,7 +42,7 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
![Titles page: 123 extracted titles with AUTO / AWAITING BGG status chips, source photo links, edit and split-into-copies actions, and a shaky-reads filter](docs/screenshots/03.png)
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto). Here: everything resolvable is reviewed, 106 titles parked awaiting the API token.
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.
![Review page in its done state: 123 extracted, 17 recognized, 2 with versions, 0 rejected, with 106 titles waiting on the BGG API token](docs/screenshots/04.png)
@@ -50,9 +50,9 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
![Upload queue: six new collection entries with game, version, and source photos, plus empty version-updates and upload-log sections](docs/screenshots/05.png)
**Library** — the enriched collection browser (board games and RPGs), waiting for real BGG data to fill its shelves.
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
![Library page empty state with search and Board games / RPGs filters, explaining that the library fills after the enrich stage runs](docs/screenshots/06.png)
![Library page filled with 136 games as art cards — search, kind filters, player-count filter and sort controls above; RPG entries carry local-only chips](docs/screenshots/06.png)
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
@@ -111,7 +111,7 @@ Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic fixtures ever regenerate). Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
```sh
uv run bggpipe extract # photos → titles.json (+ retake prompts)
+2 -2
View File
@@ -11,7 +11,7 @@ Beyond the bare game, capture **which edition/version I own** wherever the photo
## Constraints & Context
- BGG has **no write API**. Reads go through the XML API2 (`https://boardgamegeek.com/xmlapi2/`); writes must automate the website itself with a real login session.
- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer <token>` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+, so development runs on recorded/stub XML fixtures until then.
- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer <token>` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+; offline development and tests run on recorded XML fixtures.
- BGG's XML API queues collection requests: a first call may return HTTP 202 ("try again"). Retry with backoff.
- BGG will throttle aggressive clients. Target ≤1 request every 2 seconds to any BGG endpoint, with jittered backoff on 429/503.
- BGG changed API access policies in 2025; some older community tools broke. Don't depend on undocumented endpoints beyond XML API2 and the public website.
@@ -88,7 +88,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
- Log in to boardgamegeek.com with credentials from env vars (`BGG_USERNAME`, `BGG_PASSWORD`). Never write credentials to disk or logs. Persist the browser session/storage state locally so repeat runs don't re-login.
- For each row in `to_add.csv`: navigate to the game page, use the "Add to Collection" flow, set status **Owned**, and — when a `version_id` is present — set the specific version in the collection item's version picker before saving. Manually walk this flow once and document the selectors before automating; the version UI is the most fragile part.
- Adding a second copy of an already-owned game must create a NEW collection entry, not edit the existing one.
- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. Verify the already-owned dialog behavior manually first — flagged as unverified in `docs/bgg-upload-flow.md`.
- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. The verified route (2026-08-06, documented in `docs/bgg-upload-flow.md`): the collection table's version cell (`td.collection_version[onclick*="<collid>"]`) opens an inline editor whose radio values ARE version ids; clicking a radio saves immediately.
- Log every attempt to `upload_log.csv`: `bgg_id, name, status (added|already_present|failed), timestamp, error`.
- Idempotent: skip IDs already logged as `added`; re-verify against a fresh collection fetch on `--verify`.
- Deliberately slow: 24 s randomized delay between games. This is a real account on a community site — behave like a polite human.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 332 KiB

After

Width:  |  Height:  |  Size: 224 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 364 KiB

After

Width:  |  Height:  |  Size: 1.1 MiB