diff --git a/CLAUDE.md b/CLAUDE.md index 599a4bc..cc0b1f2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co | 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (real API data since 2026-08-05) | | 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working | | 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working | -| 5 | `bggpipe upload` | add games via a logged-in Playwright session | built; browser flows unverified until real data exists (`--dry-run` works now) | +| 5 | `bggpipe upload` | add games via a logged-in Playwright session | working — all browser flows verified live 2026-08-06 (62 adds + 36 version updates landed) | | 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working | Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`. @@ -29,14 +29,14 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel - `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`); `templates/shell.html` + `templates/pages/*` + `static/app.{css,js}` are the web UI (the stylesheet is the design system — tokens derive from the mascot art), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `jobs.py` (single-slot background stage runner for the web UI), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`, `fsio.py` (atomic writes), `init_wizard.py` (first-run setup). - `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists. -- `tests/fixtures/bgg_cache/` — stub XML fixtures the offline tests run against. +- `tests/fixtures/bgg_cache/` — recorded real XML fixtures the offline tests run against. - `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git). ## Hard rules (from spec — never violate) - **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 2–4 s randomized delay between games. - **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored. -- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. Until Eric's registration at boardgamegeek.com/applications is approved, tests run on the stub fixtures in `tests/fixtures/bgg_cache/`; re-record them with `scripts/record_fixtures.py` once the token exists. +- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. The token is ACTIVE and `tests/fixtures/bgg_cache/` holds real recorded responses; add fixtures for new tests with `scripts/record_fixtures.py`. - **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items. - Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025). - BGG has **no write API**: writes drive the real website with a logged-in Playwright session. diff --git a/README.md b/README.md index c2b40fe..a011580 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv → [6 enrich] → games.json ``` -> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).) +> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection, with every upload browser flow verified against the live site (`docs/bgg-upload-flow.md` records what the site actually does, selector by selector). Still sensible on a first run: `--dry-run`, then `--limit 1`. (This repo also carries its author's pipeline data; see [Bring your own shelves](#bring-your-own-shelves).) ## Why this exists @@ -30,9 +30,9 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. ## What it looks like -**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) are banners, not surprises. +**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added. -![Pipeline dashboard: six stage cards with live counts, Run buttons, and warning banners for the missing BGG token and the stub-data upload lock](docs/screenshots/01.png) +![Pipeline dashboard: six stage cards with live counts and Run buttons, showing a completed pipeline — 136 titles extracted, 115 resolved, 62 games added, 136 in the library](docs/screenshots/01.png) **Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow. @@ -42,7 +42,7 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. ![Titles page: 123 extracted titles with AUTO / AWAITING BGG status chips, source photo links, edit and split-into-copies actions, and a shaky-reads filter](docs/screenshots/03.png) -**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto). Here: everything resolvable is reviewed, 106 titles parked awaiting the API token. +**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database. ![Review page in its done state: 123 extracted, 17 recognized, 2 with versions, 0 rejected, with 106 titles waiting on the BGG API token](docs/screenshots/04.png) @@ -50,9 +50,9 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. ![Upload queue: six new collection entries with game, version, and source photos, plus empty version-updates and upload-log sections](docs/screenshots/05.png) -**Library** — the enriched collection browser (board games and RPGs), waiting for real BGG data to fill its shelves. +**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there. -![Library page empty state with search and Board games / RPGs filters, explaining that the library fills after the enrich stage runs](docs/screenshots/06.png) +![Library page filled with 136 games as art cards — search, kind filters, player-count filter and sort controls above; RPG entries carry local-only chips](docs/screenshots/06.png) **Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app. @@ -111,7 +111,7 @@ Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead). -Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state: +Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic fixtures ever regenerate). Prefer the terminal? Every stage is also a command, and the two interfaces share all state: ```sh uv run bggpipe extract # photos → titles.json (+ retake prompts) diff --git a/bgg-shelf-pipeline-spec.md b/bgg-shelf-pipeline-spec.md index df0bca9..908ac4f 100644 --- a/bgg-shelf-pipeline-spec.md +++ b/bgg-shelf-pipeline-spec.md @@ -11,7 +11,7 @@ Beyond the bare game, capture **which edition/version I own** wherever the photo ## Constraints & Context - BGG has **no write API**. Reads go through the XML API2 (`https://boardgamegeek.com/xmlapi2/`); writes must automate the website itself with a real login session. -- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer ` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+, so development runs on recorded/stub XML fixtures until then. +- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer ` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+; offline development and tests run on recorded XML fixtures. - BGG's XML API queues collection requests: a first call may return HTTP 202 ("try again"). Retry with backoff. - BGG will throttle aggressive clients. Target ≤1 request every 2 seconds to any BGG endpoint, with jittered backoff on 429/503. - BGG changed API access policies in 2025; some older community tools broke. Don't depend on undocumented endpoints beyond XML API2 and the public website. @@ -88,7 +88,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a - Log in to boardgamegeek.com with credentials from env vars (`BGG_USERNAME`, `BGG_PASSWORD`). Never write credentials to disk or logs. Persist the browser session/storage state locally so repeat runs don't re-login. - For each row in `to_add.csv`: navigate to the game page, use the "Add to Collection" flow, set status **Owned**, and — when a `version_id` is present — set the specific version in the collection item's version picker before saving. Manually walk this flow once and document the selectors before automating; the version UI is the most fragile part. - Adding a second copy of an already-owned game must create a NEW collection entry, not edit the existing one. -- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. Verify the already-owned dialog behavior manually first — flagged as unverified in `docs/bgg-upload-flow.md`. +- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. The verified route (2026-08-06, documented in `docs/bgg-upload-flow.md`): the collection table's version cell (`td.collection_version[onclick*=""]`) opens an inline editor whose radio values ARE version ids; clicking a radio saves immediately. - Log every attempt to `upload_log.csv`: `bgg_id, name, status (added|already_present|failed), timestamp, error`. - Idempotent: skip IDs already logged as `added`; re-verify against a fresh collection fetch on `--verify`. - Deliberately slow: 2–4 s randomized delay between games. This is a real account on a community site — behave like a polite human. diff --git a/docs/screenshots/01.png b/docs/screenshots/01.png index a6a1726..0c8075b 100644 Binary files a/docs/screenshots/01.png and b/docs/screenshots/01.png differ diff --git a/docs/screenshots/06.png b/docs/screenshots/06.png index 6e2773f..2f81840 100644 Binary files a/docs/screenshots/06.png and b/docs/screenshots/06.png differ