Docs catch up with reality: the stub era is over and upload is proven
The README still warned "not yet battle-tested / browser flows await their first real run" — they ran, against a live account: 62 adds and 36 version updates landed, every flow verified. Status blurb, stage table, spec's update-mode paragraph (now describing the verified collection-cell route instead of asking for manual verification), and the CLAUDE.md token/fixture notes all now describe the present. Screenshots 01 and 06 retaken: the Pipeline card shows a completed run's real numbers instead of stub-lock banners, and the Library shows 136 games as art cards instead of the empty state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
This commit is contained in:
co-authored by
Claude Fable 5
parent
9663f29702
commit
22fa17b5ee
@@ -12,7 +12,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||||||
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (real API data since 2026-08-05) |
|
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (real API data since 2026-08-05) |
|
||||||
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
|
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
|
||||||
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
|
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
|
||||||
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | built; browser flows unverified until real data exists (`--dry-run` works now) |
|
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | working — all browser flows verified live 2026-08-06 (62 adds + 36 version updates landed) |
|
||||||
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
|
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
|
||||||
|
|
||||||
Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
|
Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
|
||||||
@@ -29,14 +29,14 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
|
|||||||
|
|
||||||
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`); `templates/shell.html` + `templates/pages/*` + `static/app.{css,js}` are the web UI (the stylesheet is the design system — tokens derive from the mascot art), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `jobs.py` (single-slot background stage runner for the web UI), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`, `fsio.py` (atomic writes), `init_wizard.py` (first-run setup).
|
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`); `templates/shell.html` + `templates/pages/*` + `static/app.{css,js}` are the web UI (the stylesheet is the design system — tokens derive from the mascot art), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `jobs.py` (single-slot background stage runner for the web UI), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`, `fsio.py` (atomic writes), `init_wizard.py` (first-run setup).
|
||||||
- `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
|
- `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
|
||||||
- `tests/fixtures/bgg_cache/` — stub XML fixtures the offline tests run against.
|
- `tests/fixtures/bgg_cache/` — recorded real XML fixtures the offline tests run against.
|
||||||
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).
|
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).
|
||||||
|
|
||||||
## Hard rules (from spec — never violate)
|
## Hard rules (from spec — never violate)
|
||||||
|
|
||||||
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 2–4 s randomized delay between games.
|
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 2–4 s randomized delay between games.
|
||||||
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
|
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
|
||||||
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. Until Eric's registration at boardgamegeek.com/applications is approved, tests run on the stub fixtures in `tests/fixtures/bgg_cache/`; re-record them with `scripts/record_fixtures.py` once the token exists.
|
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. The token is ACTIVE and `tests/fixtures/bgg_cache/` holds real recorded responses; add fixtures for new tests with `scripts/record_fixtures.py`.
|
||||||
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
|
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
|
||||||
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
|
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
|
||||||
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
|
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
|
|||||||
→ [6 enrich] → games.json
|
→ [6 enrich] → games.json
|
||||||
```
|
```
|
||||||
|
|
||||||
> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
|
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection, with every upload browser flow verified against the live site (`docs/bgg-upload-flow.md` records what the site actually does, selector by selector). Still sensible on a first run: `--dry-run`, then `--limit 1`. (This repo also carries its author's pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
|
||||||
|
|
||||||
## Why this exists
|
## Why this exists
|
||||||
|
|
||||||
@@ -30,9 +30,9 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
|
|||||||
|
|
||||||
## What it looks like
|
## What it looks like
|
||||||
|
|
||||||
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) are banners, not surprises.
|
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
|
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
|
||||||
|
|
||||||
@@ -42,7 +42,7 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto). Here: everything resolvable is reviewed, 106 titles parked awaiting the API token.
|
**Review** — keyboard-first decisions on ambiguous matches, editions, and cross-photo merges (with veto), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
@@ -50,9 +50,9 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
**Library** — the enriched collection browser (board games and RPGs), waiting for real BGG data to fill its shelves.
|
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
|
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
|
||||||
|
|
||||||
@@ -111,7 +111,7 @@ Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera
|
|||||||
|
|
||||||
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
|
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
|
||||||
|
|
||||||
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help, each pictured above. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic fixtures ever regenerate). Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
uv run bggpipe extract # photos → titles.json (+ retake prompts)
|
uv run bggpipe extract # photos → titles.json (+ retake prompts)
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ Beyond the bare game, capture **which edition/version I own** wherever the photo
|
|||||||
## Constraints & Context
|
## Constraints & Context
|
||||||
|
|
||||||
- BGG has **no write API**. Reads go through the XML API2 (`https://boardgamegeek.com/xmlapi2/`); writes must automate the website itself with a real login session.
|
- BGG has **no write API**. Reads go through the XML API2 (`https://boardgamegeek.com/xmlapi2/`); writes must automate the website itself with a real login session.
|
||||||
- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer <token>` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+, so development runs on recorded/stub XML fixtures until then.
|
- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer <token>` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+; offline development and tests run on recorded XML fixtures.
|
||||||
- BGG's XML API queues collection requests: a first call may return HTTP 202 ("try again"). Retry with backoff.
|
- BGG's XML API queues collection requests: a first call may return HTTP 202 ("try again"). Retry with backoff.
|
||||||
- BGG will throttle aggressive clients. Target ≤1 request every 2 seconds to any BGG endpoint, with jittered backoff on 429/503.
|
- BGG will throttle aggressive clients. Target ≤1 request every 2 seconds to any BGG endpoint, with jittered backoff on 429/503.
|
||||||
- BGG changed API access policies in 2025; some older community tools broke. Don't depend on undocumented endpoints beyond XML API2 and the public website.
|
- BGG changed API access policies in 2025; some older community tools broke. Don't depend on undocumented endpoints beyond XML API2 and the public website.
|
||||||
@@ -88,7 +88,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
|
|||||||
- Log in to boardgamegeek.com with credentials from env vars (`BGG_USERNAME`, `BGG_PASSWORD`). Never write credentials to disk or logs. Persist the browser session/storage state locally so repeat runs don't re-login.
|
- Log in to boardgamegeek.com with credentials from env vars (`BGG_USERNAME`, `BGG_PASSWORD`). Never write credentials to disk or logs. Persist the browser session/storage state locally so repeat runs don't re-login.
|
||||||
- For each row in `to_add.csv`: navigate to the game page, use the "Add to Collection" flow, set status **Owned**, and — when a `version_id` is present — set the specific version in the collection item's version picker before saving. Manually walk this flow once and document the selectors before automating; the version UI is the most fragile part.
|
- For each row in `to_add.csv`: navigate to the game page, use the "Add to Collection" flow, set status **Owned**, and — when a `version_id` is present — set the specific version in the collection item's version picker before saving. Manually walk this flow once and document the selectors before automating; the version UI is the most fragile part.
|
||||||
- Adding a second copy of an already-owned game must create a NEW collection entry, not edit the existing one.
|
- Adding a second copy of an already-owned game must create a NEW collection entry, not edit the existing one.
|
||||||
- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. Verify the already-owned dialog behavior manually first — flagged as unverified in `docs/bgg-upload-flow.md`.
|
- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. The verified route (2026-08-06, documented in `docs/bgg-upload-flow.md`): the collection table's version cell (`td.collection_version[onclick*="<collid>"]`) opens an inline editor whose radio values ARE version ids; clicking a radio saves immediately.
|
||||||
- Log every attempt to `upload_log.csv`: `bgg_id, name, status (added|already_present|failed), timestamp, error`.
|
- Log every attempt to `upload_log.csv`: `bgg_id, name, status (added|already_present|failed), timestamp, error`.
|
||||||
- Idempotent: skip IDs already logged as `added`; re-verify against a fresh collection fetch on `--verify`.
|
- Idempotent: skip IDs already logged as `added`; re-verify against a fresh collection fetch on `--verify`.
|
||||||
- Deliberately slow: 2–4 s randomized delay between games. This is a real account on a community site — behave like a polite human.
|
- Deliberately slow: 2–4 s randomized delay between games. This is a real account on a community site — behave like a polite human.
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 332 KiB After Width: | Height: | Size: 224 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 364 KiB After Width: | Height: | Size: 1.1 MiB |
Reference in New Issue
Block a user