Eric Wagoner
38e20f2c30
Correctness: review vetoes persist via a dedupe_veto column (resolve re-runs no longer overturn humans); diff emits second copies whose confident version matches no owned copy (spec: pairs own only on both ids) and fetches the live collection with refresh; resolve pairs titles.json entries to rows by title so a reshoot photo updates provenance instead of duplicating rows; version lookups survive empty /thing results; publisher tie-break now honors the mixed base/expansion veto and refuses multi-candidate picks; empty-normalized (non-Latin) titles never count as exact. Upload: LoginError aborts a run instead of logging N bogus failures (and 3 identical consecutive failures abort as systemic); Cloudflare interstitials are detected; added-without-version gets its own logged status that verify understands; same-game updates run one per pass so the name-targeted row edit can't overwrite a fresh version; absent diff outputs fail loudly; pagination clicks are paced. Web review: a lock serializes freshen/decide (threadpool race dropped decisions); failed saves roll memory back and always alert the browser (non-JSON 500s included); session warnings reach the page instead of a StringIO; state-load failures and dead servers show banners instead of a blank page; duplicate (title, photos) rows are addressable by ordinal. Consistency: shared CONFIDENT_VERSION_STATUSES, client_for(), Config paths for every artifact, one review-port constant, named matching thresholds, strict collection-id parsing, error-doc responses never cached, unknown config keys warn, extract reports dropped vision entries, fixture generators share escaping + marker text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
55 lines
5.4 KiB
Markdown
55 lines
5.4 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project
|
||
|
||
`bggpipe` — a resumable, idempotent CLI pipeline that turns shelf photos into a BoardGameGeek collection, in six stages:
|
||
|
||
| # | Command | What it does | Status |
|
||
|---|---------|--------------|--------|
|
||
| 1 | `bggpipe extract` | Claude vision reads titles + edition cues from `photos/` | working |
|
||
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (stub data — see hard rules) |
|
||
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
|
||
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
|
||
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | built; browser flows unverified until real data exists (`--dry-run` works now) |
|
||
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
|
||
|
||
Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
|
||
|
||
## Commands
|
||
|
||
- `uv sync` — install deps (Python 3.12+, managed by **uv**; use `uv add`, never pip). Playwright needs a one-time `uv run playwright install chromium`.
|
||
- `uv run bggpipe <stage>` — run a pipeline stage. Non-secret settings come from `config.toml` (username, dirs, vision model, rate limit); `--config` overrides the path.
|
||
- `uv run pytest` — the suite runs fully offline against fixtures. Tests marked `live` hit the real BGG API (read-only) and are skipped unless you pass `--run-live`.
|
||
- `uv run ruff check` / `uv run ruff format` — lint (rules E, F, I, UP, B, SIM) and format.
|
||
|
||
## Layout
|
||
|
||
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`.
|
||
- `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
|
||
- `tests/fixtures/bgg_cache/` — stub XML fixtures the offline tests run against.
|
||
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).
|
||
|
||
## Hard rules (from spec — never violate)
|
||
|
||
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 2–4 s randomized delay between games.
|
||
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
|
||
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. Until Eric's registration at boardgamegeek.com/applications is approved, tests run on the stub fixtures in `tests/fixtures/bgg_cache/`; re-record them with `scripts/record_fixtures.py` once the token exists.
|
||
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
|
||
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
|
||
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
|
||
- **Stub-resolved data is never upload-ready.** All version_ids (and some game data) in `matches.csv`, `to_add.csv`, and `to_update.csv` currently come from SYNTHETIC stub fixtures — placeholders until real fixtures exist. When `BGG_API_TOKEN` arrives: delete both cache dirs, re-record fixtures, `resolve --force`, re-review. Two provenance markers guard this (both written by the fixture generators): `data/bgg_cache/STUB_FIXTURES.marker` (gitignored, travels with the stub XML) and `data/STUB_DATA.marker` (**committed**, so a fresh clone stays guarded). The upload stage MUST refuse to run while either exists; delete `data/STUB_DATA.marker` only after re-resolving from real fixtures.
|
||
|
||
## Domain gotchas
|
||
|
||
- Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan).
|
||
- Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry.
|
||
- Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance.
|
||
- Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift.
|
||
|
||
## Git
|
||
|
||
- Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub** — `gh` CLI does not work here.
|
||
- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`.
|