eric/bggpipe
1
0
Fork 0
Code Issues Pull Requests Releases 2 Activity
Files
6c95953dc0abc8f8ee79c4c17f07a7204577a3cf
bggpipe/CLAUDE.md
T
Eric WagonerandClaude Fable 5 6c95953dc0 "It's here anyway" + the tape measure: the human overrules the packer 
Eric, looking at a dense cube full of "no room" and "unmeasured":
how do I say it really IS here, and in what orientation? Two answers.

The packer models one flat pile and one standing row; real shelves are
3-D — boxes sit crosswise, ride on top of piles, stand proud of the
edge. Acknowledge generalizes from overhang-only to the human
overruling the model: "it's here anyway" on any resident the lanes
can't take. An acknowledged box takes a lane when one works (honoring
a flat/standing override; the spine-out case lands relaxed standing,
now tagged ", proud" by the packer itself instead of guessed from
misfit-ness) and otherwise is WEDGED IN: really there, in some
arrangement the model can't see, outside the capacity bars and never
warned about. The previous commit's ackable gate is gone — accepting
always does something now, so no placebo remains to guard against.

And "unmeasured" is no longer a dead end: the chip is a button. Tap,
enter width × length × depth from your own tape measure, and it saves
to data/measurements.json (a new committed curation store) with dims
source "measured" — overlaid on every games.json read (web, dims,
export, CSV import), because the owner's ruler outranks BGG's
database. All three fields empty clears one.

Verified live: Etherfields on the tower reads "standing, proud", an
acknowledged box in a width-starved cube reads "wedged in", the
measure form opens with three inputs, zero page errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
2026-08-09 16:10:35 -04:00

61 lines
8.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project
`bggpipe` — a resumable, idempotent CLI pipeline that turns shelf photos into a BoardGameGeek collection, in six stages:
| # | Command | What it does | Status |
|---|---------|--------------|--------|
| 1 | `bggpipe extract` | Claude vision reads titles + edition cues from `photos/` | working |
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (real API data since 2026-08-05) |
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | working — all browser flows verified live 2026-08-06 (62 adds + 36 version updates landed) |
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
| + | `bggpipe export` | render the library as self-contained static pages (covers downloaded, never hotlinked; shelf photos never included; Powered-by-BGG badge slot) | working |
| + | `bggpipe dims` | offline shelf-space report over box dimensions enrich collects from BGG *versions* (dims live per-version, not per-game; 0 = never entered; versionless games need a unanimous chorus within 0.5"/axis else `conflicting` — never guessed); fit checks run against `data/furniture.json` openings | working |
| + | `bggpipe shelve --import` | bulk-assign games to openings from a name,opening CSV; rejects (unknown/ambiguous/doesn't-fit/contained) reported, never guessed | working |
Full design lives in `docs/spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
## Commands
- `uv sync` — install deps (Python 3.12+, managed by **uv**; use `uv add`, never pip). `uv run bggpipe init` handles first-run setup (folders, .env credentials, the one-time `playwright install chromium`).
- `uv run bggpipe web` — the app: eight pages (Pipeline `/`, Photos, Titles, Review, Queue, Library, Shelves, Help) in a shared sidebar shell (responsive: hamburger nav + stacked tables under 900px); stage runs execute one-at-a-time in a background job. `--lan` binds 0.0.0.0 behind a per-device access key: persisted in `data/.lan_key` (gitignored), printed as a QR at startup, cookie-paired for a year, required on EVERY network request (loopback clients and `/static/*` are exempt; the Host/Origin guard still applies). Phone camera uploads (generic `image.jpg` names) get minted `shelf-<timestamp>` names — only explicitly-named files trigger the replace-to-reshoot flow.
- `uv run bggpipe <stage>` — run a pipeline stage. Non-secret settings come from `config.toml` (dirs, rate limit, `vision_provider` + per-provider `[vision.*]` blocks — "anthropic" or any OpenAI-compatible endpoint incl. local Ollama); `--config` overrides the path.
- `uv run pytest` — the suite runs fully offline against fixtures. Tests marked `live` hit the real BGG API (read-only) and are skipped unless you pass `--run-live`.
- `uv run ruff check` / `uv run ruff format` — lint (rules E, F, I, UP, B, SIM) and format.
## Layout
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`); `templates/shell.html` + `templates/pages/*` + `static/app.{css,js}` are the web UI (the stylesheet is the design system — tokens derive from the mascot art), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `jobs.py` (single-slot background stage runner for the web UI), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`, `fsio.py` (atomic writes), `init_wizard.py` (first-run setup).
- `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
- `tests/fixtures/bgg_cache/` — recorded real XML fixtures the offline tests run against.
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).
## Hard rules (from spec — never violate)
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 2–4 s randomized delay between games.
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. The token is ACTIVE and `tests/fixtures/bgg_cache/` holds real recorded responses; add fixtures for new tests with `scripts/record_fixtures.py`.
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
- **Synthetic data must never reach upload.** The stub era ended 2026-08-05: `matches.csv` and `tests/fixtures/bgg_cache/` now hold real API data. The guard mechanism stays armed: the stub-fixture generators write `data/bgg_cache/STUB_FIXTURES.marker` (gitignored) and `data/STUB_DATA.marker` (committed), and the upload stage MUST refuse to run while either exists — regenerating stubs re-locks upload automatically.
## Domain gotchas
- Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan).
- Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry.
- Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance.
- **Human curation is durable**: `data/title_splits.json` (photo-scoped split-into-copies decisions, honored by extract's dedupe AND resolve's dedupe), `data/title_edits.json` (corrected reads/cues, applied before dedupe on every titles.json rebuild), `data/title_removals.json` (lines removed from the catalog — filtered out of every rebuild; delete the record to undo), `data/title_additions.json` (games added without a photo — joined into every rebuild; a later photo sighting dedupe-merges with them), `data/local_games.json` + `data/local_art/` (hand-written facts and a cover photo for off-BGG games — the ONLY source for them, merged over the photo reads by enrich), and `data/furniture.json` + `data/locations.json` (the owner's shelf layout and each game's physical spot — local-only, never BGG's business; a location can carry `lane` and `acknowledged` — the human overruling the packer), and `data/measurements.json` (hand-measured box dims, source `measured` — the owner's tape measure outranks BGG, overlaid on every games.json read) persist forever. Row-level decisions persist via the `dedupe_veto` column — edits never drop veto'd rows (a rename retitles them in place); removal drops them (explicitly discarding the line).
- **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). RPGGeek items carry their OWN link types (`rpgdesigner`, `rpgpublisher`, `rpggenre`, `rpgcategory`, `rpgmechanic`) — a board-game-only parser silently returns nothing for them. Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope).
- Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift.
## Git
- Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub** — `gh` CLI does not work here.
- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/title_splits.json`, `data/title_edits.json`, `data/title_removals.json`, `data/title_additions.json`, `data/local_games.json`, `data/local_art/`, `data/furniture.json`, `data/locations.json`, `data/measurements.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, `data/.lan_key`, Playwright storage state, or `.env`.
Reference in New Issue View Git Blame Copy Permalink