diff --git a/CLAUDE.md b/CLAUDE.md index 86dde7e..efbb369 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,7 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel - **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items. - Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025). - BGG has **no write API**: writes drive the real website with a logged-in Playwright session. -- **Stub-resolved data is never upload-ready.** All version_ids (and some game data) in `matches.csv`, `to_add.csv`, and `to_update.csv` currently come from SYNTHETIC stub fixtures — placeholders until real fixtures exist. When `BGG_API_TOKEN` arrives: delete both cache dirs, re-record fixtures, `resolve --force`, re-review. The caches carry a `STUB_FIXTURES.marker` provenance file (written by the fixture generators); the upload stage MUST refuse to run while `data/bgg_cache/STUB_FIXTURES.marker` exists. +- **Stub-resolved data is never upload-ready.** All version_ids (and some game data) in `matches.csv`, `to_add.csv`, and `to_update.csv` currently come from SYNTHETIC stub fixtures — placeholders until real fixtures exist. When `BGG_API_TOKEN` arrives: delete both cache dirs, re-record fixtures, `resolve --force`, re-review. Two provenance markers guard this (both written by the fixture generators): `data/bgg_cache/STUB_FIXTURES.marker` (gitignored, travels with the stub XML) and `data/STUB_DATA.marker` (**committed**, so a fresh clone stays guarded). The upload stage MUST refuse to run while either exists; delete `data/STUB_DATA.marker` only after re-resolving from real fixtures. ## Domain gotchas @@ -51,4 +51,4 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel ## Git - Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub** — `gh` CLI does not work here. -- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`. +- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`. diff --git a/README.md b/README.md index f727edf..7e0a0aa 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv → [6 enrich] → games.json ``` -> **Status: pre-release.** The design is complete ([full spec](bgg-shelf-pipeline-spec.md)); the code is being built. Nothing below works yet. +> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).) ## Why this exists @@ -32,10 +32,73 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. - macOS or Linux, Python 3.12+, [uv](https://docs.astral.sh/uv/) - An [Anthropic API key](https://console.anthropic.com/) (vision extraction) -- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more — apply early), then create a token. Each user needs their own — tokens must not be shared. -- Playwright Chromium: `uv run playwright install chromium` +- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more — **apply on day one**), then create a token. Each user needs their own — tokens must not be shared. -Secrets come from environment variables only — `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` — and are never written to disk or logs. +## Quick start + +```sh +git clone https://git.kestrelsnest.social/eric/bggpipe.git +cd bggpipe +uv sync # installs Python deps +uv run playwright install chromium # browser for the upload stage +cp .env.example .env # then fill in your keys +``` + +Secrets live in environment variables only — never in config files, code, or logs. `.env` is gitignored. If you use [direnv](https://direnv.net/), the committed `.envrc` loads `.env` automatically after a one-time `direnv allow`; otherwise export the variables yourself (e.g. `set -a; source .env; set +a`). + +| Variable | Used by | What it is | +|---|---|---| +| `ANTHROPIC_API_KEY` | extract | Anthropic API key | +| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application | +| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) | +| `BGG_PASSWORD` | upload (website login) | Your BGG password | + +Non-secret knobs (`photos_dir`, `data_dir`, the vision model, the rate limit) live in `config.toml`. Drop your shelf photos into `photos/` and run the stages in order: + +```sh +uv run bggpipe extract # photos → titles.json (+ retake prompts) +uv run bggpipe resolve # titles → BGG ids/versions in matches.csv +uv run bggpipe review --web # review UI at http://127.0.0.1:8377/ +uv run bggpipe diff # compare against your BGG collection +uv run bggpipe upload --dry-run # ALWAYS inspect this first +uv run bggpipe upload --limit 1 # then one game, then small batches +uv run bggpipe enrich # full metadata → data/games.json +``` + +Each stage skips work it has already done; `--force`/`--refresh` flags redo it. `review` without `--web` runs in the terminal instead. `upload` also has `--retry-failed`, `--verify` (re-fetches your collection and cross-checks the log), and runs a **headed** browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused. + +### Taking good shelf photos + +Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human — if you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again. + +## Bring your own shelves + +This repo doubles as its author's live pipeline, so `data/` ships with his real artifacts — extracted titles, matches, and 2018 collection snapshots — and `config.toml` carries his username (your `BGG_USERNAME` in `.env` overrides it). Before running against *your* shelves, clear the data: + +```sh +rm data/*.csv data/*.json data/*.xml data/STUB_DATA.marker +rm -rf data/bgg_cache data/extract_raw +``` + +Two of those files deserve a word: + +- **`data/STUB_DATA.marker`** — the committed CSVs were resolved from *hand-written stub fixtures* (the author's BGG application is still awaiting approval), so every version id in them is a synthetic placeholder. The upload stage refuses to run while this marker exists, precisely so nobody — including a fresh clone — can push placeholder data to a real BGG account. Starting fresh with your own token, you'll never see it again. +- **`data/collection_snapshot_*.xml`** — with `BGG_API_TOKEN` set, `diff` fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds): + - `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1` + - `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion` + +No token yet? `extract` works immediately (it only needs the Anthropic key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go. + +## Development + +```sh +uv run pytest # offline test suite (stub fixtures, no network) +uv run pytest --run-live # + a read-only live-API smoke test (needs token) +uv run ruff check src tests # lint +uv run bggpipe review --web --dev # review UI with code hot-reload +``` + +The review UI live-follows the data files — run `extract` or `resolve` in another terminal and the page updates itself. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md). ## A note on being a good BGG citizen diff --git a/config.toml b/config.toml index f131f23..2416a82 100644 --- a/config.toml +++ b/config.toml @@ -1,9 +1,12 @@ # Non-secret configuration for bggpipe. Secrets (ANTHROPIC_API_KEY, -# BGG_PASSWORD) come from env vars only and must never appear here. +# BGG_PASSWORD, BGG_API_TOKEN) come from env vars only and must never +# appear here. -# TODO(eric): set your BGG username — the live smoke test and diff/upload -# stages need it. Leave empty to skip the live smoke test. -bgg_username = "" +# The BGG username isn't a secret (it's public on your profile), so it may +# live here — but BGG_USERNAME from .env OVERRIDES it, and since upload +# needs BGG_USERNAME anyway, .env alone is enough. Cloners: change or +# clear this. +bgg_username = "ewagoner" photos_dir = "photos" data_dir = "data" diff --git a/data/STUB_DATA.marker b/data/STUB_DATA.marker new file mode 100644 index 0000000..ed284bc --- /dev/null +++ b/data/STUB_DATA.marker @@ -0,0 +1 @@ +The CSVs in this directory were resolved from hand-written stub fixtures, not real BGG data — version_ids are SYNTHETIC. The upload stage refuses to run while this file exists. Delete it only after re-resolving against real recorded fixtures (BGG_API_TOKEN + scripts/record_fixtures.py + resolve --force). diff --git a/scripts/write_photo_fixtures.py b/scripts/write_photo_fixtures.py index 275559c..dead1b2 100644 --- a/scripts/write_photo_fixtures.py +++ b/scripts/write_photo_fixtures.py @@ -20,6 +20,11 @@ from pathlib import Path from bggpipe.bgg_client import cache_key TARGETS = (Path("tests/fixtures/bgg_cache"), Path("data/bgg_cache")) + +# Committed alongside the stub-derived CSVs (the cache markers are +# gitignored, so this is what protects a fresh clone): upload refuses to +# run while it exists. +DATA_MARKER = Path("data/STUB_DATA.marker") SEARCH_TYPES = "boardgame,boardgameexpansion" BG, EXP = "boardgame", "boardgameexpansion" @@ -363,7 +368,16 @@ def main() -> None: "This cache contains hand-written stub XML, not real BGG " "responses. Data resolved from it must not be uploaded.\n" ) + DATA_MARKER.parent.mkdir(parents=True, exist_ok=True) + DATA_MARKER.write_text( + "The CSVs in this directory were resolved from hand-written stub " + "fixtures, not real BGG data — version_ids are SYNTHETIC. The " + "upload stage refuses to run while this file exists. Delete it " + "only after re-resolving against real recorded fixtures " + "(BGG_API_TOKEN + scripts/record_fixtures.py + resolve --force).\n" + ) print(f"Wrote {len(files)} fixture file(s) to {' and '.join(map(str, TARGETS))}") + print(f"Wrote {DATA_MARKER} (committed; upload refuses while it exists)") if __name__ == "__main__": diff --git a/src/bggpipe/upload.py b/src/bggpipe/upload.py index 20a23e8..2c40c30 100644 --- a/src/bggpipe/upload.py +++ b/src/bggpipe/upload.py @@ -415,14 +415,23 @@ def run_upload( rng = rng or random.Random() now = now or (lambda: datetime.now(UTC).isoformat(timespec="seconds")) - marker = cfg.cache_dir / "STUB_FIXTURES.marker" - if marker.exists(): + # Two provenance markers guard the same fact from different angles: the + # cache marker travels with the stub XML (gitignored, so a fresh clone + # loses it), while data/STUB_DATA.marker is COMMITTED alongside the + # stub-derived CSVs — so a clone can never upload placeholder ids. + markers = [ + cfg.cache_dir / "STUB_FIXTURES.marker", + cfg.data_dir / "STUB_DATA.marker", + ] + marker = next((m for m in markers if m.exists()), None) + if marker is not None: if not dry_run: typer.echo( f"Refusing to upload: {marker} exists — every resolved " "version_id is a synthetic stub placeholder. Once " "BGG_API_TOKEN arrives: delete both cache dirs, re-record " - "fixtures, `resolve --force`, re-review, re-diff." + "fixtures, `resolve --force`, re-review, re-diff, then " + "delete the marker(s)." ) raise typer.Exit(code=1) typer.echo( diff --git a/tests/test_upload.py b/tests/test_upload.py index 605a4bb..f113ba4 100644 --- a/tests/test_upload.py +++ b/tests/test_upload.py @@ -290,3 +290,13 @@ def test_verify_checks_version_on_adds_and_updates(): def test_verify_ignores_failed_rows(): log = [_log_row(action="add", bgg_id="5", status="failed")] assert verify_uploads(log, []) == [] + + +def test_fresh_clone_marker_blocks_upload_without_cache_dir(tmp_path): + # A fresh clone has the committed data/STUB_DATA.marker but no + # gitignored bgg_cache/ at all — upload must still refuse. + cfg = _cfg(tmp_path) + (tmp_path / "STUB_DATA.marker").write_text("stub-derived CSVs") + _seed_data(tmp_path, to_add=[_add_row()]) + with pytest.raises(typer.Exit): + run_upload(cfg, uploader=FakeUploader(), sleep=lambda s: None, now=NOW)