Docs for strangers: real quick start, own-shelves guide, clone-safe stub guard

README rewritten for a fresh clone: working status, quick start with
env-var table and per-stage commands, photo-taking guidance, a
bring-your-own-shelves section (this repo carries the author's live
data), tokenless workflows, and a dev section. New committed
data/STUB_DATA.marker closes a real gap: the cache marker is
gitignored, so a fresh clone had stub-derived to_add.csv with no
guard — upload now refuses on either marker. Username docs: BGG_USERNAME
in .env is the one place it's needed; config.toml is a fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -39,7 +39,7 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
 - **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
 - Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
 - BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
-- **Stub-resolved data is never upload-ready.** All version_ids (and some game data) in `matches.csv`, `to_add.csv`, and `to_update.csv` currently come from SYNTHETIC stub fixtures — placeholders until real fixtures exist. When `BGG_API_TOKEN` arrives: delete both cache dirs, re-record fixtures, `resolve --force`, re-review. The caches carry a `STUB_FIXTURES.marker` provenance file (written by the fixture generators); the upload stage MUST refuse to run while `data/bgg_cache/STUB_FIXTURES.marker` exists.
+- **Stub-resolved data is never upload-ready.** All version_ids (and some game data) in `matches.csv`, `to_add.csv`, and `to_update.csv` currently come from SYNTHETIC stub fixtures — placeholders until real fixtures exist. When `BGG_API_TOKEN` arrives: delete both cache dirs, re-record fixtures, `resolve --force`, re-review. Two provenance markers guard this (both written by the fixture generators): `data/bgg_cache/STUB_FIXTURES.marker` (gitignored, travels with the stub XML) and `data/STUB_DATA.marker` (**committed**, so a fresh clone stays guarded). The upload stage MUST refuse to run while either exists; delete `data/STUB_DATA.marker` only after re-resolving from real fixtures.
 
 ## Domain gotchas
 
@@ -51,4 +51,4 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
 ## Git
 
 - Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub** — `gh` CLI does not work here.
-- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`.
+- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`.

README.md

@@ -11,7 +11,7 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
         → [6 enrich] → games.json
 ```
 
-> **Status: pre-release.** The design is complete ([full spec](bgg-shelf-pipeline-spec.md)); the code is being built. Nothing below works yet.
+> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
 
 ## Why this exists
 
@@ -32,10 +32,73 @@ Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing.
 
 - macOS or Linux, Python 3.12+, [uv](https://docs.astral.sh/uv/)
 - An [Anthropic API key](https://console.anthropic.com/) (vision extraction)
-- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more — apply early), then create a token. Each user needs their own — tokens must not be shared.
+- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more — **apply on day one**), then create a token. Each user needs their own — tokens must not be shared.
-- Playwright Chromium: `uv run playwright install chromium`
 
-Secrets come from environment variables only — `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` — and are never written to disk or logs.
+## Quick start
+
+```sh
+git clone https://git.kestrelsnest.social/eric/bggpipe.git
+cd bggpipe
+uv sync                                # installs Python deps
+uv run playwright install chromium     # browser for the upload stage
+cp .env.example .env                   # then fill in your keys
+```
+
+Secrets live in environment variables only — never in config files, code, or logs. `.env` is gitignored. If you use [direnv](https://direnv.net/), the committed `.envrc` loads `.env` automatically after a one-time `direnv allow`; otherwise export the variables yourself (e.g. `set -a; source .env; set +a`).
+
+| Variable | Used by | What it is |
+|---|---|---|
+| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
+| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
+| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
+| `BGG_PASSWORD` | upload (website login) | Your BGG password |
+
+Non-secret knobs (`photos_dir`, `data_dir`, the vision model, the rate limit) live in `config.toml`. Drop your shelf photos into `photos/` and run the stages in order:
+
+```sh
+uv run bggpipe extract              # photos → titles.json (+ retake prompts)
+uv run bggpipe resolve              # titles → BGG ids/versions in matches.csv
+uv run bggpipe review --web        # review UI at http://127.0.0.1:8377/
+uv run bggpipe diff                 # compare against your BGG collection
+uv run bggpipe upload --dry-run    # ALWAYS inspect this first
+uv run bggpipe upload --limit 1    # then one game, then small batches
+uv run bggpipe enrich               # full metadata → data/games.json
+```
+
+Each stage skips work it has already done; `--force`/`--refresh` flags redo it. `review` without `--web` runs in the terminal instead. `upload` also has `--retry-failed`, `--verify` (re-fetches your collection and cross-checks the log), and runs a **headed** browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
+
+### Taking good shelf photos
+
+Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human — if you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again.
+
+## Bring your own shelves
+
+This repo doubles as its author's live pipeline, so `data/` ships with his real artifacts — extracted titles, matches, and 2018 collection snapshots — and `config.toml` carries his username (your `BGG_USERNAME` in `.env` overrides it). Before running against *your* shelves, clear the data:
+
+```sh
+rm data/*.csv data/*.json data/*.xml data/STUB_DATA.marker
+rm -rf data/bgg_cache data/extract_raw
+```
+
+Two of those files deserve a word:
+
+- **`data/STUB_DATA.marker`** — the committed CSVs were resolved from *hand-written stub fixtures* (the author's BGG application is still awaiting approval), so every version id in them is a synthetic placeholder. The upload stage refuses to run while this marker exists, precisely so nobody — including a fresh clone — can push placeholder data to a real BGG account. Starting fresh with your own token, you'll never see it again.
+- **`data/collection_snapshot_*.xml`** — with `BGG_API_TOKEN` set, `diff` fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
+  - `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
+  - `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
+
+No token yet? `extract` works immediately (it only needs the Anthropic key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
+
+## Development
+
+```sh
+uv run pytest                 # offline test suite (stub fixtures, no network)
+uv run pytest --run-live      # + a read-only live-API smoke test (needs token)
+uv run ruff check src tests   # lint
+uv run bggpipe review --web --dev   # review UI with code hot-reload
+```
+
+The review UI live-follows the data files — run `extract` or `resolve` in another terminal and the page updates itself. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
 
 ## A note on being a good BGG citizen
 

config.toml

@@ -1,9 +1,12 @@
 # Non-secret configuration for bggpipe. Secrets (ANTHROPIC_API_KEY,
-# BGG_PASSWORD) come from env vars only and must never appear here.
+# BGG_PASSWORD, BGG_API_TOKEN) come from env vars only and must never
+# appear here.
 
-# TODO(eric): set your BGG username — the live smoke test and diff/upload
+# The BGG username isn't a secret (it's public on your profile), so it may
-# stages need it. Leave empty to skip the live smoke test.
+# live here — but BGG_USERNAME from .env OVERRIDES it, and since upload
-bgg_username = ""
+# needs BGG_USERNAME anyway, .env alone is enough. Cloners: change or
+# clear this.
+bgg_username = "ewagoner"
 
 photos_dir = "photos"
 data_dir = "data"

data/STUB_DATA.marker

@@ -0,0 +1 @@
+The CSVs in this directory were resolved from hand-written stub fixtures, not real BGG data — version_ids are SYNTHETIC. The upload stage refuses to run while this file exists. Delete it only after re-resolving against real recorded fixtures (BGG_API_TOKEN + scripts/record_fixtures.py + resolve --force).

scripts/write_photo_fixtures.py

@@ -20,6 +20,11 @@ from pathlib import Path
 from bggpipe.bgg_client import cache_key
 
 TARGETS = (Path("tests/fixtures/bgg_cache"), Path("data/bgg_cache"))
+
+# Committed alongside the stub-derived CSVs (the cache markers are
+# gitignored, so this is what protects a fresh clone): upload refuses to
+# run while it exists.
+DATA_MARKER = Path("data/STUB_DATA.marker")
 SEARCH_TYPES = "boardgame,boardgameexpansion"
 
 BG, EXP = "boardgame", "boardgameexpansion"
@@ -363,7 +368,16 @@ def main() -> None:
             "This cache contains hand-written stub XML, not real BGG "
             "responses. Data resolved from it must not be uploaded.\n"
         )
+    DATA_MARKER.parent.mkdir(parents=True, exist_ok=True)
+    DATA_MARKER.write_text(
+        "The CSVs in this directory were resolved from hand-written stub "
+        "fixtures, not real BGG data — version_ids are SYNTHETIC. The "
+        "upload stage refuses to run while this file exists. Delete it "
+        "only after re-resolving against real recorded fixtures "
+        "(BGG_API_TOKEN + scripts/record_fixtures.py + resolve --force).\n"
+    )
     print(f"Wrote {len(files)} fixture file(s) to {' and '.join(map(str, TARGETS))}")
+    print(f"Wrote {DATA_MARKER} (committed; upload refuses while it exists)")
 
 
 if __name__ == "__main__":

src/bggpipe/upload.py

@@ -415,14 +415,23 @@ def run_upload(
     rng = rng or random.Random()
     now = now or (lambda: datetime.now(UTC).isoformat(timespec="seconds"))
 
-    marker = cfg.cache_dir / "STUB_FIXTURES.marker"
+    # Two provenance markers guard the same fact from different angles: the
-    if marker.exists():
+    # cache marker travels with the stub XML (gitignored, so a fresh clone
+    # loses it), while data/STUB_DATA.marker is COMMITTED alongside the
+    # stub-derived CSVs — so a clone can never upload placeholder ids.
+    markers = [
+        cfg.cache_dir / "STUB_FIXTURES.marker",
+        cfg.data_dir / "STUB_DATA.marker",
+    ]
+    marker = next((m for m in markers if m.exists()), None)
+    if marker is not None:
         if not dry_run:
             typer.echo(
                 f"Refusing to upload: {marker} exists — every resolved "
                 "version_id is a synthetic stub placeholder. Once "
                 "BGG_API_TOKEN arrives: delete both cache dirs, re-record "
-                "fixtures, `resolve --force`, re-review, re-diff."
+                "fixtures, `resolve --force`, re-review, re-diff, then "
+                "delete the marker(s)."
             )
             raise typer.Exit(code=1)
         typer.echo(

tests/test_upload.py

@@ -290,3 +290,13 @@ def test_verify_checks_version_on_adds_and_updates():
 def test_verify_ignores_failed_rows():
     log = [_log_row(action="add", bgg_id="5", status="failed")]
     assert verify_uploads(log, []) == []
+
+
+def test_fresh_clone_marker_blocks_upload_without_cache_dir(tmp_path):
+    # A fresh clone has the committed data/STUB_DATA.marker but no
+    # gitignored bgg_cache/ at all — upload must still refuse.
+    cfg = _cfg(tmp_path)
+    (tmp_path / "STUB_DATA.marker").write_text("stub-derived CSVs")
+    _seed_data(tmp_path, to_add=[_add_row()])
+    with pytest.raises(typer.Exit):
+        run_upload(cfg, uploader=FakeUploader(), sleep=lambda s: None, now=NOW)