Docs catch-up + in-app Help page
README: the Catalog page description and a new "Fixing what the model gets wrong" section cover the curation actions (edit/split/remove) and their durability contract. Spec: amended-as-built notes — duplicate same-edition copies are now supported via explicit splits, review is both a TUI and the web app, and the data model lists the three curation stores plus the dismissed-tickets file. New Help page in the app (nav entry on every page): the six-stage flow, what each page is for, the curation semantics with undo instructions, a status-chip legend rendered with the real chips, keyboard shortcuts, and where the data lives on disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
This commit is contained in:
@@ -60,7 +60,7 @@ Non-secret knobs (`photos_dir`, `data_dir`, the vision model, the rate limit) li
|
||||
uv run bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
|
||||
```
|
||||
|
||||
Six pages in one local app: **Pipeline** (run stages, watch live output), **Photos** (drag-and-drop upload, gallery, reshoot tickets), **Review** (keyboard-first match and edition decisions), **Catalog** (every extracted title and its status), **Queue** (exactly what upload will do, plus its full log), and **Library** (your enriched collection, browsable once real BGG data lands). The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
||||
Six pages in one local app: **Pipeline** (run stages, watch live output), **Photos** (drag-and-drop upload, gallery, reshoot tickets), **Review** (keyboard-first match and edition decisions), **Catalog** (every extracted title, alphabetized, with curation: fix misreads, add cues, split multi-copy lines, remove non-games), **Queue** (exactly what upload will do, plus its full log), and **Library** (your enriched collection, browsable once real BGG data lands). The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
||||
|
||||
```sh
|
||||
uv run bggpipe extract # photos → titles.json (+ retake prompts)
|
||||
@@ -78,6 +78,10 @@ Each stage skips work it has already done; `--force`/`--refresh` flags redo it.
|
||||
|
||||
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them.
|
||||
|
||||
### Fixing what the model gets wrong
|
||||
|
||||
Vision reads aren't perfect, and you know things the photos don't show. The Catalog page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, and **remove** lines that aren't games at all. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store. The in-app **Help** page documents all of it.
|
||||
|
||||
### Taking good shelf photos
|
||||
|
||||
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again.
|
||||
|
||||
@@ -66,7 +66,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
|
||||
|
||||
### Stage 3 — `review`: Human review of ambiguous/unmatched
|
||||
|
||||
- A minimal local review flow. A TUI is fine (e.g., `rich`/`textual`), or a tiny localhost web page — builder's choice, but keep it dependency-light.
|
||||
- A minimal local review flow. A TUI is fine (e.g., `rich`/`textual`), or a tiny localhost web page — builder's choice, but keep it dependency-light. *(As built: both — a rich TUI and a six-page FastAPI app sharing one decision engine.)*
|
||||
- For each `ambiguous` item: show the raw title, source photo filename, and candidate list (name, year, type, BGG rank, owned count) → pick one, skip, or reject.
|
||||
- For each `unmatched` item: allow manual BGG ID entry or a free-text re-search.
|
||||
- For each `version_ambiguous` item: show the version candidates (version name, publisher, year, language) next to the edition cues from the photo → pick one, or mark `version_unknown`. Keep this pass optional/skippable — version review shouldn't block getting games uploaded.
|
||||
@@ -107,6 +107,8 @@ All artifacts are flat files in a `data/` directory — human-readable, git-frie
|
||||
|
||||
- `titles.json` — extraction output (stage 1)
|
||||
- `unidentified.json` — game boxes seen but not identified (stage 1); retake prompts
|
||||
- `title_edits.json`, `title_splits.json`, `title_removals.json` — durable human curation (corrected reads/cues, split-into-copies decisions, removed lines); replayed on every titles.json rebuild so re-extraction never undoes them
|
||||
- `unidentified_dismissed.json` — dismissed retake prompts (kept apart so rebuilds can't resurrect them)
|
||||
- `bgg_cache/` — cached XML API responses
|
||||
- `matches.csv` — the master matching table (stages 2–3)
|
||||
- `to_add.csv` — upload queue, new entries (stage 4)
|
||||
@@ -122,7 +124,7 @@ All artifacts are flat files in a `data/` directory — human-readable, git-frie
|
||||
|
||||
- **202 queue** on `/collection`: retry with backoff (2s, 5s, 10s, 30s; give up after ~5 tries with a clear message).
|
||||
- **HEIC photos** from iPhone: convert transparently.
|
||||
- **Duplicate copies**: a title appearing in multiple photos with consistent edition cues is one game (dedupe). But genuinely distinct editions of the same game ARE in scope — they stay separate entries end-to-end and become separate BGG collection entries. Identical duplicate copies of the same edition are out of scope (assume dedupe).
|
||||
- **Duplicate copies**: a title appearing in multiple photos with consistent edition cues is one game (dedupe). But genuinely distinct editions of the same game ARE in scope — they stay separate entries end-to-end and become separate BGG collection entries. *(Amended as built:)* identical duplicate copies of the same edition are supported too, as an explicit human decision — a photo-scoped "split into copies" recorded in `data/title_splits.json`, honored by every dedupe pass thereafter.
|
||||
- **Non-game items** on shelves (books, card sleeves, storage boxes): the vision prompt should be instructed to include only board/card games; anything that slips through will fail resolution and land in review.
|
||||
- **Special characters** in titles (é, colons, ampersands): normalize consistently on both sides of the match.
|
||||
- **Base game vs. expansion vs. new edition**: the most common failure mode. Bias toward surfacing these as `ambiguous` rather than auto-matching.
|
||||
|
||||
@@ -336,6 +336,13 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
border-radius: 999px; padding: .1rem .55rem; white-space: nowrap;
|
||||
border: 1px solid currentColor;
|
||||
}
|
||||
.card code {
|
||||
font-family: var(--font-mono); font-size: .82em;
|
||||
background: #fff; border: 1px solid var(--board-edge);
|
||||
border-radius: 4px; padding: 0 .3em;
|
||||
}
|
||||
.card ol { padding-left: 1.3rem; }
|
||||
.card ol li { margin: .35rem 0; }
|
||||
.chip.ok { background: #e2f2e4; color: var(--go-ink); }
|
||||
.chip.wait { background: var(--ticket); color: var(--gold-ink); }
|
||||
.chip.no { background: #fbe3da; color: var(--stop-ink); }
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
<h1>Help</h1>
|
||||
<div class="pagebar">
|
||||
<a href="#flow">The flow</a>
|
||||
<a href="#pages">Pages</a>
|
||||
<a href="#curation">Fixing the catalog</a>
|
||||
<a href="#statuses">Statuses</a>
|
||||
<a href="#keys">Keyboard</a>
|
||||
<a href="#files">Your data</a>
|
||||
</div>
|
||||
|
||||
<h2 id="flow">The flow: shelves → collection</h2>
|
||||
<div class="card">
|
||||
<p>Six stages, run from the <a href="/">Pipeline</a> page (or the CLI — both share all state and either can pick up where the other left off):</p>
|
||||
<ol>
|
||||
<li><b>extract</b> — every photo goes to Claude vision once; titles and edition cues (publisher, edition wording, year, language) land in the catalog. Boxes it can see but can't read become <b>reshoot tickets</b> on the Photos page.</li>
|
||||
<li><b>resolve</b> — titles are matched to BoardGameGeek games and editions. Anything uncertain is flagged for review, never guessed. Without a BGG API token, titles wait as <i>awaiting BGG</i> and are picked up automatically once the token exists.</li>
|
||||
<li><b>review</b> — you decide the flagged ones on the <a href="/review">Review</a> page. Every decision saves immediately.</li>
|
||||
<li><b>diff</b> — your existing BGG collection is fetched and compared, per copy. What's genuinely new lands in the <a href="/queue">Queue</a>.</li>
|
||||
<li><b>upload</b> — a real browser logs into BGG and adds each queued game, slowly and politely. Always dry-run first; the buttons enforce that order.</li>
|
||||
<li><b>enrich</b> — full metadata (players, weight, rank, artwork) fills the <a href="/library">Library</a>.</li>
|
||||
</ol>
|
||||
<p>Every stage is resumable: stop anything mid-run and nothing is lost. Re-runs skip work already done.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="pages">What each page is for</h2>
|
||||
<div class="card">
|
||||
<p><b><a href="/">Pipeline</a></b> — run stages one at a time and watch their live output. Shows what's blocking (missing keys, stub data) and the counts at every step.</p>
|
||||
<p><b><a href="/photos">Photos</a></b> — drag photos in (or drop them in the <code>photos/</code> folder). Each photo has its own page listing every title read from it and any reshoot tickets — boxes seen but not identified. Photograph those up close, drop the new shot in, and extract again. Re-uploading a photo with the same name re-extracts it.</p>
|
||||
<p><b><a href="/review">Review</a></b> — the decisions only you can make: which game a title is, which edition a copy is, and whether two same-game reads are really one box (merges show a veto). Keyboard-first; see <a href="#keys">shortcuts</a>.</p>
|
||||
<p><b><a href="/catalog">Catalog</a></b> — every title read from your shelves, alphabetized, with its status and photos. This is also where you <a href="#curation">fix the data itself</a>: edit, split, remove.</p>
|
||||
<p><b><a href="/queue">Queue</a></b> — exactly what upload will do (new entries and version upgrades) and the log of everything it has done. Nothing reaches BGG that isn't visible here first.</p>
|
||||
<p><b><a href="/library">Library</a></b> — your enriched collection: filter by board games or RPGs. RPG matches are identified and enriched but never uploaded — BGG collections can't hold them, so they stay local citizens.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="curation">Fixing the catalog: edit, split, remove</h2>
|
||||
<div class="card">
|
||||
<p>Vision reads aren't perfect, and you know things the photos don't show. Every catalog line has curation actions, and every one of them is <b>durable</b>: the decision is saved in a small committed file and replayed on every rebuild, so re-running extract or resolve can never undo it.</p>
|
||||
<p><b>edit</b> — fix a misread title or add cues you already know (publisher, edition, year, language). A corrected misspelling automatically merges with a correctly-read sighting of the same game from another photo. If the line already had a BGG match, saving re-queues it so resolve searches again with the corrected data.</p>
|
||||
<p><b>split into copies</b> — one line, several physical boxes? Splitting makes each photo its own copy, and each copy picks its own edition afterward. Appears on any line whose title was seen in more than one photo. Splitting one game never affects a same-named different edition.</p>
|
||||
<p><b>remove from catalog</b> (inside the edit panel) — for lines that shouldn't exist at all: a book read as a game, box art misread as a title. The line and its matches are discarded and stay gone. This is different from <i>reject</i> on the Review page, which keeps the line visible as "no BGG match" — right for real games BGG doesn't know.</p>
|
||||
<p>Undo: each decision is one record in <code>data/title_edits.json</code>, <code>data/title_splits.json</code>, or <code>data/title_removals.json</code> — delete the record and the next rebuild restores the old state.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="statuses">What the status chips mean</h2>
|
||||
<div class="card">
|
||||
<p><span class="chip wait">awaiting BGG</span> extracted, not yet matched — usually waiting on the BGG API token.</p>
|
||||
<p><span class="chip ok">auto</span> matched confidently, no review needed. <span class="chip ok">approved</span> you picked the match yourself.</p>
|
||||
<p><span class="chip open">ambiguous</span> several plausible games — needs your pick on Review. <span class="chip open">unmatched</span> nothing plausible found — enter a BGG id or re-search on Review.</p>
|
||||
<p><span class="chip merged">merged</span> two reads judged to be the same physical box; the merge is veto-able on Review. <span class="chip merged">copy</span> one copy of a title you split.</p>
|
||||
<p><span class="chip no">rejected</span> you ruled it's not on BGG (or not a game worth matching); it stays listed but goes no further.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="keys">Keyboard shortcuts</h2>
|
||||
<div class="card">
|
||||
<p><b>Review:</b> <kbd>j</kbd>/<kbd>k</kbd> move between cards · <kbd>1</kbd>–<kbd>9</kbd> pick a candidate · <kbd>r</kbd> reject · <kbd>m</kbd> manual BGG id · <kbd>u</kbd> edition unknown · <kbd>v</kbd> veto a merge.</p>
|
||||
<p><b>Photo pages:</b> <kbd>←</kbd>/<kbd>→</kbd> move between photos.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="files">Your data, on disk</h2>
|
||||
<div class="card">
|
||||
<p>Everything lives in flat files under <code>data/</code> — inspectable, hand-editable, and git-friendly. The pipeline artifacts: <code>titles.json</code> (what was read), <code>matches.csv</code> (what it matched), <code>to_add.csv</code>/<code>to_update.csv</code> (what upload will do), <code>upload_log.csv</code> (what it did), <code>games.json</code> (the library). Your curation: <code>title_edits.json</code>, <code>title_splits.json</code>, <code>title_removals.json</code>, <code>unidentified_dismissed.json</code>.</p>
|
||||
<p>Credentials never live in files — only environment variables, set up by <code>bggpipe init</code>. The app runs on localhost only.</p>
|
||||
<p>More depth: the README covers setup and photo technique; <code>docs/bgg-upload-flow.md</code> documents the upload automation.</p>
|
||||
</div>
|
||||
@@ -235,6 +235,7 @@ NAV_PAGES = (
|
||||
("/catalog", "catalog", "Catalog", ""),
|
||||
("/queue", "queue", "Queue", "queue"),
|
||||
("/library", "library", "Library", ""),
|
||||
("/help", "help", "Help", ""),
|
||||
)
|
||||
|
||||
PHOTO_SUFFIXES = {".jpg", ".jpeg", ".png", ".heic"}
|
||||
@@ -555,6 +556,10 @@ def create_app(
|
||||
def library_page() -> str:
|
||||
return render_page("library")
|
||||
|
||||
@app.get("/help", response_class=HTMLResponse)
|
||||
def help_page() -> str:
|
||||
return render_page("help")
|
||||
|
||||
@app.get("/api/photos-list")
|
||||
def api_photos_list() -> list[dict]:
|
||||
out = []
|
||||
|
||||
@@ -766,3 +766,12 @@ def test_remove_title_is_durable_and_drops_vetoed_rows(tmp_path):
|
||||
).status_code
|
||||
== 404
|
||||
)
|
||||
|
||||
|
||||
def test_help_page_serves_and_is_in_nav(tmp_path):
|
||||
web, _ = make_client(tmp_path)
|
||||
res = web.get("/help")
|
||||
assert res.status_code == 200
|
||||
assert "Fixing the catalog" in res.text
|
||||
# every page's nav includes the Help entry
|
||||
assert 'href="/help"' in web.get("/").text
|
||||
|
||||
Reference in New Issue
Block a user