Compare commits
@@ -51,7 +51,20 @@ The first `/collection` call typically returns **HTTP 202** with a "please retry
|
||||
|
||||
## Search & matching heuristics
|
||||
|
||||
1. Exact normalized-name match → strong candidate.
|
||||
0. BGG's search is UNRELIABLE for generic and punctuated queries: results
|
||||
truncate in the several-hundreds unordered (the game named "Dungeon!"
|
||||
appears in neither the "Dungeon!" nor the "Dungeon" search), and
|
||||
punctuation can hide matches. Every title is searched raw AND with
|
||||
punctuation stripped, merged by id. A LONE surviving candidate never
|
||||
auto-matches unless it's a true exact match whose owned-count clears
|
||||
the dominance floor — the sole survivor may be an impostor standing
|
||||
where a famous game should be, and the real match may need review's
|
||||
manual-id entry.
|
||||
1. Exact normalized-name match → strong candidate. A name that becomes
|
||||
exact once its trailing parenthetical is stripped ("Wiz-War (Eighth
|
||||
Edition)") is a SIBLING EDITION — BGG files new editions as separate
|
||||
games — and counts as exact-grade so the choice between lineages
|
||||
reaches review instead of hiding behind a confident auto.
|
||||
2. Fuzzy match: `rapidfuzz` `token_sort_ratio ≥ 90` → good candidate.
|
||||
3. Ties: fetch `/thing?stats=1` for top ~5 candidates; prefer higher owned-count / better BGG rank (well-known game beats obscure duplicate of the same name).
|
||||
- Search results include `boardgameexpansion` as a distinct `type` — keep expansions but tag them so review catches base/expansion confusion.
|
||||
|
||||
@@ -8,4 +8,6 @@ BGG_USERNAME=
|
||||
BGG_PASSWORD=
|
||||
|
||||
# Anthropic API key for the vision extract stage
|
||||
# default vision provider; not needed for an openai-compatible/local
|
||||
# setup — see the [vision.*] blocks in config.toml
|
||||
ANTHROPIC_API_KEY=
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Secrets & credential-adjacent state
|
||||
.env
|
||||
*.storage_state.json
|
||||
data/.lan_key
|
||||
data/.own_repo
|
||||
playwright/.auth/
|
||||
storage_state.json
|
||||
|
||||
|
||||
@@ -9,19 +9,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
| # | Command | What it does | Status |
|
||||
|---|---------|--------------|--------|
|
||||
| 1 | `bggpipe extract` | Claude vision reads titles + edition cues from `photos/` | working |
|
||||
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (stub data — see hard rules) |
|
||||
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (real API data since 2026-08-05) |
|
||||
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
|
||||
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
|
||||
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | built; browser flows unverified until real data exists (`--dry-run` works now) |
|
||||
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | working — all browser flows verified live 2026-08-06 (62 adds + 36 version updates landed) |
|
||||
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
|
||||
|
||||
Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
|
||||
Full design lives in `docs/spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
|
||||
|
||||
## Commands
|
||||
|
||||
- `uv sync` — install deps (Python 3.12+, managed by **uv**; use `uv add`, never pip). `uv run bggpipe init` handles first-run setup (folders, .env credentials, the one-time `playwright install chromium`).
|
||||
- `uv run bggpipe web` — the app: six pages (Pipeline `/`, Photos, Review, Catalog, Queue, Library) in a shared sidebar shell; stage runs execute one-at-a-time in a background job.
|
||||
- `uv run bggpipe <stage>` — run a pipeline stage. Non-secret settings come from `config.toml` (username, dirs, vision model, rate limit); `--config` overrides the path.
|
||||
- `uv run bggpipe web` — the app: seven pages (Pipeline `/`, Photos, Titles, Review, Queue, Library, Help) in a shared sidebar shell (responsive: hamburger nav + stacked tables under 900px); stage runs execute one-at-a-time in a background job. `--lan` binds 0.0.0.0 behind a per-device access key: persisted in `data/.lan_key` (gitignored), printed as a QR at startup, cookie-paired for a year, required on EVERY network request (loopback clients and `/static/*` are exempt; the Host/Origin guard still applies). Phone camera uploads (generic `image.jpg` names) get minted `shelf-<timestamp>` names — only explicitly-named files trigger the replace-to-reshoot flow.
|
||||
- `uv run bggpipe <stage>` — run a pipeline stage. Non-secret settings come from `config.toml` (dirs, rate limit, `vision_provider` + per-provider `[vision.*]` blocks — "anthropic" or any OpenAI-compatible endpoint incl. local Ollama); `--config` overrides the path.
|
||||
- `uv run pytest` — the suite runs fully offline against fixtures. Tests marked `live` hit the real BGG API (read-only) and are skipped unless you pass `--run-live`.
|
||||
- `uv run ruff check` / `uv run ruff format` — lint (rules E, F, I, UP, B, SIM) and format.
|
||||
|
||||
@@ -29,29 +29,29 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
|
||||
|
||||
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`); `templates/shell.html` + `templates/pages/*` + `static/app.{css,js}` are the web UI (the stylesheet is the design system — tokens derive from the mascot art), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `jobs.py` (single-slot background stage runner for the web UI), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`, `fsio.py` (atomic writes), `init_wizard.py` (first-run setup).
|
||||
- `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
|
||||
- `tests/fixtures/bgg_cache/` — stub XML fixtures the offline tests run against.
|
||||
- `tests/fixtures/bgg_cache/` — recorded real XML fixtures the offline tests run against.
|
||||
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).
|
||||
|
||||
## Hard rules (from spec — never violate)
|
||||
|
||||
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 2–4 s randomized delay between games.
|
||||
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD`, `BGG_API_TOKEN` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
|
||||
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. Until Eric's registration at boardgamegeek.com/applications is approved, tests run on the stub fixtures in `tests/fixtures/bgg_cache/`; re-record them with `scripts/record_fixtures.py` once the token exists.
|
||||
- **The XML API requires a registered app token** (`Authorization: Bearer`, from `BGG_API_TOKEN`) — unregistered requests get 401. The token is ACTIVE and `tests/fixtures/bgg_cache/` holds real recorded responses; add fixtures for new tests with `scripts/record_fixtures.py`.
|
||||
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
|
||||
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
|
||||
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
|
||||
- **Stub-resolved data is never upload-ready.** All version_ids (and some game data) in `matches.csv`, `to_add.csv`, and `to_update.csv` currently come from SYNTHETIC stub fixtures — placeholders until real fixtures exist. When `BGG_API_TOKEN` arrives: delete both cache dirs, re-record fixtures, `resolve --force`, re-review. Two provenance markers guard this (both written by the fixture generators): `data/bgg_cache/STUB_FIXTURES.marker` (gitignored, travels with the stub XML) and `data/STUB_DATA.marker` (**committed**, so a fresh clone stays guarded). The upload stage MUST refuse to run while either exists; delete `data/STUB_DATA.marker` only after re-resolving from real fixtures.
|
||||
- **Synthetic data must never reach upload.** The stub era ended 2026-08-05: `matches.csv` and `tests/fixtures/bgg_cache/` now hold real API data. The guard mechanism stays armed: the stub-fixture generators write `data/bgg_cache/STUB_FIXTURES.marker` (gitignored) and `data/STUB_DATA.marker` (committed), and the upload stage MUST refuse to run while either exists — regenerating stubs re-locks upload automatically.
|
||||
|
||||
## Domain gotchas
|
||||
|
||||
- Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan).
|
||||
- Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry.
|
||||
- Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance.
|
||||
- **Human curation is durable**: `data/title_splits.json` (photo-scoped split-into-copies decisions, honored by extract's dedupe AND resolve's dedupe) and `data/title_edits.json` (corrected reads/cues, applied before dedupe on every titles.json rebuild) persist forever. Row-level decisions persist via the `dedupe_veto` column — edits never drop veto'd rows (a rename retitles them in place).
|
||||
- **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope).
|
||||
- **Human curation is durable**: `data/title_splits.json` (photo-scoped split-into-copies decisions, honored by extract's dedupe AND resolve's dedupe), `data/title_edits.json` (corrected reads/cues, applied before dedupe on every titles.json rebuild), `data/title_removals.json` (lines removed from the catalog — filtered out of every rebuild; delete the record to undo), `data/title_additions.json` (games added without a photo — joined into every rebuild; a later photo sighting dedupe-merges with them), and `data/local_games.json` + `data/local_art/` (hand-written facts and a cover photo for off-BGG games — the ONLY source for them, merged over the photo reads by enrich) persist forever. Row-level decisions persist via the `dedupe_veto` column — edits never drop veto'd rows (a rename retitles them in place); removal drops them (explicitly discarding the line).
|
||||
- **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). RPGGeek items carry their OWN link types (`rpgdesigner`, `rpgpublisher`, `rpggenre`, `rpgcategory`, `rpgmechanic`) — a board-game-only parser silently returns nothing for them. Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope).
|
||||
- Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift.
|
||||
|
||||
## Git
|
||||
|
||||
- Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub** — `gh` CLI does not work here.
|
||||
- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/title_splits.json`, `data/title_edits.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`.
|
||||
- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/title_splits.json`, `data/title_edits.json`, `data/title_removals.json`, `data/title_additions.json`, `data/local_games.json`, `data/local_art/`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, `data/.lan_key`, Playwright storage state, or `.env`.
|
||||
|
||||
@@ -11,105 +11,69 @@ photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
|
||||
→ [6 enrich] → games.json
|
||||
```
|
||||
|
||||
> **Status: working, not yet battle-tested.** All six stages are implemented with an offline test suite. The upload stage's browser flows follow documented selectors but await their first real run — start with `--dry-run`, then `--limit 1`. (This repo also carries its author's in-progress pipeline data; see [Bring your own shelves](#bring-your-own-shelves).)
|
||||
> **Status: battle-tested end to end.** The full pipeline has run against a live BGG account: shelf photos → 136 identified games → 62 additions and 36 version updates on a real collection. The result is public — [the author's collection on BGG](https://boardgamegeek.com/collection/user/ewagoner) is what this pipeline built. Still sensible on a first run: `--dry-run`, then `--limit 1`.
|
||||
|
||||
## Why this exists
|
||||
|
||||
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done.
|
||||
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. I wanted to point a camera at my shelves instead. This pipeline replaces the typing with: take photos, run a command, settle a handful of ambiguous matches in a review step, done — and every judgment call along the way is yours, made in a review UI, never guessed by the machine.
|
||||
|
||||
## How it works
|
||||
|
||||
1. **extract** — Shelf photos go to the Anthropic API (Claude vision), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
|
||||
1. **extract** — Shelf photos go to a vision model (Claude by default; any OpenAI-compatible endpoint or a local Ollama model works), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
|
||||
2. **resolve** — Titles are matched to BGG game IDs via the [XML API2](https://boardgamegeek.com/wiki/page/BGG_XML_API2) (exact + fuzzy matching, popularity tiebreaks), then edition cues are matched against BGG's version list for each game. Anything uncertain is flagged rather than guessed.
|
||||
3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
|
||||
4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
|
||||
5. **upload** — A Playwright browser session logs into your BGG account and adds each game (with its version, when known) politely and slowly. Dry-run mode, per-game logging, and resumability included.
|
||||
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in `data/games.json`, the seed data for a future web frontend.
|
||||
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork, version details) lands in `data/games.json`, feeding a browsable library of your shelves.
|
||||
|
||||
Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit.
|
||||
Everything runs locally, every stage survives being killed mid-run, and all artifacts are flat CSV/JSON files you can inspect and edit. You can drive it from the terminal or from a local web app:
|
||||
|
||||

|
||||
|
||||
**➔ [See the full tour](docs/tour.md)** — all seven pages, the game-detail view, and the phone experience.
|
||||
|
||||
## Requirements
|
||||
|
||||
- macOS or Linux, Python 3.12+, [uv](https://docs.astral.sh/uv/)
|
||||
- An [Anthropic API key](https://console.anthropic.com/) (vision extraction)
|
||||
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared.
|
||||
- Python 3.12+ and [uv](https://docs.astral.sh/uv/), on macOS, Linux, or Windows. (Development happens on macOS; Windows is untested but nothing is platform-specific. One caveat: the owner-only file permissions bggpipe sets on `.env` and browser session state are POSIX-only — on Windows, keep those files in a directory protected by your account.)
|
||||
- A vision model for extraction — an [Anthropic API key](https://console.anthropic.com/) by default, or any OpenAI-compatible endpoint including a free local [Ollama](https://ollama.com/) model. **Cost expectation:** extracting the author's whole collection — 65 shelf photos, 136 games — cost under a dollar with the default model (Claude Sonnet), one-time. Re-runs are free: every photo's read is cached, and only new or replaced photos go back to the model.
|
||||
- A BoardGameGeek account **and a registered BGG application** — as of BGG's [2025 API policy](https://boardgamegeek.com/using_the_xml_api), the XML API requires a Bearer token from a registered app. Register a free non-commercial application at [boardgamegeek.com/applications](https://boardgamegeek.com/applications) (approval can take a week or more, so **apply on day one**), then create a token. Each user needs their own; tokens must not be shared. ([You can start before it arrives.](docs/guide.md#running-before-your-bgg-token-arrives))
|
||||
|
||||
### Why it asks for your BGG password — and where your credentials go
|
||||
|
||||
BGG has no write API: the **only** way to add games to a collection is the website itself. So the upload stage signs into boardgamegeek.com in a real browser window, on your machine, and clicks the same buttons you would — you can literally watch it work (the browser is visible by default). That's the whole reason the password is needed, and it's used for exactly that login and nothing else.
|
||||
|
||||
Everything stays on your computer. There is no bggpipe server, no telemetry, no analytics, and no account with me — I never see your credentials, your collection, or anything else, and the code is right here to check. Credentials live in a local `.env` file (owner-only permissions, never written to logs) and are sent only to boardgamegeek.com itself. The only other service the pipeline ever contacts is the vision provider *you* configure, which receives your shelf photos and nothing more — and with a local Ollama model, even those never leave the house.
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
uv tool install git+https://git.kestrelsnest.social/eric/bggpipe
|
||||
mkdir shelves && cd shelves # any directory of your own — NOT a clone of this repo
|
||||
bggpipe init # guided setup: folders, credentials, browser download
|
||||
bggpipe web # the whole app at http://127.0.0.1:8377/
|
||||
```
|
||||
|
||||
Drop shelf photos on the Photos page (or straight [from your phone's camera](docs/guide.md#from-your-phone)) and follow the pipeline left to right. Your photos and every pipeline artifact live in the directory where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data — which is also why running inside a clone of this repo is the one unsupported setup: this repo carries its author's live pipeline data at git-tracked paths, and [a `git pull` on top of yours could destroy it](docs/guide.md#keeping-your-data-safe-from-git). bggpipe warns if it catches you doing this.
|
||||
|
||||
**➔ [The user's guide](docs/guide.md)** — credentials, every stage and flag, phone pairing, RPG handling, fixing misreads, uploading safely, and running before your token arrives.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
git clone https://git.kestrelsnest.social/eric/bggpipe.git
|
||||
cd bggpipe
|
||||
uv sync # installs Python deps
|
||||
uv run bggpipe init # guided setup: folders, credentials, browser download
|
||||
```
|
||||
|
||||
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? `cp .env.example .env`, fill it in, and run `uv run playwright install chromium` yourself.
|
||||
|
||||
Secrets live in environment variables only, never in config files, code, or logs. `.env` is gitignored. If you use [direnv](https://direnv.net/), the committed `.envrc` loads `.env` automatically after a one-time `direnv allow`; otherwise export the variables yourself (e.g. `set -a; source .env; set +a`).
|
||||
|
||||
| Variable | Used by | What it is |
|
||||
|---|---|---|
|
||||
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
|
||||
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
|
||||
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
|
||||
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
|
||||
|
||||
Non-secret knobs (`photos_dir`, `data_dir`, the vision model, the rate limit) live in `config.toml`. From here you can drive everything from the browser:
|
||||
|
||||
```sh
|
||||
uv run bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
|
||||
```
|
||||
|
||||
Six pages in one local app: **Pipeline** (run stages, watch live output), **Photos** (drag-and-drop upload, gallery, reshoot tickets), **Review** (keyboard-first match and edition decisions), **Catalog** (every extracted title and its status), **Queue** (exactly what upload will do, plus its full log), and **Library** (your enriched collection, browsable once real BGG data lands). The real upload sits behind a confirmation and behind the stub-data lock. Prefer the terminal? Every stage is also a command, and the two interfaces share all state:
|
||||
|
||||
```sh
|
||||
uv run bggpipe extract # photos → titles.json (+ retake prompts)
|
||||
uv run bggpipe resolve # titles → BGG ids/versions in matches.csv
|
||||
uv run bggpipe review --web # review UI only
|
||||
uv run bggpipe diff # compare against your BGG collection
|
||||
uv run bggpipe upload --dry-run # ALWAYS inspect this first
|
||||
uv run bggpipe upload --limit 1 # then one game, then small batches
|
||||
uv run bggpipe enrich # full metadata → data/games.json
|
||||
```
|
||||
|
||||
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. `review` without `--web` runs in the terminal instead. `upload` also has `--retry-failed`, `--verify` (re-fetches your collection and cross-checks the log), and runs a **headed** browser by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
|
||||
|
||||
### RPGs on your shelves
|
||||
|
||||
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched, and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them.
|
||||
|
||||
### Taking good shelf photos
|
||||
|
||||
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the review UI's "reshoot" list: photograph those boxes up close, drop the new photo in `photos/`, and run `extract` again.
|
||||
|
||||
## Bring your own shelves
|
||||
|
||||
This repo doubles as its author's live pipeline, so `data/` ships with his real artifacts — extracted titles, matches, and 2018 collection snapshots. Before running against *your* shelves, clear the data:
|
||||
|
||||
```sh
|
||||
rm data/*.csv data/*.json data/*.xml data/STUB_DATA.marker
|
||||
rm -rf data/bgg_cache data/extract_raw
|
||||
```
|
||||
|
||||
Two of those files deserve a word:
|
||||
|
||||
- **`data/STUB_DATA.marker`** — the committed CSVs were resolved from *hand-written stub fixtures* (the author's BGG application is still awaiting approval), so every version id in them is a synthetic placeholder. The upload stage refuses to run while this marker exists, precisely so nobody (including a fresh clone) can push placeholder data to a real BGG account. Starting fresh with your own token, you'll never see it again.
|
||||
- **`data/collection_snapshot_*.xml`** — with `BGG_API_TOKEN` set, `diff` fetches your collection live and you don't need these. Without a token (still waiting on approval?), you can use the logged-in-browser exemption: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
|
||||
|
||||
No token yet? `extract` works immediately (it only needs the Anthropic key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
uv run pytest # offline test suite (stub fixtures, no network)
|
||||
uv run pytest # offline test suite (recorded fixtures, no network)
|
||||
uv run pytest --run-live # + a read-only live-API smoke test (needs token)
|
||||
uv run ruff check src tests # lint
|
||||
uv run bggpipe review --web --dev # review UI with code hot-reload
|
||||
uv run bggpipe web --dev # the app with code hot-reload
|
||||
```
|
||||
|
||||
The review UI live-follows the data files — run `extract` or `resolve` in another terminal and the page updates itself. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [full spec](bgg-shelf-pipeline-spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
|
||||
Run your own *pipeline* from a different directory — the clone's `data/` is the author's live data. Architecture and contributor guidance: [CLAUDE.md](CLAUDE.md) and the [design contract](docs/spec.md); BGG automation notes: [docs/bgg-upload-flow.md](docs/bgg-upload-flow.md).
|
||||
|
||||
## Questions, bugs, ideas
|
||||
|
||||
Email [eric@ericwagoner.com](mailto:eric@ericwagoner.com), or find me on Mastodon at [@eric@toots.kestrelsnest.social](https://toots.kestrelsnest.social/@eric) or Bluesky at [@kestrelsnest.social](https://bsky.app/profile/kestrelsnest.social) — bug reports, confusions, and "it worked, here's my collection" notes all welcome. (This Gitea instance doesn't take public registrations, so there's no issue tracker to file into yet; if enough people show up, one will materialize.)
|
||||
|
||||
## A note on being a good BGG citizen
|
||||
|
||||
|
||||
@@ -3,5 +3,19 @@
|
||||
|
||||
photos_dir = "photos"
|
||||
data_dir = "data"
|
||||
model = "claude-sonnet-5"
|
||||
rate_limit_seconds = 2.0
|
||||
|
||||
# Which vision backend reads your shelf photos. Both recipes below stay
|
||||
# on file; this line picks one.
|
||||
vision_provider = "anthropic"
|
||||
|
||||
[vision.anthropic]
|
||||
# reads ANTHROPIC_API_KEY from the environment
|
||||
model = "claude-sonnet-5"
|
||||
|
||||
[vision."openai-compatible"]
|
||||
# OpenAI, OpenRouter, or a local runtime (Ollama, LM Studio, vLLM).
|
||||
# key_env names the env var holding the key; "" = endpoint needs none.
|
||||
base_url = "http://localhost:11434/v1"
|
||||
model = "qwen2.5vl:7b"
|
||||
key_env = ""
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
The CSVs in this directory were resolved from hand-written stub fixtures, not real BGG data — version_ids are SYNTHETIC. The upload stage refuses to run while this file exists. Delete it only after re-resolving against real recorded fixtures (BGG_API_TOKEN + scripts/record_fixtures.py + resolve --force).
|
||||
@@ -0,0 +1,32 @@
|
||||
[
|
||||
{
|
||||
"title_raw": "Civilization: West Extension Map",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "",
|
||||
"art_notes": "",
|
||||
"source_photos": []
|
||||
},
|
||||
{
|
||||
"title_raw": "Artistry: Delightful Doorways Mini Expansion",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "",
|
||||
"art_notes": "",
|
||||
"source_photos": []
|
||||
},
|
||||
{
|
||||
"title_raw": "castle panic wizard's tower",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "",
|
||||
"art_notes": "",
|
||||
"source_photos": []
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,99 @@
|
||||
[
|
||||
{
|
||||
"match": "Hangermuger 4 800",
|
||||
"title_raw": "Huggermugger"
|
||||
},
|
||||
{
|
||||
"match": "DOCTOR WHO THE GAME OF TIME",
|
||||
"title_raw": "DOCTOR WHO THE card GAME"
|
||||
},
|
||||
{
|
||||
"match": "CAPTAIN MARVEL SECRET SKRULLS",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "CHECKERS",
|
||||
"title_raw": "Super Mario CHECKERS",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "DOCTOR WHO THE card GAME",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "DUNGEONS & DRAGONS",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "Huggermugger",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "Patch Work",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "The Ain't It Cool Trivia Game",
|
||||
"title_raw": "The Rocky Horror Trivia Game",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "Verdant",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "Wiz-War",
|
||||
"photos": [
|
||||
"IMG_4502.jpeg"
|
||||
],
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "ACTION CASTLE",
|
||||
"title_raw": "Parsely",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "Patch Work",
|
||||
"title_raw": "Patchwork",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "SMALLWORLD",
|
||||
"title_raw": "SMALL WORLD",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "ART NOUVEAU ARTISTRY",
|
||||
"title_raw": "ARTISTRY",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "MAGE WARS ACADEMY CORE SET",
|
||||
"title_raw": "MAGE WARS ACADEMY",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "MANTIS FALLS (a game of trust)",
|
||||
"title_raw": "MANTIS FALLS",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "BOTANY EXPANSION TANTALIZING TREES",
|
||||
"title_raw": "BOTANY: TANTALIZING TREES",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "BOTANY EXPANSION PERILOUS PERFUMES POISONOUS, CARNIVOROUS, PARASITIC AND BIZARRE PLANTS",
|
||||
"title_raw": "BOTANY: PERILOUS PERFUMES",
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"match": "WIZ-WAR",
|
||||
"photos": [
|
||||
"IMG_4504.jpeg"
|
||||
],
|
||||
"year_hint": 2012,
|
||||
"confidence": "high"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,86 @@
|
||||
[
|
||||
{
|
||||
"title": "THE CATAN CARD GAME",
|
||||
"photos": [
|
||||
"IMG_4523.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "SLUGFEST GAMES",
|
||||
"photos": [
|
||||
"IMG_4528.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "SCRABBLE",
|
||||
"photos": [
|
||||
"IMG_4508.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "SCERU MERRY MEN",
|
||||
"photos": [
|
||||
"IMG_4518.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Patchwork",
|
||||
"photos": [
|
||||
"IMG_4556.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Jokin Ha...",
|
||||
"photos": [
|
||||
"IMG_4506.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "CHEAPASS GAMES",
|
||||
"photos": [
|
||||
"IMG_4519.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "CATAN",
|
||||
"photos": [
|
||||
"IMG_4523.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "CAR WARS",
|
||||
"photos": [
|
||||
"IMG_4532.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Bird on Your Bread?!",
|
||||
"photos": [
|
||||
"IMG_4517.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Super Mario CHECKERS",
|
||||
"photos": [
|
||||
"IMG_4507.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "ACTION CASTLE I",
|
||||
"photos": [
|
||||
"IMG_4573.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "SIX-GUN SHOWDOWN",
|
||||
"photos": [
|
||||
"IMG_4573.jpeg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Blackboa[rd]",
|
||||
"photos": [
|
||||
"IMG_4573.jpeg"
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -22,7 +22,8 @@
|
||||
"art_notes": "Box top shown horizontally on shelf, tagline 'Who's to Blame Logic Game', orange cat illustration",
|
||||
"source_photos": [
|
||||
"IMG_4499.jpeg",
|
||||
"IMG_4505.jpeg"
|
||||
"IMG_4505.jpeg",
|
||||
"shelf-20260803-215723.jpg"
|
||||
],
|
||||
"title_normalized": "cat crimes"
|
||||
},
|
||||
@@ -158,8 +159,8 @@
|
||||
"title_normalized": "gentle rain"
|
||||
},
|
||||
{
|
||||
"title_raw": "Patch Work",
|
||||
"confidence": "medium",
|
||||
"title_raw": "Patchwork",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -168,7 +169,7 @@
|
||||
"source_photos": [
|
||||
"IMG_4501.jpeg"
|
||||
],
|
||||
"title_normalized": "patch work"
|
||||
"title_normalized": "patchwork"
|
||||
},
|
||||
{
|
||||
"title_raw": "SANTORINI",
|
||||
@@ -260,7 +261,7 @@
|
||||
},
|
||||
{
|
||||
"title_raw": "Wiz-War",
|
||||
"confidence": "medium",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -354,7 +355,7 @@
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Fantasy Flight Games",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"year_hint": 2012,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Dark spine with wizard/warrior artwork, sepia tones",
|
||||
"source_photos": [
|
||||
@@ -416,7 +417,7 @@
|
||||
"title_normalized": "onitama"
|
||||
},
|
||||
{
|
||||
"title_raw": "MANTIS FALLS (a game of trust)",
|
||||
"title_raw": "MANTIS FALLS",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Diligent Games",
|
||||
"edition_hint": "",
|
||||
@@ -426,20 +427,7 @@
|
||||
"source_photos": [
|
||||
"IMG_4505.jpeg"
|
||||
],
|
||||
"title_normalized": "mantis falls game of trust"
|
||||
},
|
||||
{
|
||||
"title_raw": "Jokin Ha...",
|
||||
"confidence": "medium",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "White box with simple stick-figure character illustration in blue shirt, partially obscured by a blue 'Nostalgia Pack' sleeve item and card deck; likely 'Joking Hazard' based on visible fragment and cyanide & happiness-style art",
|
||||
"source_photos": [
|
||||
"IMG_4506.jpeg"
|
||||
],
|
||||
"title_normalized": "jokin ha"
|
||||
"title_normalized": "mantis falls"
|
||||
},
|
||||
{
|
||||
"title_raw": "Herbaceous",
|
||||
@@ -468,19 +456,6 @@
|
||||
],
|
||||
"title_normalized": "labyrinth"
|
||||
},
|
||||
{
|
||||
"title_raw": "CHECKERS",
|
||||
"confidence": "medium",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Top-left corner box, red/black checkered pattern visible",
|
||||
"source_photos": [
|
||||
"IMG_4507.jpeg"
|
||||
],
|
||||
"title_normalized": "checkers"
|
||||
},
|
||||
{
|
||||
"title_raw": "Super Mario Checkers Collector's Edition",
|
||||
"confidence": "high",
|
||||
@@ -522,19 +497,6 @@
|
||||
],
|
||||
"title_normalized": "bugs in kitchen"
|
||||
},
|
||||
{
|
||||
"title_raw": "SCRABBLE",
|
||||
"confidence": "medium",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Red box edge visible top right, only partial letters 'RABBLE' legible",
|
||||
"source_photos": [
|
||||
"IMG_4508.jpeg"
|
||||
],
|
||||
"title_normalized": "scrabble"
|
||||
},
|
||||
{
|
||||
"title_raw": "CATAN DICE GAME",
|
||||
"confidence": "high",
|
||||
@@ -718,7 +680,7 @@
|
||||
"title_normalized": "doom that came to atlantic city"
|
||||
},
|
||||
{
|
||||
"title_raw": "SMALLWORLD",
|
||||
"title_raw": "SMALL WORLD",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Days of Wonder",
|
||||
"edition_hint": "",
|
||||
@@ -728,7 +690,7 @@
|
||||
"source_photos": [
|
||||
"IMG_4513.jpeg"
|
||||
],
|
||||
"title_normalized": "smallworld"
|
||||
"title_normalized": "small world"
|
||||
},
|
||||
{
|
||||
"title_raw": "Flick'em UP!",
|
||||
@@ -783,8 +745,8 @@
|
||||
"title_normalized": "dixit"
|
||||
},
|
||||
{
|
||||
"title_raw": "Hangermuger 4 800",
|
||||
"confidence": "low",
|
||||
"title_raw": "Huggermugger",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -793,11 +755,11 @@
|
||||
"source_photos": [
|
||||
"IMG_4514.jpeg"
|
||||
],
|
||||
"title_normalized": "hangermuger 4 800"
|
||||
"title_normalized": "huggermugger"
|
||||
},
|
||||
{
|
||||
"title_raw": "The Ain't It Cool Trivia Game",
|
||||
"confidence": "medium",
|
||||
"title_raw": "The Rocky Horror Trivia Game",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -806,10 +768,10 @@
|
||||
"source_photos": [
|
||||
"IMG_4514.jpeg"
|
||||
],
|
||||
"title_normalized": "ain t it cool trivia game"
|
||||
"title_normalized": "rocky horror trivia game"
|
||||
},
|
||||
{
|
||||
"title_raw": "ART NOUVEAU ARTISTRY",
|
||||
"title_raw": "ARTISTRY",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Dux Somnium",
|
||||
"edition_hint": "",
|
||||
@@ -819,7 +781,7 @@
|
||||
"source_photos": [
|
||||
"IMG_4515.jpeg"
|
||||
],
|
||||
"title_normalized": "art nouveau artistry"
|
||||
"title_normalized": "artistry"
|
||||
},
|
||||
{
|
||||
"title_raw": "SPEECHLESS",
|
||||
@@ -875,7 +837,7 @@
|
||||
},
|
||||
{
|
||||
"title_raw": "CAPTAIN MARVEL SECRET SKRULLS",
|
||||
"confidence": "medium",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Marvel",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -899,19 +861,6 @@
|
||||
],
|
||||
"title_normalized": "pie face"
|
||||
},
|
||||
{
|
||||
"title_raw": "Bird on Your Bread?!",
|
||||
"confidence": "low",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Blue box with cartoon bird/character illustration, tall narrow spine",
|
||||
"source_photos": [
|
||||
"IMG_4517.jpeg"
|
||||
],
|
||||
"title_normalized": "bird on your bread"
|
||||
},
|
||||
{
|
||||
"title_raw": "Walk the Plank!",
|
||||
"confidence": "high",
|
||||
@@ -953,7 +902,7 @@
|
||||
"title_normalized": "meteor"
|
||||
},
|
||||
{
|
||||
"title_raw": "MAGE WARS ACADEMY CORE SET",
|
||||
"title_raw": "MAGE WARS ACADEMY",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Arcane Wonders",
|
||||
"edition_hint": "Core Set",
|
||||
@@ -963,24 +912,11 @@
|
||||
"source_photos": [
|
||||
"IMG_4518.jpeg"
|
||||
],
|
||||
"title_normalized": "mage wars academy core set"
|
||||
"title_normalized": "mage wars academy"
|
||||
},
|
||||
{
|
||||
"title_raw": "SCERU MERRY MEN",
|
||||
"confidence": "medium",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Orange box with dinosaur/creature illustration, fantasy medieval theme art",
|
||||
"source_photos": [
|
||||
"IMG_4518.jpeg"
|
||||
],
|
||||
"title_normalized": "sceru merry men"
|
||||
},
|
||||
{
|
||||
"title_raw": "DOCTOR WHO THE GAME OF TIME",
|
||||
"confidence": "medium",
|
||||
"title_raw": "DOCTOR WHO THE card GAME",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -989,11 +925,11 @@
|
||||
"source_photos": [
|
||||
"IMG_4518.jpeg"
|
||||
],
|
||||
"title_normalized": "doctor who game of time"
|
||||
"title_normalized": "doctor who card game"
|
||||
},
|
||||
{
|
||||
"title_raw": "DUNGEONS & DRAGONS",
|
||||
"confidence": "medium",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "TSR",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -1017,19 +953,6 @@
|
||||
],
|
||||
"title_normalized": "dungeons and dragons fantasy game rules"
|
||||
},
|
||||
{
|
||||
"title_raw": "CHEAPASS GAMES",
|
||||
"confidence": "low",
|
||||
"publisher_hint": "Cheapass Games",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Vertical white spine text near right side of shelf, appears to be publisher branding rather than a title",
|
||||
"source_photos": [
|
||||
"IMG_4519.jpeg"
|
||||
],
|
||||
"title_normalized": "cheapass games"
|
||||
},
|
||||
{
|
||||
"title_raw": "Before I Kill You, Mister Bond...",
|
||||
"confidence": "high",
|
||||
@@ -1161,32 +1084,6 @@
|
||||
],
|
||||
"title_normalized": "sleeping gods distant skies"
|
||||
},
|
||||
{
|
||||
"title_raw": "CATAN",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Catan",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Red spine, small circular emblem visible on spine",
|
||||
"source_photos": [
|
||||
"IMG_4523.jpeg"
|
||||
],
|
||||
"title_normalized": "catan"
|
||||
},
|
||||
{
|
||||
"title_raw": "THE CATAN CARD GAME",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Catan",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Red/orange spine with yellow banner text",
|
||||
"source_photos": [
|
||||
"IMG_4523.jpeg"
|
||||
],
|
||||
"title_normalized": "catan card game"
|
||||
},
|
||||
{
|
||||
"title_raw": "THE RIVALS FOR CATAN CARD GAME",
|
||||
"confidence": "high",
|
||||
@@ -1304,19 +1201,6 @@
|
||||
],
|
||||
"title_normalized": "wiz war"
|
||||
},
|
||||
{
|
||||
"title_raw": "SLUGFEST GAMES",
|
||||
"confidence": "low",
|
||||
"publisher_hint": "Slugfest Games",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Partial box visible at top of image, red/orange background with illustrated characters, this may be a publisher logo rather than a game title",
|
||||
"source_photos": [
|
||||
"IMG_4528.jpeg"
|
||||
],
|
||||
"title_normalized": "slugfest games"
|
||||
},
|
||||
{
|
||||
"title_raw": "RISK THE LORD OF THE RINGS: The Middle-earth Conquest Game",
|
||||
"confidence": "high",
|
||||
@@ -1382,19 +1266,6 @@
|
||||
],
|
||||
"title_normalized": "gloomhaven"
|
||||
},
|
||||
{
|
||||
"title_raw": "CAR WARS",
|
||||
"confidence": "medium",
|
||||
"publisher_hint": "sjgames.com",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Red box top left showing miniature cars, partially visible at top of frame",
|
||||
"source_photos": [
|
||||
"IMG_4532.jpeg"
|
||||
],
|
||||
"title_normalized": "car wars"
|
||||
},
|
||||
{
|
||||
"title_raw": "DEADLY DOODLES",
|
||||
"confidence": "high",
|
||||
@@ -1436,7 +1307,7 @@
|
||||
},
|
||||
{
|
||||
"title_raw": "Verdant",
|
||||
"confidence": "medium",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "AEG",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
@@ -1720,19 +1591,6 @@
|
||||
],
|
||||
"title_normalized": "kill doctor lucky"
|
||||
},
|
||||
{
|
||||
"title_raw": "Patchwork",
|
||||
"confidence": "low",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Orange/tan box, only partial text '...TCH ...ORK' visible with quilt/patchwork square pattern",
|
||||
"source_photos": [
|
||||
"IMG_4556.jpeg"
|
||||
],
|
||||
"title_normalized": "patchwork"
|
||||
},
|
||||
{
|
||||
"title_raw": "ALICE IS MISSING SILENT FALLS EXPANSION",
|
||||
"confidence": "high",
|
||||
@@ -1758,5 +1616,168 @@
|
||||
"IMG_4566.jpeg"
|
||||
],
|
||||
"title_normalized": "alice is missing silent role playing game"
|
||||
},
|
||||
{
|
||||
"title_raw": "ETHERFIELDS KITTENBURG EXPANSION",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Awaken Realms",
|
||||
"edition_hint": "Expansion by Adrian Krawczyk",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Dark reddish-brown background with a large anthropomorphic cat figure in ornate robe, surrounded by smaller kittens, gothic fantasy style artwork",
|
||||
"source_photos": [
|
||||
"IMG_4570.jpeg"
|
||||
],
|
||||
"title_normalized": "etherfields kittenburg expansion"
|
||||
},
|
||||
{
|
||||
"title_raw": "BOTANY: TANTALIZING TREES",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Dux Somnium",
|
||||
"edition_hint": "Expansion",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Dark green box with gold ornate border, gold tree illustration",
|
||||
"source_photos": [
|
||||
"IMG_4571.jpeg"
|
||||
],
|
||||
"title_normalized": "botany tantalizing trees"
|
||||
},
|
||||
{
|
||||
"title_raw": "BOTANY: PERILOUS PERFUMES",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Dux Somnium",
|
||||
"edition_hint": "Expansion",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Dark purple/plum box with gold ornate border, gold carnivorous plant illustration",
|
||||
"source_photos": [
|
||||
"IMG_4571.jpeg"
|
||||
],
|
||||
"title_normalized": "botany perilous perfumes"
|
||||
},
|
||||
{
|
||||
"title_raw": "Parsely",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Memento More Computers Inc.",
|
||||
"edition_hint": "",
|
||||
"year_hint": 2007,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Green monochrome text on black computer screen display, styled as retro terminal game",
|
||||
"source_photos": [
|
||||
"IMG_4573.jpeg"
|
||||
],
|
||||
"title_normalized": "parsely"
|
||||
},
|
||||
{
|
||||
"title_raw": "...and then, we held hands.",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "LudiCreations",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Blue sky background with white stylized doves/hands motif, cursive white title text",
|
||||
"source_photos": [
|
||||
"image.jpg"
|
||||
],
|
||||
"title_normalized": "and then we held hands"
|
||||
},
|
||||
{
|
||||
"title_raw": "Consentacle",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Dead Pixel • www.deadpixel.co",
|
||||
"edition_hint": "",
|
||||
"year_hint": 2018,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Black box with purple tentacle/octopus illustration on spine",
|
||||
"source_photos": [
|
||||
"shelf-20260803-171316.jpg"
|
||||
],
|
||||
"title_normalized": "consentacle"
|
||||
},
|
||||
{
|
||||
"title_raw": "The Simpsons Trivia Game",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Matt Groening / The Simpsons",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Green background with die-cut character silhouette shapes of Homer, Marge, Bart, Lisa, and Maggie; includes 'Ay Carumba!' badge and note about included cast poster",
|
||||
"source_photos": [
|
||||
"shelf-20260803-213735.jpg"
|
||||
],
|
||||
"title_normalized": "simpsons trivia game"
|
||||
},
|
||||
{
|
||||
"title_raw": "THE CAT GAME",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Spin Master",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English/French",
|
||||
"art_notes": "White box with faux-fur textured letters spelling CAT, image of cat wearing top hat and bow tie standing on a scratching post, hand holding a marker/pointer stick, tagline 'The Hair-Raising Drawing Game (with Cats)'",
|
||||
"source_photos": [
|
||||
"shelf-20260803-213756.jpg"
|
||||
],
|
||||
"title_normalized": "cat game"
|
||||
},
|
||||
{
|
||||
"title_raw": "BLINK",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Mattel Games",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Orange sunburst background box with green bubble letters spelling BLINK, cards showing green stars, red flowers, blue triangles, yellow droplets",
|
||||
"source_photos": [
|
||||
"shelf-20260803-215650.jpg"
|
||||
],
|
||||
"title_normalized": "blink"
|
||||
},
|
||||
{
|
||||
"title_raw": "The Office Trivia Game",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Cardinal",
|
||||
"edition_hint": "Ready to Roll, Fast-Paced Game!",
|
||||
"year_hint": null,
|
||||
"language_hint": "English",
|
||||
"art_notes": "Box shows three characters from The Office in an office/warehouse setting, notebook paper design border, red 'Ready to Roll' circular badge",
|
||||
"source_photos": [
|
||||
"shelf-20260803-215703.jpg"
|
||||
],
|
||||
"title_normalized": "office trivia game"
|
||||
},
|
||||
{
|
||||
"title_raw": "Civilization: West Extension Map",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "",
|
||||
"art_notes": "",
|
||||
"source_photos": [],
|
||||
"title_normalized": "civilization west extension map"
|
||||
},
|
||||
{
|
||||
"title_raw": "Artistry: Delightful Doorways Mini Expansion",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "",
|
||||
"art_notes": "",
|
||||
"source_photos": [],
|
||||
"title_normalized": "artistry delightful doorways mini expansion"
|
||||
},
|
||||
{
|
||||
"title_raw": "castle panic wizard's tower",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "",
|
||||
"edition_hint": "",
|
||||
"year_hint": null,
|
||||
"language_hint": "",
|
||||
"art_notes": "",
|
||||
"source_photos": [],
|
||||
"title_normalized": "castle panic wizard s tower"
|
||||
}
|
||||
]
|
||||
|
||||
@@ -1,7 +1 @@
|
||||
bgg_id,bgg_name,year,type,version_id,version_name,title_raw,source_photos,second_copy
|
||||
235096,Cat Crimes,2017,boardgame,360982,English edition,CAT CRIMES,IMG_4499.jpeg;IMG_4505.jpeg,
|
||||
268163,A Gentle Rain,2019,boardgame,268164,Bloom Edition,a Gentle Rain,IMG_4501.jpeg;IMG_4507.jpeg,
|
||||
50381,Cards Against Humanity,2009,boardgame,,,Cards Against Humanity,IMG_4501.jpeg;IMG_4556.jpeg,
|
||||
1078,Skip-Bo,1967,boardgame,,,SKIP-BO,IMG_4501.jpeg;IMG_4542.jpeg,
|
||||
218866,Scrawl,2017,boardgame,,,SCRAWL,IMG_4502.jpeg,
|
||||
195314,Herbaceous,2017,boardgame,,,Herbaceous,IMG_4507.jpeg,
|
||||
|
||||
|
@@ -360,5 +360,17 @@
|
||||
"partial_text": "TOWN...FUKU (possibly Japanese text)",
|
||||
"art_notes": "Colorful box with cartoon character illustrations, appears to be a small/medium sized game box, mostly obscured by foreground items"
|
||||
}
|
||||
],
|
||||
"IMG_4573.jpeg": [
|
||||
{
|
||||
"location": "Upper right area of the book cover art, stacked above 'ACTION CASTLE I' box",
|
||||
"partial_text": "AC... (partially obscured by hand/fingers)",
|
||||
"art_notes": "Red/orange box spine, appears to be part of a series with other 'Action Castle' related titles"
|
||||
},
|
||||
{
|
||||
"location": "Bottom right of cover art, near 'Blackboard' box",
|
||||
"partial_text": "",
|
||||
"art_notes": "Small dark box with logo icon, title illegible due to size and angle"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -108,5 +108,7 @@
|
||||
"IMG_4556.jpeg|Top right corner, quilted patchwork-pattern box, right of the box with 'TCH ORK' text||Multicolored quilt/patchwork square pattern box, title not legible, partially cut at frame edge",
|
||||
"IMG_4556.jpeg|Top row, green textured spine with cartoon dinosaur, left of 'Ravensbu...' spine||Green speckled/scaly texture spine with small cartoon dinosaur illustration",
|
||||
"IMG_4556.jpeg|Top row, second shelf, spine reading 'Ravensbu...' between an unidentified dark box and green dinosaur-patterned spine|Ravensbu...|White spine with blue text, likely Ravensburger logo/publisher rather than title, top cut off",
|
||||
"IMG_4566.jpeg|Top shelf, background, partially obscured behind and above the two Alice Is Missing boxes|TOWN...FUKU (possibly Japanese text)|Colorful box with cartoon character illustrations, appears to be a small/medium sized game box, mostly obscured by foreground items"
|
||||
"IMG_4566.jpeg|Top shelf, background, partially obscured behind and above the two Alice Is Missing boxes|TOWN...FUKU (possibly Japanese text)|Colorful box with cartoon character illustrations, appears to be a small/medium sized game box, mostly obscured by foreground items",
|
||||
"IMG_4573.jpeg|Bottom right of cover art, near 'Blackboard' box||Small dark box with logo icon, title illegible due to size and angle",
|
||||
"IMG_4573.jpeg|Upper right area of the book cover art, stacked above 'ACTION CASTLE I' box|AC... (partially obscured by hand/fingers)|Red/orange box spine, appears to be part of a series with other 'Action Castle' related titles"
|
||||
]
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
action,bgg_id,collid,name,version_id,second_copy,status,timestamp,error
|
||||
add,334011,,A Gentle Rain,701315,,failed,2026-08-06T02:15:34+00:00,"TimeoutError: Locator.wait_for: Timeout 15000ms exceeded. Call log: - waiting for get_by_role(""dialog"").get_by_role(""heading"", name=re.compile(r""A\ Gentle\ Rain"", re.IGNORECASE)) to be visible"
|
||||
add,589,,Wiz-War,27352,,failed,2026-08-06T02:20:33+00:00,"Error: Locator.check: Error: strict mode violation: get_by_role(""dialog"").get_by_label(""Own"") resolved to 2 elements: 1) <input type=""checkbox"" ng-model=""item.status.own"" class=""ng-pristine ng-untouched ng-valid ng-empty""/> aka get_by_role(""checkbox"", name=""Own"", exact=True) 2) <input type=""checkbox"" ng-model=""item.status.prevowned"" class=""ng-pristine ng-untouched ng-valid ng-empty""/> aka get_by_role(""checkbox"", name=""Prev. Owned"") Call log: - waiting for get_by_role(""dialog"").get_by_label(""Own"")"
|
||||
add,334011,,A Gentle Rain,701315,,added,2026-08-06T02:21:32+00:00,
|
||||
add,125921,,Catan: Junior,476263,,added,2026-08-06T02:24:22+00:00,
|
||||
add,31260,,Agricola,297589,,added,2026-08-06T02:24:28+00:00,
|
||||
add,181304,,Mysterium,536289,,added_no_version,2026-08-06T02:24:34+00:00,version 'English edition 2018-2' not in picker; added without version
|
||||
add,312786,,Poetry for Neanderthals,514050,,added_no_version,2026-08-06T02:24:40+00:00,version 'English edition 2020' not in picker; added without version
|
||||
add,273240,,The Red Dragon Inn Smorgasbox,446069,,added,2026-08-06T02:24:46+00:00,
|
||||
add,419687,,Munchkin Big Box,711400,,failed,2026-08-06T02:38:12+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""dialog"").locator(""ul.pagination a[title=\""First Page\""]"").first - locator resolved to <a href="""" role=""menuitem"" title=""First Page"" class=""pagination-pager"" ng-click=""selectPage(1)""> ⇆⇆⇆⇆First ⇆⇆⇆</a> - attempting click action 2 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 20ms 2 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 100ms 55 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 500ms"
|
||||
add,252153,,Tang Garden,404702,,failed,2026-08-06T02:38:48+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""dialog"").locator(""ul.pagination a[title=\""First Page\""]"").first - locator resolved to <a href="""" role=""menuitem"" title=""First Page"" class=""pagination-pager"" ng-click=""selectPage(1)""> ⇆⇆⇆⇆First ⇆⇆⇆</a> - attempting click action 2 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 20ms 2 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 100ms 56 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 500ms"
|
||||
add,255984,,Sleeping Gods,701034,,failed,2026-08-06T02:39:23+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""dialog"").locator(""ul.pagination a[title=\""First Page\""]"").first - locator resolved to <a href="""" role=""menuitem"" title=""First Page"" class=""pagination-pager"" ng-click=""selectPage(1)""> ⇆⇆⇆⇆First ⇆⇆⇆</a> - attempting click action 2 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 20ms 2 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 100ms 56 × waiting for element to be visible, enabled and stable - element is not visible - retrying click action - waiting 500ms"
|
||||
add,589,,Wiz-War,27352,,added,2026-08-06T02:45:10+00:00,
|
||||
add,419687,,Munchkin Big Box,711400,,added,2026-08-06T02:45:16+00:00,
|
||||
add,252153,,Tang Garden,404702,,added,2026-08-06T02:45:23+00:00,
|
||||
add,255984,,Sleeping Gods,701034,,failed,2026-08-06T02:45:58+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""dialog"").locator(""li:has(.summary-item-thumbnail)"").filter(has_text=re.compile(r""Sleeping\ Gods\ \(English\ Gamefound\ edition\)\ \(2023\)"", re.IGNORECASE)).first"
|
||||
add,283766,,Sleeping Gods: Tides of Ruin,463140,,added,2026-08-06T02:46:07+00:00,
|
||||
add,255984,,Sleeping Gods,701034,,failed,2026-08-06T02:49:44+00:00,RuntimeError: 'Sleeping Gods (English Gamefound edition) (2023)' was listed a moment ago but vanished on the second pass — retryable
|
||||
add,280794,,Etherfields,458300,,added,2026-08-06T02:49:53+00:00,
|
||||
add,380837,,Botany,650545,,added,2026-08-06T02:50:00+00:00,
|
||||
add,393672,,Gloomhaven: Buttons & Bugs,669782,,failed,2026-08-06T02:50:07+00:00,RuntimeError: 'Gloomhaven: Buttons & Bugs (English edition) (2024)' was listed a moment ago but vanished on the second pass — retryable
|
||||
add,14535,,SPANC: Space Pirate Amazon Ninja Catgirls,29075,,added,2026-08-06T02:50:15+00:00,
|
||||
add,255984,,Sleeping Gods,701034,,added,2026-08-06T02:54:05+00:00,
|
||||
add,393672,,Gloomhaven: Buttons & Bugs,669782,,added,2026-08-06T02:54:12+00:00,
|
||||
add,9666,,Greed Quest,215137,,added,2026-08-06T02:54:19+00:00,
|
||||
add,40508,,Scrabble Slam!,644326,,added,2026-08-06T02:54:27+00:00,
|
||||
add,3181,,Farkle,87931,,added,2026-08-06T02:54:36+00:00,
|
||||
add,193485,,Dastardly Dirigibles,300446,,added,2026-08-06T02:54:54+00:00,
|
||||
add,226065,,Sheriff of Nottingham: Merry Men,353389,,added,2026-08-06T02:55:01+00:00,
|
||||
add,143884,,Machi Koro,453051,,added,2026-08-06T02:55:09+00:00,
|
||||
add,358880,,Etherfields: Kittenburg Expansion,603694,,added,2026-08-06T02:55:15+00:00,
|
||||
add,432858,,Artistry,734890,,added,2026-08-06T02:55:21+00:00,
|
||||
add,417060,,Botany: Tantalizing Trees,706506,,added,2026-08-06T02:55:27+00:00,
|
||||
add,417064,,Botany: Perilous Perfumes,706510,,added,2026-08-06T02:55:33+00:00,
|
||||
add,161928,,Utter Nonsense,,,added,2026-08-06T02:55:40+00:00,
|
||||
add,50381,,Cards Against Humanity,,,added,2026-08-06T02:55:44+00:00,
|
||||
add,1269,,Skip-Bo,,,added,2026-08-06T02:55:50+00:00,
|
||||
add,202982,,Scrawl,,,added,2026-08-06T02:55:56+00:00,
|
||||
add,195314,,Herbaceous,,,added,2026-08-06T02:56:02+00:00,
|
||||
add,320202,,Dragon Land,,,added,2026-08-06T02:56:07+00:00,
|
||||
add,2452,,Jenga,,,added,2026-08-06T02:56:12+00:00,
|
||||
add,42063,,Code Master,,,added,2026-08-06T02:56:18+00:00,
|
||||
add,340420,,Throw Throw Avocado,,,added,2026-08-06T02:56:23+00:00,
|
||||
add,1260,,Rook,,,added,2026-08-06T02:56:28+00:00,
|
||||
add,116,,Guillotine,,,added,2026-08-06T02:56:34+00:00,
|
||||
add,169124,,Flick 'em Up!,,,added,2026-08-06T02:56:40+00:00,
|
||||
add,20866,,The Rocky Horror Trivia Game,,,added,2026-08-06T02:56:45+00:00,
|
||||
add,269564,,Captain Marvel: Secret Skrulls,,,added,2026-08-06T02:57:37+00:00,
|
||||
add,18755,,Pie Face!,,,added,2026-08-06T02:57:41+00:00,
|
||||
add,140509,,Dungeons & Dragons,,,failed,2026-08-06T02:58:01+00:00,"TimeoutError: Locator.wait_for: Timeout 15000ms exceeded. Call log: - waiting for get_by_role(""dialog"").get_by_role(""heading"", name=re.compile(r""Dungeons\ \&\ Dragons"", re.IGNORECASE)) to be visible"
|
||||
add,201248,,Evolution: The Beginning,,,added,2026-08-06T02:58:05+00:00,
|
||||
add,362205,,Sleeping Gods: Primeval Peril,,,added,2026-08-06T02:58:11+00:00,
|
||||
add,453845,,Murder at the Manor,,,added,2026-08-06T02:58:16+00:00,
|
||||
add,329230,,Bluffaneer,,,added,2026-08-06T02:58:20+00:00,
|
||||
add,358320,,Sleeping Gods: Distant Skies,,,added,2026-08-06T02:58:26+00:00,
|
||||
add,389113,,Rivals,,,added,2026-08-06T02:58:31+00:00,
|
||||
add,324937,,Wiz-War (9th Edition),,,added,2026-08-06T02:58:35+00:00,
|
||||
add,295770,,Frosthaven,,,added,2026-08-06T02:58:40+00:00,
|
||||
add,174430,,Gloomhaven,,,added,2026-08-06T02:58:46+00:00,
|
||||
add,442312,,Deadly Doodles,,,added,2026-08-06T02:58:51+00:00,
|
||||
add,332398,,Everdell: The Complete Collection,,,added,2026-08-06T02:58:56+00:00,
|
||||
add,334065,,Verdant,,,added,2026-08-06T02:59:01+00:00,
|
||||
add,283155,,Calico,,,added,2026-08-06T02:59:06+00:00,
|
||||
add,291457,,Gloomhaven: Jaws of the Lion,,,added,2026-08-06T02:59:12+00:00,
|
||||
add,150925,,Hold Your Breath!,,,added,2026-08-06T02:59:18+00:00,
|
||||
add,2980,,The Simpsons Trivia Game,,,added,2026-08-06T02:59:23+00:00,
|
||||
add,1197,,Blink,,,added,2026-08-06T02:59:29+00:00,
|
||||
add,140509,,Dungeons & Dragons,,,failed,2026-08-06T03:05:04+00:00,"TimeoutError: Locator.wait_for: Timeout 15000ms exceeded. Call log: - waiting for get_by_role(""dialog"").get_by_role(""heading"", name=re.compile(r""Dungeons\ \&\ Dragons"", re.IGNORECASE)) to be visible"
|
||||
add,352819,,The Office Trivia Game,,,added,2026-08-06T03:05:09+00:00,
|
||||
add,163412,,Patchwork,,,added,2026-08-06T03:05:14+00:00,
|
||||
add,291847,,Mantis Falls,,,added,2026-08-06T03:05:19+00:00,
|
||||
add,2058,,Civilization: West Extension Map,,,added,2026-08-06T03:05:24+00:00,
|
||||
add,452833,,Artistry: Delightful Doorways Mini Expansion,,,added,2026-08-06T03:05:31+00:00,
|
||||
update,240,53429559,Britannia,24621,,failed,2026-08-06T03:06:04+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""row"").filter(has_text=re.compile(r""Britannia"", re.IGNORECASE)).first.get_by_role(""link"", name=re.compile(r""own"", re.IGNORECASE)).first"
|
||||
update,71,53429530,Civilization,24006,,failed,2026-08-06T03:06:39+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""row"").filter(has_text=re.compile(r""Civilization"", re.IGNORECASE)).first.get_by_role(""link"", name=re.compile(r""own"", re.IGNORECASE)).first"
|
||||
update,177,53429542,Advanced Civilization,24972,,failed,2026-08-06T03:07:12+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""row"").filter(has_text=re.compile(r""Advanced\ Civilization"", re.IGNORECASE)).first.get_by_role(""link"", name=re.compile(r""own"", re.IGNORECASE)).first"
|
||||
update,240,53429559,Britannia,24621,,failed,2026-08-06T03:08:42+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""row"").filter(has_text=re.compile(r""Britannia"", re.IGNORECASE)).first.get_by_role(""link"", name=re.compile(r""own"", re.IGNORECASE)).first"
|
||||
update,71,53429530,Civilization,24006,,failed,2026-08-06T03:09:17+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""row"").filter(has_text=re.compile(r""Civilization"", re.IGNORECASE)).first.get_by_role(""link"", name=re.compile(r""own"", re.IGNORECASE)).first"
|
||||
update,177,53429542,Advanced Civilization,24972,,failed,2026-08-06T03:09:50+00:00,"TimeoutError: Locator.click: Timeout 30000ms exceeded. Call log: - waiting for get_by_role(""row"").filter(has_text=re.compile(r""Advanced\ Civilization"", re.IGNORECASE)).first.get_by_role(""link"", name=re.compile(r""own"", re.IGNORECASE)).first"
|
||||
update,240,53429559,Britannia,24621,,updated,2026-08-06T03:17:05+00:00,
|
||||
update,71,53429530,Civilization,24006,,failed,2026-08-06T03:17:11+00:00,"Error: Page.goto: net::ERR_NETWORK_CHANGED at https://boardgamegeek.com/collection/user/***?objectid=71&own=1 Call log: - navigating to ""https://boardgamegeek.com/collection/user/***?objectid=71&own=1"", waiting until ""domcontentloaded"""
|
||||
update,177,53429542,Advanced Civilization,24972,,updated,2026-08-06T03:17:17+00:00,
|
||||
update,71,53429530,Civilization,24006,,updated,2026-08-06T03:17:36+00:00,
|
||||
update,3090,53429504,Doctor Who: The Game of Time & Space,25782,,updated,2026-08-06T03:17:55+00:00,
|
||||
update,3585,53429482,Sorcerer: The Game of Magical Conflict,211154,,updated,2026-08-06T03:18:02+00:00,
|
||||
update,2524,53429367,StarForce 'Alpha Centauri': Interstellar Conflict in the 25th Century,219137,,updated,2026-08-06T03:18:06+00:00,
|
||||
update,224,53429816,History of the World,20727,,updated,2026-08-06T03:18:11+00:00,
|
||||
update,9209,53429636,Ticket to Ride,294188,,updated,2026-08-06T03:18:17+00:00,
|
||||
update,13,53430595,Catan,347121,,updated,2026-08-06T03:18:24+00:00,
|
||||
update,1219,53430485,Labyrinth,488621,,updated,2026-08-06T03:18:32+00:00,
|
||||
update,137909,53429649,Bugs in the Kitchen,216474,,updated,2026-08-06T03:18:38+00:00,
|
||||
update,27710,53430142,Catan Dice Game,606657,,updated,2026-08-06T03:18:44+00:00,
|
||||
update,169784,53429953,Castle Panic: The Dark Titan,260454,,updated,2026-08-06T03:18:49+00:00,
|
||||
update,147370,53429666,Robot Turtles,231664,,updated,2026-08-06T03:18:55+00:00,
|
||||
update,43443,53430559,Castle Panic,103068,,updated,2026-08-06T03:19:02+00:00,
|
||||
update,24310,53429632,The Red Dragon Inn,374258,,updated,2026-08-06T03:19:06+00:00,
|
||||
update,39856,53429748,Dixit,296529,,updated,2026-08-06T03:19:11+00:00,
|
||||
update,35505,53430257,Walk the Plank!,781669,,updated,2026-08-06T03:19:15+00:00,
|
||||
update,164,53430020,"Before I Kill You, Mister Bond",28465,,updated,2026-08-06T03:19:20+00:00,
|
||||
update,181304,148198097,Mysterium,536289,,updated,2026-08-06T03:19:27+00:00,
|
||||
update,65244,53429887,Forbidden Island,31208,,updated,2026-08-06T03:19:33+00:00,
|
||||
update,95386,53429908,Tempurra,290572,,updated,2026-08-06T03:19:41+00:00,
|
||||
update,258,53430200,Fluxx,20840,,updated,2026-08-06T03:19:47+00:00,
|
||||
update,312786,148198100,Poetry for Neanderthals,514050,,updated,2026-08-06T03:19:52+00:00,
|
||||
update,188614,53430332,Simon's Cat Card Game,779792,,updated,2026-08-06T03:19:57+00:00,
|
||||
update,4324,53429850,Risk: The Lord of the Rings,26859,,updated,2026-08-06T03:20:01+00:00,
|
||||
update,1927,53429933,Munchkin,28144,,updated,2026-08-06T03:20:06+00:00,
|
||||
update,1784,53430007,Dark Cults,110926,,updated,2026-08-06T03:20:11+00:00,
|
||||
update,176,53429986,Give Me the Brain!,27673,,updated,2026-08-06T03:20:16+00:00,
|
||||
update,257,53430040,Kill Doctor Lucky,520364,,updated,2026-08-06T03:20:21+00:00,
|
||||
update,153999,53612880,"...and then, we held hands.",280845,,updated,2026-08-06T03:20:26+00:00,
|
||||
update,166976,53612866,Consentacle,410382,,updated,2026-08-06T03:20:32+00:00,
|
||||
update,231302,53430652,The Cat Game,361880,,updated,2026-08-06T03:20:36+00:00,
|
||||
update,40692,53430517,Small World,294178,,updated,2026-08-06T03:20:45+00:00,
|
||||
update,172503,53429718,Mage Wars Academy,265107,,updated,2026-08-06T03:20:49+00:00,
|
||||
update,104710,53429642,Wiz-War (Eighth Edition),117685,,updated,2026-08-06T03:20:53+00:00,
|
||||
update,1339,53430097,Dungeon!,34390,,updated,2026-08-06T04:01:15+00:00,
|
||||
|
@@ -100,3 +100,89 @@ the existing one).
|
||||
page but did not overlay the form in the probe.
|
||||
- Logged-in detection heuristic (unverified): the header shows a "Sign In"
|
||||
link only when logged out.
|
||||
|
||||
|
||||
## Verified against the live site (2026-08-06, first real uploads)
|
||||
|
||||
The add flow works end to end; every failure on the way was in code the
|
||||
earlier walkthrough had marked *verified*, and the parts marked
|
||||
*unverified* were mostly right. Corrections:
|
||||
|
||||
- **Sign In is an `<a class="btn">` with no `href`.** It therefore has no
|
||||
implicit `link` role: `get_by_role("link", name="Sign In")` matches
|
||||
nothing in any state. The header also hydrates after
|
||||
`domcontentloaded`, so for a moment neither Sign In nor Sign Out
|
||||
exists — a check resting on one absence silently concludes "signed in"
|
||||
and browses anonymously. Poll until one control or the other proves the
|
||||
state; treat "neither, after 30s" as an error.
|
||||
- **`get_by_label("Own")` also matches "Prev. Owned."** Use
|
||||
`get_by_role("checkbox", name="Own", exact=True)`.
|
||||
- **Version rows** are the `<li>`s carrying a thumbnail:
|
||||
`li:has(.summary-item-thumbnail)`. Plain `listitem` also catches the
|
||||
paging `<li>`s ("First", "Prev", "1", "…").
|
||||
- **Paging is an AngularJS `<ul class="pagination">` of anchors**, not
|
||||
buttons: `a[title="Next Page"]`, with the parent `<li>` gaining
|
||||
`disabled` at the end. Paging is client-side over an already-loaded
|
||||
list — no request per page.
|
||||
- **Every paging control renders TWICE**: a desktop set and a mobile set
|
||||
inside `<li class="visible-xs-*">`. A selector matches both, and
|
||||
`.first` may be the hidden one — Playwright then waits for it to become
|
||||
visible until it times out. Always click the first *visible* match.
|
||||
First/Prev may be unclickable on a desktop viewport (mobile-only
|
||||
variant). To return to page 1, click the visible numbered **"1"**
|
||||
anchor: the sub-view's paging state SURVIVES closing and reopening it
|
||||
(Angular keeps the scope), so a reopen lands wherever it was left, not
|
||||
on page 1.
|
||||
- **Match row text in Python, not with `has_text`.** Playwright's
|
||||
`has_text` regex tests raw `textContent`, which carries the markup's
|
||||
tabs and newlines; a whitespace-normalized capture will never equal it.
|
||||
Normalize both sides yourself and click the row by index.
|
||||
- **Row text is `<game name> (<version name>) (<year>)`**, and the game
|
||||
name is localized (a Czech edition's row starts "Spící bohové"). Match
|
||||
the version name inside its parentheses.
|
||||
- **The API's version name is not always the picker's string.** BGG's
|
||||
XML gives e.g. `English edition 2018-2` where the picker shows
|
||||
`(English edition) (2018)`. Match the full name first, then retry with
|
||||
a trailing year/printing qualifier stripped — and if that relaxed match
|
||||
hits more than one row, refuse and add version-less (never guess).
|
||||
- **An owned game's page has no "Add To" button**; it reads
|
||||
"In Collections (Own…)". That is the update flow's entry point.
|
||||
|
||||
### The update flow (verified 2026-08-06)
|
||||
|
||||
Do **not** go through the game page's dialog for an existing entry — it
|
||||
cannot say which copy it edits. The collection table can:
|
||||
|
||||
1. `/collection/user/<user>?objectid=<bgg_id>&own=1`.
|
||||
2. The version cell carries its own collid:
|
||||
`td.collection_version[onclick*="<collid>"]`. Clicking it opens an
|
||||
inline editor (the cell shows *Editing* meanwhile).
|
||||
3. The editor is a radio list whose **values are version ids**:
|
||||
`form[id^="form_version"] input[type="radio"][value="<version_id>"]`.
|
||||
Both the copy and the edition are therefore addressed exactly — no
|
||||
name matching, no pagination.
|
||||
4. Clicking the radio fires `CE_SaveData(cellid, collid, 'version')`
|
||||
itself. **There is no Save button.** The save has landed when the
|
||||
cell's text stops reading *Editing*.
|
||||
|
||||
This is strictly additive: it sets one field on one collid and cannot
|
||||
create a duplicate entry.
|
||||
|
||||
### BGG's collection export lags the site
|
||||
|
||||
After a successful update the website shows the new version immediately
|
||||
(the row's version cell reads e.g. "English first edition Year: 2012",
|
||||
and reopening the editor shows that radio checked), but the XML API's
|
||||
`/collection` export can still report the entry with no version — even
|
||||
with a cache-busting re-request. `diff` reads the API, so it will
|
||||
re-queue work that has already landed.
|
||||
|
||||
Consequences to keep in mind rather than "fix":
|
||||
|
||||
- The upload log is the authority on what this tool did; the API is the
|
||||
authority on what BGG has published. They disagree for a while.
|
||||
- `build_queue` skipping a job whose log says done is CORRECT here — the
|
||||
work is applied, and re-running it would be a no-op at best.
|
||||
- Anything user-facing should count PENDING jobs (queue rows minus what
|
||||
the log completed), never raw queue rows, or settled work reads as
|
||||
outstanding until the next diff.
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
# The bggpipe user's guide
|
||||
|
||||
Everything past the [README](../README.md)'s quick start: credentials, every stage and its flags, the web app, phones, RPGs, fixing the model's mistakes, and running without a BGG token. Screenshots of everything described here: the [tour](tour.md).
|
||||
|
||||
## Contents
|
||||
|
||||
- [Credentials and configuration](#credentials-and-configuration)
|
||||
- [The web app](#the-web-app)
|
||||
- [From your phone](#from-your-phone)
|
||||
- [The stages, from the terminal](#the-stages-from-the-terminal)
|
||||
- [Taking good shelf photos](#taking-good-shelf-photos)
|
||||
- [Fixing what the model gets wrong](#fixing-what-the-model-gets-wrong)
|
||||
- [RPGs on your shelves](#rpgs-on-your-shelves)
|
||||
- [Uploading safely](#uploading-safely)
|
||||
- [Running before your BGG token arrives](#running-before-your-bgg-token-arrives)
|
||||
- [Keeping your data safe from git](#keeping-your-data-safe-from-git)
|
||||
|
||||
## Credentials and configuration
|
||||
|
||||
The `init` wizard is idempotent — re-run it anytime to check status or add keys you skipped. It prompts for the credentials below (hidden input, saved to a `.env` it creates with owner-only permissions) and offers the one-time Playwright Chromium download. Prefer doing it by hand? Copy [.env.example](../.env.example) beside your data, fill it in, and run `playwright install chromium` yourself.
|
||||
|
||||
Secrets live in environment variables only, never in config files, code, or logs, and `.env` is gitignored. Every `bggpipe` command loads `.env` from the working directory by itself — real environment variables always win over the file, so [direnv](https://direnv.net/) users and CI overrides keep working unchanged.
|
||||
|
||||
Your credentials never leave your machine except to sign in to boardgamegeek.com itself — [the README spells out the full privacy picture](../README.md#why-it-asks-for-your-bgg-password--and-where-your-credentials-go). The password exists solely because BGG has no write API: adding games means driving the real website, in a visible browser window, on your computer. The saved browser session (`storage_state.json`) is credential-adjacent — it stays local and gitignored too.
|
||||
|
||||
| Variable | Used by | What it is |
|
||||
|---|---|---|
|
||||
| `ANTHROPIC_API_KEY` | extract | Anthropic API key |
|
||||
| `BGG_API_TOKEN` | resolve, diff, enrich | Bearer token from your registered BGG application |
|
||||
| `BGG_USERNAME` | diff, upload, enrich | Your BGG username (public, but kept in `.env` so it lives in one place) |
|
||||
| `BGG_PASSWORD` | upload (website login) | Your BGG password |
|
||||
|
||||
Non-secret knobs live in `config.toml`: `photos_dir`, `data_dir`, the BGG rate limit, and the vision setup — a `[vision.<provider>]` block per provider ("anthropic" or any OpenAI-compatible endpoint, including a local [Ollama](https://ollama.com/)), with `vision_provider` picking one. Local models read spines noticeably worse than frontier ones — expect a longer proofread pass on the Titles page, not a broken pipeline.
|
||||
|
||||
**What extraction costs:** the vision call is the pipeline's only paid step, and it's small — the author's full collection (65 shelf photos, 136 games) came to under a dollar on the default model (Claude Sonnet). It's also one-time: raw reads are cached per photo, so re-running extract is free, and only new or replaced photos are ever sent again. Everything BGG-side is free (the API token costs nothing).
|
||||
|
||||
## The web app
|
||||
|
||||
```sh
|
||||
bggpipe web # opens http://127.0.0.1:8377/ — the whole app in the browser
|
||||
```
|
||||
|
||||
Seven pages — Pipeline, Photos, Titles, Review, Queue, Library, and Help — all [pictured in the tour](tour.md). Stage runs execute one at a time in the background with live output:
|
||||
|
||||

|
||||
|
||||
Stage runs execute one at a time; every decision saves immediately; the pages live-follow the data files, so a stage run in another terminal shows up without a refresh. The real upload sits behind a confirmation (and behind a stub-data lock if synthetic test fixtures ever regenerate). The in-app **Help** page documents every status chip and keyboard shortcut.
|
||||
|
||||
## From your phone
|
||||
|
||||
The app is localhost-only by default. To use it from a phone or tablet on your network — proofreading from the couch, or shooting shelf photos straight into the pipeline — serve it to the LAN instead:
|
||||
|
||||
```sh
|
||||
bggpipe web --lan # localhost + your network, behind an access key
|
||||
```
|
||||
|
||||
Startup prints a pairing link (`?k=...`) and a QR code: point the phone's camera at the terminal and tap. Pairing is one-time per device — the key persists across restarts (`data/.lan_key`; delete it to revoke every paired device) and the cookie lasts a year. Save the page to the phone's home screen for the full-screen treatment, piper icon included.
|
||||
|
||||
To photograph shelves from the phone: on the Photos page, tap the drop zone and choose "Take Photo." The upload narrates its progress, and camera captures get unique `shelf-<timestamp>` names so rapid-fire shots never overwrite each other. The key is the only lock — there is no login behind it — so still prefer networks you trust (or use a device VPN like Tailscale against the localhost default instead).
|
||||
|
||||
## The stages, from the terminal
|
||||
|
||||
Every stage is also a command, and the two interfaces share all state:
|
||||
|
||||
```sh
|
||||
bggpipe extract # photos → titles.json (+ retake prompts)
|
||||
bggpipe resolve # titles → BGG ids/versions in matches.csv
|
||||
bggpipe review --web # review UI only
|
||||
bggpipe diff # compare against your BGG collection
|
||||
bggpipe upload --dry-run # ALWAYS inspect this first
|
||||
bggpipe upload --limit 1 # then one game, then small batches
|
||||
bggpipe enrich # full metadata → data/games.json
|
||||
```
|
||||
|
||||
Each stage skips work it has already done; `--force`/`--refresh` flags redo it. Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit. `review` without `--web` runs in the terminal instead.
|
||||
|
||||
## Taking good shelf photos
|
||||
|
||||
Straight-on, one shelf (or part of one) per shot, close enough that spine text is legible to a human. If you can't read it, the model can't either. Overlap between shots is fine: duplicate reads are deduped automatically, with the merge shown (and veto-able) in review. Boxes the model spots but can't identify become retake prompts in `unidentified.json` and the Photos page's "reshoot" tickets: photograph those boxes up close, drop the new photo in, and run `extract` again.
|
||||
|
||||
## Fixing what the model gets wrong
|
||||
|
||||
Vision reads aren't perfect, and you know things the photos don't show. The Titles page lets you **edit** a title (fix a misspelling, add publisher/edition/year/language cues you know offhand), **split** a line into per-photo copies when one title is actually several boxes, **remove** lines that aren't games at all, and **add** a game no photo caught. Every one of these is durable: the decision lands in a small committed store (`data/title_edits.json`, `data/title_splits.json`, `data/title_removals.json`, `data/title_additions.json`) that is replayed on every rebuild — re-running extract or resolve can never undo your curation. Undo any decision by deleting its record from the store.
|
||||
|
||||
Games BGG doesn't have at all can be kept as **local** library citizens: their detail page in the Library takes hand-written facts (players, playtime, publisher, notes) and a cover photo of your own, stored in `data/local_games.json` and `data/local_art/` — the only source such a game will ever have.
|
||||
|
||||
## RPGs on your shelves
|
||||
|
||||
Tabletop RPGs aren't in BGG's board-game database — they live on RPGGeek, which shares the same underlying API. When a title isn't found as a board game, bggpipe retries as an RPG: matches are identified, enriched (designers, publishers, genres from RPGGeek), and browsable in the library (filter: RPGs), but they stay **local only** — they're never uploaded, since your BGG collection can't hold them. When the automatic search can't reach the right database (BGG has board games named "Dungeons & Dragons" too), every Review card has explicit **search BGG** / **search RPGGeek** buttons.
|
||||
|
||||
## Uploading safely
|
||||
|
||||
`upload` drives a real logged-in browser session against your real account, so it is deliberately careful:
|
||||
|
||||
- `--dry-run` logs what would happen without touching the site — always read it first, then `--limit 1`, then small batches:
|
||||
|
||||

|
||||
- The browser runs **headed** by default — BGG's Cloudflare check blocks headless ones, and a first login may need one human click before the session is saved locally and reused.
|
||||
- Requests are slow on purpose (seconds between actions, per BGG's API policy); the Queue page shows exactly what will run before it runs, and `upload_log.csv` keeps a permanent record of every attempt.
|
||||
- `--retry-failed` re-attempts failures; `--verify` re-fetches your collection and cross-checks the log. Note that BGG's collection export can lag the website by hours — freshly-landed work may look missing to `diff`/`--verify` until it catches up.
|
||||
- Review decisions outrank the queue: re-deciding a match after `diff` retires its queued job automatically.
|
||||
|
||||
## Running before your BGG token arrives
|
||||
|
||||
BGG application approval can take a week or more. Until then: `extract` works immediately (it only needs the vision key), and `resolve` does what it can, parking the rest as "waiting on BGG API token" — it picks them up automatically once the token exists. Everything is saved as you go.
|
||||
|
||||
`diff` normally fetches your collection live, but there's a logged-in-browser exemption that needs no token: while signed in to BGG, save these two URLs as `data/collection_snapshot_base.xml` and `data/collection_snapshot_expansions.xml` (if you get a "queued" message, refresh after a few seconds):
|
||||
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1`
|
||||
- `https://boardgamegeek.com/xmlapi2/collection?username=YOU&own=1&version=1&subtype=boardgameexpansion`
|
||||
|
||||
## Keeping your data safe from git
|
||||
|
||||
The [README's quick start](../README.md#quick-start) — installed as a tool, run in a directory of your own — is the only supported way to use bggpipe on your collection. Everything the pipeline produces lives where you run it, and `uv tool upgrade bggpipe` picks up fixes without going anywhere near your data.
|
||||
|
||||
**Why not clone and run?** The source repo doubles as its author's live pipeline: `data/` ships with their real artifacts, committed and updated often. Run inside a clone and *your* data lands at git-tracked paths — the next `git pull` will refuse to merge, and the usual remedies (`git reset --hard`, `git checkout .`, `git stash`, `git clean -fdx`) would destroy your review decisions, hand-written games, upload log, and photos. bggpipe detects this arrangement and warns at `init` and on the web dashboard; don't ignore it.
|
||||
|
||||
One committed file of the author's data deserves a word: `data/STUB_DATA.marker` is normally absent. It appears only if the synthetic stub fixtures (from `scripts/write_stub_fixtures.py`) regenerate the CSVs, and the upload stage refuses to run while it exists — so placeholder data can never reach a real BGG account.
|
||||
|
After Width: | Height: | Size: 224 KiB |
|
After Width: | Height: | Size: 748 KiB |
|
After Width: | Height: | Size: 1.1 MiB |
|
After Width: | Height: | Size: 588 KiB |
|
After Width: | Height: | Size: 654 KiB |
|
After Width: | Height: | Size: 332 KiB |
|
After Width: | Height: | Size: 445 KiB |
|
After Width: | Height: | Size: 731 KiB |
|
After Width: | Height: | Size: 841 KiB |
|
After Width: | Height: | Size: 695 KiB |
|
After Width: | Height: | Size: 798 KiB |
|
After Width: | Height: | Size: 390 KiB |
|
After Width: | Height: | Size: 360 KiB |
|
After Width: | Height: | Size: 378 KiB |
|
After Width: | Height: | Size: 468 KiB |
@@ -1,4 +1,8 @@
|
||||
# Shelf-to-BGG Collection Pipeline — Build Spec
|
||||
# Shelf-to-BGG Collection Pipeline — Design Contract
|
||||
|
||||
> This is the pipeline's contract — what must stay true, and why — kept
|
||||
> current as the design evolves. Read it before changing pipeline
|
||||
> semantics. How to *use* the pipeline is the [user's guide](guide.md).
|
||||
|
||||
## Goal
|
||||
|
||||
@@ -11,7 +15,7 @@ Beyond the bare game, capture **which edition/version I own** wherever the photo
|
||||
## Constraints & Context
|
||||
|
||||
- BGG has **no write API**. Reads go through the XML API2 (`https://boardgamegeek.com/xmlapi2/`); writes must automate the website itself with a real login session.
|
||||
- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer <token>` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+, so development runs on recorded/stub XML fixtures until then.
|
||||
- The XML API **requires a registered application** (boardgamegeek.com/using_the_xml_api, policy 2025-07-02): every request carries `Authorization: Bearer <token>` from the `BGG_API_TOKEN` env var, sent to `boardgamegeek.com` without a leading `www`. Register a free non-commercial application at boardgamegeek.com/applications — approval can take a week+; offline development and tests run on recorded XML fixtures.
|
||||
- BGG's XML API queues collection requests: a first call may return HTTP 202 ("try again"). Retry with backoff.
|
||||
- BGG will throttle aggressive clients. Target ≤1 request every 2 seconds to any BGG endpoint, with jittered backoff on 429/503.
|
||||
- BGG changed API access policies in 2025; some older community tools broke. Don't depend on undocumented endpoints beyond XML API2 and the public website.
|
||||
@@ -66,7 +70,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
|
||||
|
||||
### Stage 3 — `review`: Human review of ambiguous/unmatched
|
||||
|
||||
- A minimal local review flow. A TUI is fine (e.g., `rich`/`textual`), or a tiny localhost web page — builder's choice, but keep it dependency-light.
|
||||
- A minimal local review flow. A TUI is fine (e.g., `rich`/`textual`), or a tiny localhost web page — builder's choice, but keep it dependency-light. *(As built: both — a rich TUI and a six-page FastAPI app sharing one decision engine.)*
|
||||
- For each `ambiguous` item: show the raw title, source photo filename, and candidate list (name, year, type, BGG rank, owned count) → pick one, skip, or reject.
|
||||
- For each `unmatched` item: allow manual BGG ID entry or a free-text re-search.
|
||||
- For each `version_ambiguous` item: show the version candidates (version name, publisher, year, language) next to the edition cues from the photo → pick one, or mark `version_unknown`. Keep this pass optional/skippable — version review shouldn't block getting games uploaded.
|
||||
@@ -88,7 +92,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
|
||||
- Log in to boardgamegeek.com with credentials from env vars (`BGG_USERNAME`, `BGG_PASSWORD`). Never write credentials to disk or logs. Persist the browser session/storage state locally so repeat runs don't re-login.
|
||||
- For each row in `to_add.csv`: navigate to the game page, use the "Add to Collection" flow, set status **Owned**, and — when a `version_id` is present — set the specific version in the collection item's version picker before saving. Manually walk this flow once and document the selectors before automating; the version UI is the most fragile part.
|
||||
- Adding a second copy of an already-owned game must create a NEW collection entry, not edit the existing one.
|
||||
- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. Verify the already-owned dialog behavior manually first — flagged as unverified in `docs/bgg-upload-flow.md`.
|
||||
- **Update mode** (rows from `to_update.csv`): open the EXISTING collection entry (keyed by `collid`) rather than the add flow, set the version, save. Must never create a duplicate entry and never change any other field of the entry. The verified route (2026-08-06, documented in `docs/bgg-upload-flow.md`): the collection table's version cell (`td.collection_version[onclick*="<collid>"]`) opens an inline editor whose radio values ARE version ids; clicking a radio saves immediately.
|
||||
- Log every attempt to `upload_log.csv`: `bgg_id, name, status (added|already_present|failed), timestamp, error`.
|
||||
- Idempotent: skip IDs already logged as `added`; re-verify against a fresh collection fetch on `--verify`.
|
||||
- Deliberately slow: 2–4 s randomized delay between games. This is a real account on a community site — behave like a polite human.
|
||||
@@ -107,6 +111,8 @@ All artifacts are flat files in a `data/` directory — human-readable, git-frie
|
||||
|
||||
- `titles.json` — extraction output (stage 1)
|
||||
- `unidentified.json` — game boxes seen but not identified (stage 1); retake prompts
|
||||
- `title_edits.json`, `title_splits.json`, `title_removals.json` — durable human curation (corrected reads/cues, split-into-copies decisions, removed lines); replayed on every titles.json rebuild so re-extraction never undoes them
|
||||
- `unidentified_dismissed.json` — dismissed retake prompts (kept apart so rebuilds can't resurrect them)
|
||||
- `bgg_cache/` — cached XML API responses
|
||||
- `matches.csv` — the master matching table (stages 2–3)
|
||||
- `to_add.csv` — upload queue, new entries (stage 4)
|
||||
@@ -122,7 +128,7 @@ All artifacts are flat files in a `data/` directory — human-readable, git-frie
|
||||
|
||||
- **202 queue** on `/collection`: retry with backoff (2s, 5s, 10s, 30s; give up after ~5 tries with a clear message).
|
||||
- **HEIC photos** from iPhone: convert transparently.
|
||||
- **Duplicate copies**: a title appearing in multiple photos with consistent edition cues is one game (dedupe). But genuinely distinct editions of the same game ARE in scope — they stay separate entries end-to-end and become separate BGG collection entries. Identical duplicate copies of the same edition are out of scope (assume dedupe).
|
||||
- **Duplicate copies**: a title appearing in multiple photos with consistent edition cues is one game (dedupe). But genuinely distinct editions of the same game ARE in scope — they stay separate entries end-to-end and become separate BGG collection entries. *(Amended as built:)* identical duplicate copies of the same edition are supported too, as an explicit human decision — a photo-scoped "split into copies" recorded in `data/title_splits.json`, honored by every dedupe pass thereafter.
|
||||
- **Non-game items** on shelves (books, card sleeves, storage boxes): the vision prompt should be instructed to include only board/card games; anything that slips through will fail resolution and land in review.
|
||||
- **Special characters** in titles (é, colons, ampersands): normalize consistently on both sides of the match.
|
||||
- **Base game vs. expansion vs. new edition**: the most common failure mode. Bias toward surfacing these as `ambiguous` rather than auto-matching.
|
||||
@@ -0,0 +1,48 @@
|
||||
# A tour of bggpipe
|
||||
|
||||
Seven pages in one local app — Pipeline, Photos, Titles, Review, Queue, Library, and Help. Every screenshot below is the real app on the author's real shelves, and the end product is public: [the author's BGG collection](https://boardgamegeek.com/collection/user/ewagoner) is what this pipeline built. (Back to the [README](../README.md) · how to use it all: the [user's guide](guide.md).)
|
||||
|
||||
**The Pipeline page** — every stage is a card with live counts and a Run button; blockers (missing token, stub-data lock) surface as banners, not surprises. Here: the settled state after a full run — 136 titles read, 115 matched, 62 added.
|
||||
|
||||

|
||||
|
||||
**Photos** — drag shelf photos in; boxes the vision model saw but couldn't read become illustrated reshoot tickets with shelf directions a human can follow.
|
||||
|
||||

|
||||
|
||||
**Titles** — every read off the shelves, alphabetized with status and source photos. This is the proofreading checkpoint: edit misreads, split multi-copy lines, remove non-games, add a game no photo caught.
|
||||
|
||||

|
||||
|
||||
**Review** — keyboard-first decisions on ambiguous matches. Each card shows the shelf photo beside the candidates (with rank, owner counts, and a view-on-BGG link per candidate), plus hand-steered re-searches of BGG or RPGGeek when the automatic search can't reach the right database. Here: is that box base Agricola or the Revised Edition?
|
||||
|
||||

|
||||
|
||||
**The editions pass** — after matches are settled, an optional pass picks which *printing* each copy is, scored against the cues read off the box. Skippable per game, and never blocks uploads. Here: five English Catan editions to choose between.
|
||||
|
||||

|
||||
|
||||
**Queue** — exactly what upload will do before it does it, and a permanent log of every attempt ever made. Here: six adds and one version update, pending.
|
||||
|
||||

|
||||
|
||||
**Library** — the enriched collection: searchable across titles, designers and mechanics, filterable by player count, sortable by rank/weight/year/time. Every card opens a detail page joining BGG's data with your own shelf photos; off-BGG games take hand-written facts and a cover photo there.
|
||||
|
||||

|
||||
|
||||
**A game's detail page** — BGG's stats, chips, and description joined with what only the pipeline knows: *your* edition, and the shelf photo it was read from.
|
||||
|
||||

|
||||
|
||||
**Help** — the whole flow, every page, every status, and every keyboard shortcut, documented in-app.
|
||||
|
||||

|
||||
|
||||
**And on a phone** (`--lan`) — the same app, paired once by QR code: the hamburger menu, shooting shelf photos straight into the pipeline from the camera, proofreading titles from the couch, and the piper on the Help page.
|
||||
|
||||
<p>
|
||||
<img src="screenshots/08-phone-menu.jpeg" alt="Phone view: the hamburger menu open over the Photos page, showing all seven pages with attention badges" width="24%">
|
||||
<img src="screenshots/09-phone-camera.jpeg" alt="Phone view: tapping the photo drop zone offers iOS's Photo Library / Take Photo / Choose Files sheet" width="24%">
|
||||
<img src="screenshots/10-phone-titles.jpeg" alt="Phone view: the Titles page as stacked cards with status chips, edit and split buttons, and the shaky-reads filter" width="24%">
|
||||
<img src="screenshots/11-phone-piper.jpeg" alt="Phone view: the Help page's credits card with Juniper's full piper artwork and the BGG trademark attribution" width="24%">
|
||||
</p>
|
||||
@@ -2,7 +2,25 @@
|
||||
name = "bggpipe"
|
||||
dynamic = ["version"]
|
||||
description = "Shelf-to-BGG collection pipeline: photos in, BoardGameGeek collection out"
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
license-files = ["LICENSE"]
|
||||
requires-python = ">=3.12"
|
||||
authors = [{ name = "Eric Wagoner", email = "eric@ericwagoner.com" }]
|
||||
keywords = ["boardgamegeek", "bgg", "board-games", "collection", "vision", "cataloging"]
|
||||
classifiers = [
|
||||
# Beta is deliberate while the user base is one person on one platform,
|
||||
# however battle-tested that person's collection is
|
||||
"Development Status :: 4 - Beta",
|
||||
"Environment :: Console",
|
||||
"Environment :: Web Environment",
|
||||
"Intended Audience :: End Users/Desktop",
|
||||
"Operating System :: MacOS",
|
||||
"Operating System :: POSIX :: Linux",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Games/Entertainment :: Board Games",
|
||||
]
|
||||
|
||||
dependencies = [
|
||||
"typer>=0.12",
|
||||
"httpx>=0.27",
|
||||
@@ -17,8 +35,16 @@ dependencies = [
|
||||
"playwright>=1.62.0",
|
||||
"pydantic>=2.13.4",
|
||||
"python-multipart>=0.0.32",
|
||||
"qrcode>=8.2",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://kestrelsnest.social"
|
||||
Repository = "https://git.kestrelsnest.social/eric/bggpipe"
|
||||
Documentation = "https://git.kestrelsnest.social/eric/bggpipe/src/branch/main/docs/guide.md"
|
||||
Mastodon = "https://toots.kestrelsnest.social/@eric"
|
||||
Bluesky = "https://bsky.app/profile/kestrelsnest.social"
|
||||
|
||||
[project.scripts]
|
||||
bggpipe = "bggpipe.cli:app"
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Shelf-to-BGG collection pipeline."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__version__ = "1.0.0"
|
||||
|
||||
@@ -14,6 +14,17 @@ app = typer.Typer(
|
||||
no_args_is_help=True,
|
||||
)
|
||||
|
||||
|
||||
@app.callback()
|
||||
def _load_dotenv() -> None:
|
||||
# credentials live in ./.env (written by init); load them so a fresh
|
||||
# directory works without direnv or manual sourcing. Real environment
|
||||
# variables always take precedence over the file.
|
||||
from bggpipe.init_wizard import load_env_file
|
||||
|
||||
load_env_file(Path(".env"))
|
||||
|
||||
|
||||
ConfigOpt = Annotated[
|
||||
Path | None,
|
||||
typer.Option("--config", help="Path to config.toml (default: ./config.toml)"),
|
||||
@@ -96,6 +107,14 @@ def web(
|
||||
no_browser: Annotated[
|
||||
bool, typer.Option("--no-browser", help="Don't open a browser tab")
|
||||
] = False,
|
||||
lan: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
"--lan",
|
||||
help="Also serve to your local network behind a per-run access "
|
||||
"key (trusted networks only)",
|
||||
),
|
||||
] = False,
|
||||
config: ConfigOpt = None,
|
||||
) -> None:
|
||||
"""The whole pipeline in a local web UI: photos, stages, review."""
|
||||
@@ -106,6 +125,7 @@ def web(
|
||||
cfg,
|
||||
port=port,
|
||||
dev=dev,
|
||||
lan=lan,
|
||||
config_path=config,
|
||||
landing="/",
|
||||
open_browser=not no_browser,
|
||||
|
||||
@@ -9,6 +9,7 @@ account has exactly one home.
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import tomllib
|
||||
import warnings
|
||||
from dataclasses import dataclass, replace
|
||||
@@ -22,6 +23,9 @@ STUB_CACHE_MARKER_NAME = "STUB_FIXTURES.marker"
|
||||
STUB_DATA_MARKER_NAME = "STUB_DATA.marker"
|
||||
|
||||
|
||||
VISION_PROVIDERS = ("anthropic", "openai-compatible")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Config:
|
||||
bgg_username: str = ""
|
||||
@@ -29,6 +33,11 @@ class Config:
|
||||
data_dir: Path = Path("data")
|
||||
model: str = "claude-sonnet-5"
|
||||
rate_limit_seconds: float = 2.0
|
||||
# "anthropic" (default) or "openai-compatible" — the latter covers
|
||||
# OpenAI, OpenRouter, and local runtimes (Ollama, LM Studio, vLLM)
|
||||
vision_provider: str = "anthropic"
|
||||
vision_base_url: str = "" # e.g. http://localhost:11434/v1 for Ollama
|
||||
vision_key_env: str = "OPENAI_API_KEY" # "" = endpoint needs no key
|
||||
|
||||
@property
|
||||
def cache_dir(self) -> Path:
|
||||
@@ -66,6 +75,61 @@ class Config:
|
||||
def games_path(self) -> Path:
|
||||
return self.data_dir / "games.json"
|
||||
|
||||
def tracked_data_warning(self) -> str | None:
|
||||
"""The clone-and-run trap: pipeline data written at git-TRACKED
|
||||
paths is one `git pull` + one panicked `git reset --hard` away
|
||||
from destruction. Warn whenever artifacts under data_dir are
|
||||
tracked — unless the owner has marked this checkout as their own
|
||||
repo (the marker is gitignored, so a fresh clone never inherits
|
||||
the exemption)."""
|
||||
if (self.data_dir / ".own_repo").exists():
|
||||
return None
|
||||
try:
|
||||
tracked = subprocess.run( # noqa: S603
|
||||
["git", "ls-files", "--", str(self.data_dir)], # noqa: S607
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
cwd=self.data_dir.resolve().parent,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return None # no git, no repo, no trap
|
||||
if tracked.returncode != 0 or not tracked.stdout.strip():
|
||||
return None
|
||||
return (
|
||||
f"your pipeline data ({self.data_dir}) sits at git-TRACKED "
|
||||
"paths — a `git pull` here can refuse to merge, and the usual "
|
||||
"remedies (reset --hard, checkout ., stash) would destroy your "
|
||||
"review decisions, hand-written games, and upload log. Run "
|
||||
"bggpipe from a directory outside this repository (see README: "
|
||||
"Bring your own shelves). If this repo is genuinely where you "
|
||||
f"version your own data, `touch {self.data_dir}/.own_repo` to "
|
||||
"accept the arrangement and silence this warning."
|
||||
)
|
||||
|
||||
@property
|
||||
def local_games_path(self) -> Path:
|
||||
# hand-written metadata for games BGG doesn't have — the only
|
||||
# source of truth for them, so it is committed like the other stores
|
||||
return self.data_dir / "local_games.json"
|
||||
|
||||
@property
|
||||
def local_art_dir(self) -> Path:
|
||||
# cover photos for off-BGG games (committed: nothing else has them)
|
||||
return self.data_dir / "local_art"
|
||||
|
||||
@property
|
||||
def title_additions_path(self) -> Path:
|
||||
# games the human added without a photo (expansions stored inside
|
||||
# base boxes, unshelved games) — joined into every rebuild
|
||||
return self.data_dir / "title_additions.json"
|
||||
|
||||
@property
|
||||
def title_removals_path(self) -> Path:
|
||||
# titles the human removed from the catalog (not a game, misread) —
|
||||
# filtered out of every titles.json rebuild
|
||||
return self.data_dir / "title_removals.json"
|
||||
|
||||
@property
|
||||
def title_edits_path(self) -> Path:
|
||||
# human corrections to extracted reads (misspellings, known cues) —
|
||||
@@ -89,6 +153,12 @@ class Config:
|
||||
self.data_dir / "collection_snapshot_expansions.xml",
|
||||
)
|
||||
|
||||
@property
|
||||
def lan_key_path(self) -> Path:
|
||||
# the --lan access key, persisted so restarts don't strand phones'
|
||||
# cookies; credential-adjacent like Playwright state — gitignored
|
||||
return self.data_dir / ".lan_key"
|
||||
|
||||
@property
|
||||
def storage_state_path(self) -> Path:
|
||||
# cwd-relative on purpose (credential-adjacent, gitignored) but
|
||||
@@ -119,16 +189,47 @@ def load_config(path: Path | None = None) -> Config:
|
||||
"data_dir": Path,
|
||||
"model": str,
|
||||
"rate_limit_seconds": float,
|
||||
"vision_provider": str,
|
||||
}
|
||||
updates = {key: caster(raw[key]) for key, caster in known.items() if key in raw}
|
||||
if unknown := sorted(raw.keys() - known.keys()):
|
||||
if unknown := sorted(raw.keys() - known.keys() - {"vision"}):
|
||||
# a typo'd key silently falling back to defaults is a debugging
|
||||
# trap ("No photos found in photos/") — say so up front
|
||||
warnings.warn(
|
||||
f"{p}: ignoring unknown key(s): {', '.join(unknown)}",
|
||||
stacklevel=2,
|
||||
)
|
||||
# [vision.<provider>] blocks: every provider's recipe can live in
|
||||
# the committed file; only the ACTIVE provider's block applies
|
||||
vision_blocks = raw.get("vision") or {}
|
||||
if stray := sorted(vision_blocks.keys() - set(VISION_PROVIDERS)):
|
||||
warnings.warn(
|
||||
f"{p}: ignoring [vision.*] block(s) for unknown provider(s): "
|
||||
f"{', '.join(stray)}",
|
||||
stacklevel=2,
|
||||
)
|
||||
provider = updates.get("vision_provider", cfg.vision_provider)
|
||||
block = vision_blocks.get(provider) or {}
|
||||
block_known = {
|
||||
"model": "model",
|
||||
"base_url": "vision_base_url",
|
||||
"key_env": "vision_key_env",
|
||||
}
|
||||
if odd := sorted(block.keys() - block_known.keys()):
|
||||
warnings.warn(
|
||||
f"{p}: [vision.{provider}]: ignoring unknown key(s): {', '.join(odd)}",
|
||||
stacklevel=2,
|
||||
)
|
||||
for key, field in block_known.items():
|
||||
if key in block:
|
||||
updates[field] = str(block[key])
|
||||
cfg = replace(cfg, **updates)
|
||||
if cfg.vision_provider not in VISION_PROVIDERS:
|
||||
# a typo here would surface as a confusing extract failure later
|
||||
raise ValueError(
|
||||
f"{p}: vision_provider must be one of {', '.join(VISION_PROVIDERS)}"
|
||||
f" (got {cfg.vision_provider!r})"
|
||||
)
|
||||
if username := os.environ.get("BGG_USERNAME"):
|
||||
cfg = replace(cfg, bgg_username=username)
|
||||
return cfg
|
||||
|
||||
@@ -135,6 +135,11 @@ def compute_diff(rows: list[dict], collection: list[CollectionItem]) -> DiffResu
|
||||
if status == "merged":
|
||||
result.merged += 1 # represented by its survivor row
|
||||
continue
|
||||
if status == "local":
|
||||
# the human's call: a real game BGG doesn't have — a library
|
||||
# citizen only, never queued
|
||||
result.local_only.append(row["title_raw"])
|
||||
continue
|
||||
if status not in RECOGNIZED_MATCH_STATUSES or not row["bgg_id"]:
|
||||
result.pending.append(row["title_raw"])
|
||||
continue
|
||||
@@ -178,25 +183,13 @@ def compute_diff(rows: list[dict], collection: list[CollectionItem]) -> DiffResu
|
||||
else:
|
||||
leftover.append(row)
|
||||
|
||||
# 1b — versionless copies get upgraded. Guard: upload's row-edit flow
|
||||
# targets rows by game NAME (a collid can't drive the UI), so an update
|
||||
# is only safe when EVERY copy of the game is versionless — otherwise
|
||||
# the browser could open the versioned copy and overwrite it.
|
||||
# 1b — versionless copies get upgraded. The update flow addresses the
|
||||
# copy by collid and the edition by radio value, so a versioned
|
||||
# sibling copy is never at risk.
|
||||
still_left: list[dict] = []
|
||||
for row in leftover:
|
||||
bgg_id = int(row["bgg_id"])
|
||||
versionless = [c for c in unconsumed(bgg_id) if c.version_id is None]
|
||||
any_versioned = any(c.version_id is not None for c in by_object.get(bgg_id, []))
|
||||
if versionless and any_versioned:
|
||||
consumed_collids.add(versionless[0].coll_id)
|
||||
result.already_owned.append(row["title_raw"])
|
||||
result.disagreements.append(
|
||||
f"{row['title_raw']}: a versionless copy could take version "
|
||||
f"{row['version_name']!r}, but another copy already carries "
|
||||
"a version — set it by hand on BGG (the automated row edit "
|
||||
"can't safely target a specific copy)"
|
||||
)
|
||||
continue
|
||||
if versionless:
|
||||
target = versionless[0]
|
||||
consumed_collids.add(target.coll_id)
|
||||
@@ -316,7 +309,9 @@ def run_diff(cfg: Config, *, client: BGGClient | None = None) -> DiffResult:
|
||||
|
||||
merged_note = f" · {result.merged} merged duplicate(s)" if result.merged else ""
|
||||
local_note = (
|
||||
f" · {len(result.local_only)} local-only (RPGs)" if result.local_only else ""
|
||||
f" · {len(result.local_only)} local-only (RPGs, off-BGG games)"
|
||||
if result.local_only
|
||||
else ""
|
||||
)
|
||||
typer.echo(
|
||||
f"\n{result.recognized} recognized · {len(result.already_owned)} already "
|
||||
|
||||
@@ -28,7 +28,8 @@ from bggpipe.bgg_client import (
|
||||
from bggpipe.config import Config
|
||||
from bggpipe.fsio import atomic_write_text
|
||||
from bggpipe.models import is_confident_version, is_recognized
|
||||
from bggpipe.resolve import read_matches
|
||||
from bggpipe.normalize import normalize_title
|
||||
from bggpipe.resolve import load_titles, read_matches
|
||||
|
||||
BATCH_SIZE = 20
|
||||
|
||||
@@ -95,12 +96,86 @@ def run_enrich(
|
||||
games[key] = {**fetched[bgg_id], "version": version}
|
||||
updated += 1
|
||||
|
||||
# games the human ruled off-BGG: library entries built from their own
|
||||
# photo reads — no API involved, so a blocked run still lands them
|
||||
local_keys: set[str] = set()
|
||||
local_rows = [r for r in rows if r["match_status"] == "local"]
|
||||
# hand-written metadata wins over the photo reads: for an off-BGG game
|
||||
# it is the only real source there is
|
||||
hand: dict = {}
|
||||
if cfg.local_games_path.exists():
|
||||
try:
|
||||
hand = json.loads(cfg.local_games_path.read_text())
|
||||
except json.JSONDecodeError as err:
|
||||
raise ValueError(
|
||||
f"{cfg.local_games_path} is corrupt ({err}) — it holds "
|
||||
"hand-written game data, so check git history before deleting"
|
||||
) from err
|
||||
if local_rows:
|
||||
try:
|
||||
cues = {
|
||||
(e.title_raw, ";".join(e.source_photos)): e
|
||||
for e in load_titles(cfg.titles_path)
|
||||
}
|
||||
except FileNotFoundError:
|
||||
cues = {}
|
||||
for row in local_rows:
|
||||
key = f"local:{normalize_title(row['title_raw'])}:{row['source_photos']}"
|
||||
local_keys.add(key)
|
||||
entry = cues.get((row["title_raw"], row["source_photos"]))
|
||||
games[key] = {
|
||||
"bgg_id": None,
|
||||
"name": row["title_raw"],
|
||||
"year": entry.year_hint if entry else None,
|
||||
"type": "localgame",
|
||||
"publishers": [entry.publisher_hint]
|
||||
if entry and entry.publisher_hint
|
||||
else [],
|
||||
"source_photos": [p for p in row["source_photos"].split(";") if p],
|
||||
}
|
||||
games[key].update(
|
||||
{k: v for k, v in (hand.get(key) or {}).items() if v not in (None, "")}
|
||||
)
|
||||
# the key embeds the photo list, so a new sighting or a title edit
|
||||
# strands hand data under a key no row produces anymore. Same
|
||||
# normalized title + exactly one candidate = unambiguous: migrate.
|
||||
# Anything else is reported, never silently dropped.
|
||||
orphans = [k for k in hand if k.startswith("local:") and k not in local_keys]
|
||||
migrated = {}
|
||||
for old_key in orphans:
|
||||
title_part = old_key.split(":", 2)[1]
|
||||
candidates = [k for k in local_keys if k.split(":", 2)[1] == title_part]
|
||||
if len(candidates) == 1 and candidates[0] not in hand:
|
||||
new_key = candidates[0]
|
||||
hand[new_key] = hand.pop(old_key)
|
||||
migrated[old_key] = new_key
|
||||
games[new_key].update(
|
||||
{k: v for k, v in hand[new_key].items() if v not in (None, "")}
|
||||
)
|
||||
else:
|
||||
typer.echo(
|
||||
f" warning: hand-written data for {old_key!r} matches "
|
||||
"no current catalog line — the facts are safe in "
|
||||
f"{cfg.local_games_path.name} but will not show in the "
|
||||
"library until the key matches again"
|
||||
)
|
||||
if migrated:
|
||||
atomic_write_text(
|
||||
cfg.local_games_path,
|
||||
json.dumps(hand, indent=2, ensure_ascii=False, sort_keys=True) + "\n",
|
||||
)
|
||||
for old_key, new_key in migrated.items():
|
||||
typer.echo(
|
||||
f" migrated hand-written data {old_key!r} -> {new_key!r} "
|
||||
"(photo set changed; same title)"
|
||||
)
|
||||
|
||||
# prune keys no current target claims: a row whose version was approved
|
||||
# after a bare-key run (or was later rejected) must not leave an orphan
|
||||
# entry in the frontend seed data. Only safe when nothing was blocked —
|
||||
# a token-less run knows too little to declare anything stale.
|
||||
if not blocked:
|
||||
current = {key for key, _, _ in targets}
|
||||
current = {key for key, _, _ in targets} | local_keys
|
||||
stale = [k for k in games if k not in current]
|
||||
for k in stale:
|
||||
del games[k]
|
||||
@@ -114,10 +189,18 @@ def run_enrich(
|
||||
games_path, json.dumps(games, indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
|
||||
# count the two populations separately: local entries are built from
|
||||
# photo reads and have no fetch target, so folding them into the API
|
||||
# tally would drive "already present or waiting" negative
|
||||
waiting = len(targets) - updated
|
||||
parts = [f"{updated} fetched from BGG"]
|
||||
if local_keys:
|
||||
parts.append(f"{len(local_keys)} local-only")
|
||||
if waiting:
|
||||
parts.append(f"{waiting} already present or waiting")
|
||||
typer.echo(
|
||||
f"games.json: {len(games)} entr{'y' if len(games) == 1 else 'ies'} "
|
||||
f"({updated} added/refreshed this run; "
|
||||
f"{len(targets) - updated} already present or waiting)."
|
||||
f"({'; '.join(parts)})."
|
||||
)
|
||||
if blocked:
|
||||
remaining = [i for i in need if i not in fetched]
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
"""Stage 1 — extract game titles + edition cues from shelf photos.
|
||||
|
||||
Each photo is sent to Claude vision once and the raw result is cached under
|
||||
data/extract_raw/<photo>.json (making re-runs free and --only targeted).
|
||||
Each photo is sent to the configured vision model once and the raw result
|
||||
is cached under data/extract_raw/<photo>.json (making re-runs free and
|
||||
--only targeted).
|
||||
titles.json is rebuilt from all raw files on every run, deduping identical
|
||||
titles across photos unless their edition cues conflict — conflicting cues
|
||||
mean two different editions on the shelf, which stay separate entries.
|
||||
@@ -12,6 +13,7 @@ from __future__ import annotations
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
@@ -125,7 +127,12 @@ def parse_vision_response(text: str) -> tuple[list[dict], list[dict], int]:
|
||||
if end == -1:
|
||||
raise ValueError(f"unterminated JSON in vision response: {text[:200]!r}")
|
||||
cleaned = cleaned[start : end + 1]
|
||||
try:
|
||||
data = json.loads(cleaned)
|
||||
except json.JSONDecodeError:
|
||||
# local models emit almost-JSON (trailing commas, mostly): one
|
||||
# cheap repair pass before declaring the response unusable
|
||||
data = json.loads(re.sub(r",\s*([}\]])", r"\1", cleaned))
|
||||
if isinstance(data, list):
|
||||
titles_raw, unidentified_raw = data, []
|
||||
elif isinstance(data, dict):
|
||||
@@ -176,6 +183,61 @@ def default_vision(model: str) -> VisionFn:
|
||||
return vision
|
||||
|
||||
|
||||
def openai_compatible_vision(model: str, base_url: str, key_env: str) -> VisionFn:
|
||||
"""Any endpoint speaking the OpenAI chat-completions format: OpenAI
|
||||
itself, OpenRouter, or a local runtime (Ollama, LM Studio, llama.cpp,
|
||||
vLLM — pass its /v1 base URL). A key is optional: local runtimes run
|
||||
without one, so an empty key_env just sends no Authorization header."""
|
||||
import httpx
|
||||
|
||||
headers = {}
|
||||
if key_env and (key := os.environ.get(key_env)):
|
||||
headers["Authorization"] = f"Bearer {key}"
|
||||
|
||||
def vision(image_b64: str, media_type: str) -> str:
|
||||
response = httpx.post(
|
||||
base_url.rstrip("/") + "/chat/completions",
|
||||
headers=headers,
|
||||
json={
|
||||
"model": model,
|
||||
"max_tokens": 4000,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:{media_type};base64,{image_b64}"
|
||||
},
|
||||
},
|
||||
{"type": "text", "text": VISION_PROMPT},
|
||||
],
|
||||
}
|
||||
],
|
||||
},
|
||||
timeout=300.0, # local models on modest hardware are slow
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()["choices"][0]["message"]["content"] or ""
|
||||
|
||||
return vision
|
||||
|
||||
|
||||
def vision_for(cfg: Config) -> VisionFn:
|
||||
"""The configured vision backend (config.toml: vision_provider)."""
|
||||
if cfg.vision_provider == "openai-compatible":
|
||||
if not cfg.vision_base_url:
|
||||
raise ValueError(
|
||||
"vision_provider 'openai-compatible' needs vision_base_url "
|
||||
"in config.toml (e.g. http://localhost:11434/v1 for Ollama)"
|
||||
)
|
||||
return openai_compatible_vision(
|
||||
cfg.model, cfg.vision_base_url, cfg.vision_key_env
|
||||
)
|
||||
return default_vision(cfg.model)
|
||||
|
||||
|
||||
def extract_photo(photo: Path, vision: VisionFn) -> dict:
|
||||
"""One photo -> {"titles": [...], "unidentified": [...]} (the raw-cache
|
||||
file format)."""
|
||||
@@ -243,6 +305,7 @@ def _merge(a: dict, b: dict) -> dict:
|
||||
# the fields a human correction may override on an extracted entry
|
||||
EDIT_FIELDS = (
|
||||
"title_raw",
|
||||
"confidence",
|
||||
"publisher_hint",
|
||||
"edition_hint",
|
||||
"year_hint",
|
||||
@@ -313,11 +376,10 @@ def apply_title_edits(entries: list[dict], edits: list[dict]) -> list[dict]:
|
||||
return entries
|
||||
|
||||
|
||||
def load_title_splits(path: Path) -> list[dict]:
|
||||
"""The human's split-into-copies decisions — durable: they must survive
|
||||
extract rebuilds and resolve --force. Each stored record is
|
||||
{"title": ..., "photos": [...]} scoping the split to the sightings that
|
||||
were on the split line (a bare string is legacy: unscoped). Returns
|
||||
def _scoped_records(path: Path) -> list[dict]:
|
||||
"""Parse a store of photo-scoped title decisions. Each stored record is
|
||||
{"title": ..., "photos": [...]} scoping the decision to the sightings
|
||||
that were on the line (a bare string is legacy: unscoped). Returns
|
||||
[{"norm": <normalized title>, "photos": set | None}]."""
|
||||
records = []
|
||||
for item in _load_store(path):
|
||||
@@ -334,6 +396,67 @@ def load_title_splits(path: Path) -> list[dict]:
|
||||
return records
|
||||
|
||||
|
||||
def load_title_additions(path: Path) -> list[dict]:
|
||||
"""Entries the human added without a photo — an expansion stored inside
|
||||
a base box, a game away from the shelves. Shaped like raw reads and fed
|
||||
into every rebuild BEFORE edits and dedupe, so corrections apply and a
|
||||
later photo sighting of the same game merges instead of duplicating."""
|
||||
return _load_store(path)
|
||||
|
||||
|
||||
def record_title_addition(path: Path, entry: dict) -> bool:
|
||||
"""Record a hand-added game. False = an addition with this normalized
|
||||
title is already on file (nothing written — edit that line instead);
|
||||
a silent True here would let the caller report success for a no-op."""
|
||||
existing = load_title_additions(path)
|
||||
norm = normalize_title(entry["title_raw"])
|
||||
if any(normalize_title(e["title_raw"]) == norm for e in existing):
|
||||
return False
|
||||
atomic_write_text(
|
||||
path, json.dumps([*existing, entry], indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def rescind_title_removal(path: Path, title: str) -> bool:
|
||||
"""Drop any UNSCOPED removal of this normalized title (a hand-added
|
||||
line removed earlier). Re-adding a title those records would filter
|
||||
from every rebuild is an explicit undo, not a conflict. Photo-scoped
|
||||
removals stay: they veto specific sightings, not the game."""
|
||||
norm = normalize_title(title)
|
||||
stored = _load_store(path)
|
||||
kept = [
|
||||
item
|
||||
for item in stored
|
||||
if not (
|
||||
(isinstance(item, str) and normalize_title(item) == norm)
|
||||
or (
|
||||
isinstance(item, dict)
|
||||
and not item.get("photos")
|
||||
and normalize_title(item["title"]) == norm
|
||||
)
|
||||
)
|
||||
]
|
||||
if len(kept) == len(stored):
|
||||
return False
|
||||
atomic_write_text(path, json.dumps(kept, indent=2, ensure_ascii=False) + "\n")
|
||||
return True
|
||||
|
||||
|
||||
def load_title_splits(path: Path) -> list[dict]:
|
||||
"""The human's split-into-copies decisions — durable: they must survive
|
||||
extract rebuilds and resolve --force."""
|
||||
return _scoped_records(path)
|
||||
|
||||
|
||||
def load_title_removals(path: Path) -> list[dict]:
|
||||
"""Titles the human removed from the catalog (not a game, a misread of
|
||||
box art, out of scope) — their sightings are filtered out of every
|
||||
rebuild, so re-extraction cannot resurrect them. Undo by deleting the
|
||||
record from the store file."""
|
||||
return _scoped_records(path)
|
||||
|
||||
|
||||
def is_split(norm: str, photos, splits: list[dict]) -> bool:
|
||||
"""Does a split decision cover this (normalized title, photo set)?
|
||||
Photo-scoped records only bind sightings from the photos that were on
|
||||
@@ -345,9 +468,9 @@ def is_split(norm: str, photos, splits: list[dict]) -> bool:
|
||||
)
|
||||
|
||||
|
||||
def record_title_split(path: Path, title: str, photos: list[str] | None = None) -> None:
|
||||
if is_split(normalize_title(title), photos or [], load_title_splits(path)):
|
||||
return
|
||||
def _record_scoped(path: Path, title: str, photos: list[str] | None) -> None:
|
||||
if is_split(normalize_title(title), photos or [], _scoped_records(path)):
|
||||
return # an existing record already covers this line
|
||||
existing = _load_store(path)
|
||||
record = {"title": title, "photos": sorted(photos) if photos else None}
|
||||
atomic_write_text(
|
||||
@@ -355,6 +478,16 @@ def record_title_split(path: Path, title: str, photos: list[str] | None = None)
|
||||
)
|
||||
|
||||
|
||||
def record_title_split(path: Path, title: str, photos: list[str] | None = None) -> None:
|
||||
_record_scoped(path, title, photos)
|
||||
|
||||
|
||||
def record_title_removal(
|
||||
path: Path, title: str, photos: list[str] | None = None
|
||||
) -> None:
|
||||
_record_scoped(path, title, photos)
|
||||
|
||||
|
||||
def dedupe_entries(entries: list[dict], splits: list[dict] | None = None) -> list[dict]:
|
||||
"""Collapse same-normalized-title sightings unless their cues conflict —
|
||||
or unless the human declared them split (several physical copies):
|
||||
@@ -383,16 +516,33 @@ def dedupe_entries(entries: list[dict], splits: list[dict] | None = None) -> lis
|
||||
return result
|
||||
|
||||
|
||||
def apply_title_removals(entries: list[dict], removals: list[dict]) -> list[dict]:
|
||||
"""Filter out sightings the human removed. Runs AFTER edits (records
|
||||
key on the title as displayed when removal was clicked) and before
|
||||
dedupe (so a removed sighting can't be absorbed into a survivor)."""
|
||||
if not removals:
|
||||
return entries
|
||||
return [
|
||||
e
|
||||
for e in entries
|
||||
if not is_split(
|
||||
normalize_title(e["title_raw"]), e.get("source_photos"), removals
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def rebuild_artifacts(
|
||||
raw_dir: Path,
|
||||
titles_path: Path,
|
||||
unidentified_path: Path,
|
||||
splits: list[dict] | None = None,
|
||||
edits: list[dict] | None = None,
|
||||
removals: list[dict] | None = None,
|
||||
additions: list[dict] | None = None,
|
||||
) -> tuple[list[dict], dict[str, list[dict]]]:
|
||||
"""Regenerate titles.json and unidentified.json from the per-photo raw
|
||||
cache. A raw file is either an object with titles/unidentified or a bare
|
||||
arrays — still readable."""
|
||||
array — still readable."""
|
||||
entries: list[dict] = []
|
||||
unidentified: dict[str, list[dict]] = {}
|
||||
for raw_file in sorted(raw_dir.glob("*.json")):
|
||||
@@ -410,7 +560,11 @@ def rebuild_artifacts(
|
||||
photo = raw_file.name.removesuffix(".json")
|
||||
if data.get("unidentified"):
|
||||
unidentified[photo] = data["unidentified"]
|
||||
deduped = dedupe_entries(apply_title_edits(entries, edits or []), splits)
|
||||
entries = entries + [dict(e) for e in additions or []]
|
||||
entries = apply_title_removals(
|
||||
apply_title_edits(entries, edits or []), removals or []
|
||||
)
|
||||
deduped = dedupe_entries(entries, splits)
|
||||
titles_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
atomic_write_text(
|
||||
titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n"
|
||||
@@ -429,6 +583,8 @@ def replay_titles(cfg: Config) -> None:
|
||||
photo-level splits and targeted edits can take hold."""
|
||||
splits = load_title_splits(cfg.title_splits_path)
|
||||
edits = load_title_edits(cfg.title_edits_path)
|
||||
removals = load_title_removals(cfg.title_removals_path)
|
||||
additions = load_title_additions(cfg.title_additions_path)
|
||||
raw_dir = cfg.extract_raw_dir
|
||||
raw_photos = (
|
||||
{f.name.removesuffix(".json") for f in raw_dir.glob("*.json")}
|
||||
@@ -444,7 +600,13 @@ def replay_titles(cfg: Config) -> None:
|
||||
# them would silently truncate the committed catalog
|
||||
if raw_photos and raw_photos >= known_photos:
|
||||
rebuild_artifacts(
|
||||
raw_dir, cfg.titles_path, cfg.unidentified_path, splits, edits
|
||||
raw_dir,
|
||||
cfg.titles_path,
|
||||
cfg.unidentified_path,
|
||||
splits,
|
||||
edits,
|
||||
removals,
|
||||
additions,
|
||||
)
|
||||
return
|
||||
if not cfg.titles_path.exists():
|
||||
@@ -456,7 +618,10 @@ def replay_titles(cfg: Config) -> None:
|
||||
exploded.extend({**entry, "source_photos": [p]} for p in photos)
|
||||
else:
|
||||
exploded.append(dict(entry))
|
||||
deduped = dedupe_entries(apply_title_edits(exploded, edits), splits)
|
||||
exploded.extend(dict(e) for e in additions)
|
||||
deduped = dedupe_entries(
|
||||
apply_title_removals(apply_title_edits(exploded, edits), removals), splits
|
||||
)
|
||||
atomic_write_text(
|
||||
cfg.titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n"
|
||||
)
|
||||
@@ -490,7 +655,7 @@ def run_extract(
|
||||
|
||||
raw_dir = cfg.extract_raw_dir
|
||||
raw_dir.mkdir(parents=True, exist_ok=True)
|
||||
vision = vision or default_vision(cfg.model)
|
||||
vision = vision or vision_for(cfg)
|
||||
|
||||
failed: list[str] = []
|
||||
consecutive: tuple[str, int] = ("", 0)
|
||||
@@ -547,6 +712,8 @@ def run_extract(
|
||||
cfg.unidentified_path,
|
||||
load_title_splits(cfg.title_splits_path),
|
||||
load_title_edits(cfg.title_edits_path),
|
||||
load_title_removals(cfg.title_removals_path),
|
||||
load_title_additions(cfg.title_additions_path),
|
||||
)
|
||||
typer.echo(f"Wrote {len(deduped)} unique title(s) to {cfg.titles_path}.")
|
||||
if failed:
|
||||
|
||||
@@ -28,8 +28,22 @@ CONFIG_TEMPLATE = """\
|
||||
|
||||
photos_dir = "photos"
|
||||
data_dir = "data"
|
||||
model = "claude-sonnet-5"
|
||||
rate_limit_seconds = 2.0
|
||||
|
||||
# Which vision backend reads your shelf photos. Both recipes below stay
|
||||
# on file; this line picks one.
|
||||
vision_provider = "anthropic"
|
||||
|
||||
[vision.anthropic]
|
||||
# reads ANTHROPIC_API_KEY from the environment
|
||||
model = "claude-sonnet-5"
|
||||
|
||||
[vision."openai-compatible"]
|
||||
# OpenAI, OpenRouter, or a local runtime (Ollama, LM Studio, vLLM).
|
||||
# key_env names the env var holding the key; "" = endpoint needs none.
|
||||
base_url = "http://localhost:11434/v1"
|
||||
model = "qwen2.5vl:7b"
|
||||
key_env = ""
|
||||
"""
|
||||
|
||||
ENV_HEADER = """\
|
||||
@@ -74,6 +88,32 @@ class InitReport:
|
||||
keys_written: list[str] = field(default_factory=list)
|
||||
keys_missing: list[str] = field(default_factory=list)
|
||||
browser_installed: bool = False
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
def load_env_file(env_path: Path) -> list[str]:
|
||||
"""Export .env values into this process's environment — only keys the
|
||||
environment doesn't already set (a real env var always wins). Returns
|
||||
the keys loaded. Same parsing rules as _env_file_keys; values go
|
||||
straight into os.environ and are never printed or logged."""
|
||||
if not env_path.exists():
|
||||
return []
|
||||
loaded = []
|
||||
for line in env_path.read_text().splitlines():
|
||||
line = line.strip()
|
||||
if "=" not in line or line.startswith("#"):
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
key = key.strip()
|
||||
if key.startswith("export "):
|
||||
key = key.removeprefix("export ").strip()
|
||||
value = value.strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"":
|
||||
value = value[1:-1]
|
||||
if key and value and not os.environ.get(key):
|
||||
os.environ[key] = value
|
||||
loaded.append(key)
|
||||
return loaded
|
||||
|
||||
|
||||
def _env_file_keys(env_path: Path) -> set[str]:
|
||||
@@ -139,6 +179,9 @@ def run_init(
|
||||
|
||||
report = InitReport()
|
||||
typer.echo("bggpipe setup — checks what exists, fills only the gaps.\n")
|
||||
if trap := cfg.tracked_data_warning():
|
||||
typer.echo(f"WARNING: {trap}\n", err=True)
|
||||
report.warnings.append(trap)
|
||||
|
||||
# -- directories ----------------------------------------------------
|
||||
for path in (project_dir / cfg.photos_dir, project_dir / cfg.data_dir):
|
||||
@@ -159,9 +202,33 @@ def run_init(
|
||||
typer.echo(f" found {config_path}")
|
||||
|
||||
# -- credentials ----------------------------------------------------
|
||||
# the vision key follows the configured provider: an openai-compatible
|
||||
# setup prompts for ITS key env (or none, for a local endpoint)
|
||||
from bggpipe.config import load_config
|
||||
|
||||
active = load_config(config_path if config_path.exists() else None)
|
||||
env_keys = list(ENV_KEYS)
|
||||
if active.vision_provider != "anthropic":
|
||||
env_keys = [k for k in env_keys if k[0] != "ANTHROPIC_API_KEY"]
|
||||
if active.vision_key_env:
|
||||
env_keys.insert(
|
||||
0,
|
||||
(
|
||||
active.vision_key_env,
|
||||
True,
|
||||
"vision extraction via "
|
||||
+ (active.vision_base_url or "the configured endpoint"),
|
||||
"your vision provider's console",
|
||||
),
|
||||
)
|
||||
else:
|
||||
typer.echo(
|
||||
" vision: openai-compatible endpoint with no key configured "
|
||||
"(local runtime) — nothing to prompt for"
|
||||
)
|
||||
env_path = project_dir / ".env"
|
||||
in_file = _env_file_keys(env_path)
|
||||
for key, secret, why, where in ENV_KEYS:
|
||||
for key, secret, why, where in env_keys:
|
||||
if os.environ.get(key) or key in in_file:
|
||||
report.keys_ready.append(key)
|
||||
typer.echo(f" {key}: set")
|
||||
|
||||
@@ -94,21 +94,34 @@ def _attr_int(elem: ET.Element | None, attr: str = "value") -> int | None:
|
||||
|
||||
def parse_search(xml_text: str) -> list[SearchResult]:
|
||||
results = []
|
||||
by_id: dict[int, int] = {} # bgg_id -> index in results
|
||||
skipped = 0
|
||||
for item in _root(xml_text).findall("item"):
|
||||
name_elem = item.find("name")
|
||||
if name_elem is None or item.get("id") is None:
|
||||
skipped += 1 # tolerate stragglers; wholesale drift raises below
|
||||
continue
|
||||
results.append(
|
||||
SearchResult(
|
||||
result = SearchResult(
|
||||
bgg_id=int(item.get("id")),
|
||||
name=name_elem.get("value", ""),
|
||||
name_type=name_elem.get("type", "primary"),
|
||||
year=_attr_int(item.find("yearpublished")),
|
||||
type=item.get("type", "boardgame"),
|
||||
)
|
||||
)
|
||||
# a multi-type search lists an expansion TWICE — once per matched
|
||||
# type; collapse only SAME-NAME duplicates, preferring the specific
|
||||
# type so expansion tagging (the base-vs-expansion review guard)
|
||||
# survives. A duplicate under a DIFFERENT name stays: it may be the
|
||||
# alternate name that exact-matches the query, and dropping it
|
||||
# would silently downgrade the match to fuzzy.
|
||||
dedupe_key = (result.bgg_id, result.name.casefold())
|
||||
if dedupe_key in by_id:
|
||||
seen = results[by_id[dedupe_key]]
|
||||
if seen.type == "boardgame" and result.type != "boardgame":
|
||||
results[by_id[dedupe_key]] = result
|
||||
continue
|
||||
by_id[dedupe_key] = len(results)
|
||||
results.append(result)
|
||||
if skipped and results:
|
||||
warnings.warn(
|
||||
f"search: {skipped} unparseable item(s) tolerated — schema drift?",
|
||||
@@ -146,6 +159,8 @@ def parse_things(xml_text: str) -> list[ThingDetails]:
|
||||
for item in _root(xml_text).findall("item"):
|
||||
name = item.find("name[@type='primary']")
|
||||
rank_elem = item.find(".//ranks/rank[@name='boardgame']")
|
||||
if rank_elem is None: # RPGGeek items rank in their own family
|
||||
rank_elem = item.find(".//ranks/rank[@name='rpgitem']")
|
||||
versions = [
|
||||
v
|
||||
for v_item in item.findall("versions/item")
|
||||
@@ -223,18 +238,28 @@ def parse_things_full(xml_text: str) -> list[dict]:
|
||||
"min_playtime": _attr_int(item.find("minplaytime")),
|
||||
"max_playtime": _attr_int(item.find("maxplaytime")),
|
||||
"min_age": _attr_int(item.find("minage")),
|
||||
"designers": links("boardgamedesigner"),
|
||||
"artists": links("boardgameartist"),
|
||||
"publishers": links("boardgamepublisher"),
|
||||
"categories": links("boardgamecategory"),
|
||||
"mechanics": links("boardgamemechanic"),
|
||||
# RPGGeek items live in the same database but use their own
|
||||
# link types, so a board-game-only reader finds none of them
|
||||
"designers": links("boardgamedesigner") + links("rpgdesigner"),
|
||||
"artists": links("boardgameartist") + links("rpgartist"),
|
||||
"publishers": links("boardgamepublisher") + links("rpgpublisher"),
|
||||
"categories": links("boardgamecategory")
|
||||
+ links("rpggenre")
|
||||
+ links("rpgcategory"),
|
||||
"mechanics": links("boardgamemechanic") + links("rpgmechanic"),
|
||||
"producers": links("rpgproducer"),
|
||||
"series": links("rpgseries"),
|
||||
"rating": _attr_float(ratings.find("average"))
|
||||
if ratings is not None
|
||||
else None,
|
||||
"weight": _attr_float(ratings.find("averageweight"))
|
||||
if ratings is not None
|
||||
else None,
|
||||
"rank": _attr_int(item.find(".//ranks/rank[@name='boardgame']")),
|
||||
"rank": _attr_int(
|
||||
item.find(".//ranks/rank[@name='boardgame']")
|
||||
if item.find(".//ranks/rank[@name='boardgame']") is not None
|
||||
else item.find(".//ranks/rank[@name='rpgitem']")
|
||||
),
|
||||
"users_owned": _attr_int(ratings.find("owned"))
|
||||
if ratings is not None
|
||||
else None,
|
||||
|
||||
@@ -89,6 +89,7 @@ class Candidate:
|
||||
type: str
|
||||
exact: bool
|
||||
fuzzy: float
|
||||
sibling: bool = False # exactness earned via edition-suffix stripping
|
||||
owned: int | None = None
|
||||
rank: int | None = None
|
||||
publishers: list[str] = field(default_factory=list)
|
||||
@@ -190,11 +191,13 @@ def _truncation_heads(title_raw: str) -> list[str]:
|
||||
match = _GAME_WORD.search(title_raw)
|
||||
if match and match.start() > 0:
|
||||
heads.append(title_raw[: match.start()])
|
||||
# last resort: first two words — of the pre-subtitle part, so a title
|
||||
# like "Blorvath: Quest of the Zzyzx" never yields "Blorvath: Quest"
|
||||
# last resort: shrink from the right, longest first — a printed title
|
||||
# can bury the real name in the middle ("ALICE IS MISSING A SILENT ROLE
|
||||
# PLAYING GAME" is "Alice is Missing"). Only exact normalized matches
|
||||
# count for heads, so a short head cannot match loosely.
|
||||
words = (sep_head or title_raw).split()
|
||||
if len(words) > 2:
|
||||
heads.append(" ".join(words[:2]))
|
||||
for size in range(len(words) - 1, 1, -1):
|
||||
heads.append(" ".join(words[:size]))
|
||||
|
||||
seen: set[str] = {normalize_title(title_raw)}
|
||||
unique: list[str] = []
|
||||
@@ -203,7 +206,7 @@ def _truncation_heads(title_raw: str) -> list[str]:
|
||||
if norm and norm not in seen:
|
||||
seen.add(norm)
|
||||
unique.append(head)
|
||||
return unique[:3]
|
||||
return unique[:6]
|
||||
|
||||
|
||||
def _plausible_candidates(
|
||||
@@ -226,6 +229,18 @@ def _plausible_candidates(
|
||||
for result in results:
|
||||
norm = normalize_title(result.name)
|
||||
exact = norm == entry.title_normalized
|
||||
# BGG files new editions as SEPARATE games named "X (Nth Edition)":
|
||||
# a name that equals the title once its trailing parenthetical is
|
||||
# stripped is a sibling edition — exact-grade, or the match looks
|
||||
# unanimously confident while hiding the real choice (Wiz-War has
|
||||
# three same-named lineages; the spec's top failure mode)
|
||||
sibling = False
|
||||
if not exact and result.type != "boardgameexpansion":
|
||||
sibling = (
|
||||
normalize_title(_EDITION_SUFFIX.sub("", result.name))
|
||||
== entry.title_normalized
|
||||
)
|
||||
exact = sibling
|
||||
fuzzy = fuzz.token_sort_ratio(norm, entry.title_normalized)
|
||||
if not exact and fuzzy < FUZZY_THRESHOLD:
|
||||
if not (head_normalized and norm == head_normalized):
|
||||
@@ -237,6 +252,7 @@ def _plausible_candidates(
|
||||
year=result.year,
|
||||
type=result.type,
|
||||
exact=exact,
|
||||
sibling=sibling,
|
||||
fuzzy=fuzzy,
|
||||
)
|
||||
prev = by_id.get(result.bgg_id)
|
||||
@@ -255,7 +271,23 @@ def _classify(client: BGGClient, entry: TitleEntry, cands: list[Candidate]) -> M
|
||||
return row
|
||||
|
||||
if len(cands) == 1:
|
||||
chosen = cands[0]
|
||||
# a lone candidate must still earn trust: BGG's search visibly
|
||||
# truncates generic queries (the game named "Dungeon!" appears in
|
||||
# NEITHER of its own searches), so the sole survivor may be an
|
||||
# impostor standing where the famous game should be. Sibling-grade
|
||||
# exactness never autos alone, and a true exact must clear the
|
||||
# same ownership floor the dominance rule enforces.
|
||||
only = cands[0]
|
||||
stats = {t.bgg_id: t for t in client.things([only.bgg_id], stats=True)}
|
||||
if only.bgg_id in stats:
|
||||
only.owned = stats[only.bgg_id].owned
|
||||
only.rank = stats[only.bgg_id].rank
|
||||
only.publishers = list(stats[only.bgg_id].publishers)
|
||||
row.candidates = [only]
|
||||
if only.sibling or not only.exact or (only.owned or 0) < DOMINANCE_MIN_OWNED:
|
||||
row.match_status = "ambiguous"
|
||||
return row
|
||||
chosen = only
|
||||
else:
|
||||
top = cands[:5]
|
||||
stats = {
|
||||
@@ -380,7 +412,7 @@ def resolve_version(client: BGGClient, entry: TitleEntry, row: MatchRow) -> None
|
||||
"languages": list(v.languages),
|
||||
"score": s,
|
||||
}
|
||||
for v, s in plausible[:8]
|
||||
for v, s in plausible # every plausible version: no silent cap
|
||||
]
|
||||
if len(plausible) == 1 or plausible[0][1] > plausible[1][1]:
|
||||
winner = plausible[0][0]
|
||||
@@ -391,8 +423,40 @@ def resolve_version(client: BGGClient, entry: TitleEntry, row: MatchRow) -> None
|
||||
row.version_status = "version_ambiguous"
|
||||
|
||||
|
||||
def resolve_entry(client: BGGClient, entry: TitleEntry) -> MatchRow:
|
||||
cands = _plausible_candidates(client, entry, entry.title_raw)
|
||||
_EDITION_SUFFIX = re.compile(r"\s*\([^)]*\)\s*$")
|
||||
_PUNCT = re.compile(r"[^\w\s]", re.UNICODE)
|
||||
|
||||
|
||||
def _depunct(title: str) -> str:
|
||||
"""BGG's search engine can choke on punctuation — the game literally
|
||||
named "Dungeon!" is missing from its own 824-result search. A plain
|
||||
query recovers it."""
|
||||
return " ".join(_PUNCT.sub(" ", title).split())
|
||||
|
||||
|
||||
def _merged_candidates(
|
||||
client: BGGClient, entry: TitleEntry, query: str, types: str | None = None
|
||||
) -> list[Candidate]:
|
||||
"""Search the raw title AND its punctuation-free form, merged by id
|
||||
(strongest evidence wins)."""
|
||||
cands = _plausible_candidates(client, entry, query, types=types)
|
||||
plain = _depunct(query)
|
||||
if plain.casefold() != query.casefold():
|
||||
by_id = {c.bgg_id: c for c in cands}
|
||||
for c in _plausible_candidates(client, entry, plain, types=types):
|
||||
prev = by_id.get(c.bgg_id)
|
||||
if prev is None or (c.exact, c.fuzzy) > (prev.exact, prev.fuzzy):
|
||||
by_id[c.bgg_id] = c
|
||||
cands = sorted(by_id.values(), key=lambda c: (not c.exact, -c.fuzzy))
|
||||
return cands
|
||||
|
||||
|
||||
def find_candidates(client: BGGClient, entry: TitleEntry) -> list[Candidate]:
|
||||
"""Every search this pipeline knows, in order of confidence. Public
|
||||
because review's reopen path must search exactly as resolve did — a
|
||||
partial re-implementation there silently misses whatever the later
|
||||
steps would have found."""
|
||||
cands = _merged_candidates(client, entry, entry.title_raw)
|
||||
if not cands:
|
||||
# Long transcribed box titles ("CIVILIZATION Game of the Heroic
|
||||
# Age - ...") defeat search: retry with progressively shorter heads
|
||||
@@ -406,7 +470,7 @@ def resolve_entry(client: BGGClient, entry: TitleEntry) -> MatchRow:
|
||||
# type=rpgitem (same API, same token). A hit becomes a LOCAL
|
||||
# library citizen: identified and enriched, never uploaded (diff
|
||||
# routes rpgitem rows to local_only).
|
||||
cands = _plausible_candidates(client, entry, entry.title_raw, types="rpgitem")
|
||||
cands = _merged_candidates(client, entry, entry.title_raw, types="rpgitem")
|
||||
if not cands:
|
||||
for head in _truncation_heads(entry.title_raw):
|
||||
cands = _plausible_candidates(
|
||||
@@ -414,6 +478,11 @@ def resolve_entry(client: BGGClient, entry: TitleEntry) -> MatchRow:
|
||||
)
|
||||
if cands:
|
||||
break
|
||||
return cands
|
||||
|
||||
|
||||
def resolve_entry(client: BGGClient, entry: TitleEntry) -> MatchRow:
|
||||
cands = find_candidates(client, entry)
|
||||
row = _classify(client, entry, cands)
|
||||
if row.match_status == "auto":
|
||||
resolve_version(client, entry, row)
|
||||
@@ -587,8 +656,6 @@ def run_resolve(
|
||||
return candidates[0]
|
||||
return None
|
||||
|
||||
seen_per_title: dict[str, int] = {}
|
||||
|
||||
new_rows: list[MatchRow] = []
|
||||
skipped = 0
|
||||
photos_updated = False
|
||||
@@ -607,7 +674,6 @@ def run_resolve(
|
||||
paired_by_id[id(entry)] = row_dict
|
||||
|
||||
for entry in entries:
|
||||
seen_per_title[entry.title_raw] = seen_per_title.get(entry.title_raw, 0) + 1
|
||||
row_dict = paired_by_id.get(id(entry))
|
||||
if row_dict is not None:
|
||||
photos = ";".join(entry.source_photos)
|
||||
@@ -662,6 +728,13 @@ def run_resolve(
|
||||
f"Resolved {len(new_rows)} title(s) ({summary or 'nothing new'}); "
|
||||
f"skipped {skipped} already in {cfg.matches_path}."
|
||||
)
|
||||
waiting = sum(1 for r in all_rows if r.get("match_status") == "unmatched")
|
||||
if waiting:
|
||||
typer.echo(
|
||||
f" {waiting} unmatched row(s) wait for YOUR call in review "
|
||||
"(re-search or manual id) — resolve never overrides a human "
|
||||
"decision, including 'this match is wrong'"
|
||||
)
|
||||
if blocked:
|
||||
typer.echo(
|
||||
f"\n{len(blocked)} title(s) are waiting on the BGG API "
|
||||
|
||||
@@ -30,8 +30,11 @@ from bggpipe.models import (
|
||||
)
|
||||
from bggpipe.normalize import normalize_title
|
||||
from bggpipe.resolve import (
|
||||
Candidate,
|
||||
MatchRow,
|
||||
TitleEntry,
|
||||
_score_version,
|
||||
find_candidates,
|
||||
load_titles,
|
||||
read_matches,
|
||||
resolve_version,
|
||||
@@ -270,13 +273,16 @@ class ReviewSession:
|
||||
title_raw: str,
|
||||
photos: list[str] | None = None,
|
||||
new_title: str | None = None,
|
||||
even_vetoed: bool = False,
|
||||
) -> int:
|
||||
"""An edit invalidated these rows — the BGG match was made against
|
||||
the uncorrected read. Remove them so resolve re-queries with the
|
||||
fix; `photos` narrows the cull to one copy of a split title.
|
||||
Rows carrying a human veto (dedupe_veto) are never dropped — the
|
||||
match itself was human-vetted, and dropping would erase the veto;
|
||||
a rename updates their title in place so they follow the entry."""
|
||||
a rename updates their title in place so they follow the entry.
|
||||
`even_vetoed` is for title REMOVAL, where the human is explicitly
|
||||
discarding the line, vetted or not."""
|
||||
self.reload_if_changed()
|
||||
norm = normalize_title(title_raw)
|
||||
keep: list[dict] = []
|
||||
@@ -288,7 +294,7 @@ class ReviewSession:
|
||||
)
|
||||
if not targeted:
|
||||
keep.append(row)
|
||||
elif row.get("dedupe_veto"):
|
||||
elif row.get("dedupe_veto") and not even_vetoed:
|
||||
if new_title and row["title_raw"] != new_title:
|
||||
row["title_raw"] = new_title
|
||||
renamed += 1
|
||||
@@ -325,9 +331,17 @@ class ReviewSession:
|
||||
undecided = [
|
||||
row for row in matches if row["match_status"] in UNDECIDED_MATCH_STATUSES
|
||||
]
|
||||
if undecided or matches:
|
||||
return (undecided or matches)[0]
|
||||
return None
|
||||
if undecided:
|
||||
return undecided[0]
|
||||
# both siblings decided (two-edition duplicates): prefer the one
|
||||
# whose VERSION is still open, as _adopt does — otherwise a stale
|
||||
# row_ix would land a version pick on the already-decided sibling
|
||||
version_open = [
|
||||
row for row in matches if row["version_status"] == "version_ambiguous"
|
||||
]
|
||||
if version_open:
|
||||
return version_open[0]
|
||||
return matches[0] if matches else None
|
||||
|
||||
def cues_for(
|
||||
self, title_raw: str, source_photos: str | None = None
|
||||
@@ -353,6 +367,158 @@ class ReviewSession:
|
||||
row["match_status"] = "rejected"
|
||||
self._save(row)
|
||||
|
||||
def decide_local(self, row: dict) -> None:
|
||||
"""A real game BGG simply doesn't have: a local library citizen —
|
||||
listed and enriched from its own photo reads, never uploaded."""
|
||||
row["match_status"] = "local"
|
||||
row["bgg_id"] = ""
|
||||
row["bgg_name"] = ""
|
||||
row["version_status"] = ""
|
||||
row["version_id"] = ""
|
||||
row["version_name"] = ""
|
||||
self._save(row)
|
||||
|
||||
def research(self, row: dict, query: str, types: str | None = None) -> int:
|
||||
"""Re-search on the human's terms and put the results on the row's
|
||||
ballot. `types` targets a specific database — "rpgitem" for
|
||||
RPGGeek, whose entries the automatic cascade never reaches when BGG
|
||||
has a same-named BOARD game (every D&D box hits this)."""
|
||||
query = query.strip()
|
||||
if not query:
|
||||
raise ValueError("a search needs some text")
|
||||
results = (
|
||||
self.client.search(query, types) if types else self.client.search(query)
|
||||
)
|
||||
merged: dict[int, Candidate] = {}
|
||||
for r in results:
|
||||
cand = Candidate(
|
||||
bgg_id=r.bgg_id,
|
||||
name=r.name,
|
||||
year=r.year,
|
||||
type=r.type,
|
||||
exact=normalize_title(r.name) == normalize_title(query),
|
||||
fuzzy=0.0,
|
||||
)
|
||||
prior = merged.get(r.bgg_id)
|
||||
# one ballot line per game; an alternate-name row that exact-
|
||||
# matches the query outranks the primary-name row for it
|
||||
if prior is None or (cand.exact and not prior.exact):
|
||||
merged[r.bgg_id] = cand
|
||||
cands = list(merged.values())[:12]
|
||||
if cands:
|
||||
try:
|
||||
stats = {
|
||||
t.bgg_id: t
|
||||
for t in self.client.things([c.bgg_id for c in cands], stats=True)
|
||||
}
|
||||
except _BGG_ERRORS as err:
|
||||
# owned counts and ranks only decorate the ballot — losing
|
||||
# them must not lose the search the human just asked for
|
||||
self._warn(f"couldn't fetch stats for these results ({err})")
|
||||
stats = {}
|
||||
for c in cands:
|
||||
if c.bgg_id in stats:
|
||||
c.owned = stats[c.bgg_id].owned
|
||||
c.rank = stats[c.bgg_id].rank
|
||||
c.publishers = list(stats[c.bgg_id].publishers)
|
||||
row["match_status"] = "ambiguous" if cands else "unmatched"
|
||||
row["candidates_json"] = json.dumps(
|
||||
[c.as_json() for c in cands], ensure_ascii=False
|
||||
)
|
||||
# a fresh ballot supersedes any earlier verdict on this row —
|
||||
# including the VERSION verdicts, which belong to the old game: a
|
||||
# surviving version_id would ride into the next pick and put the
|
||||
# wrong game's edition on the collection entry
|
||||
row["bgg_id"] = ""
|
||||
row["bgg_name"] = ""
|
||||
row["version_status"] = ""
|
||||
row["version_id"] = ""
|
||||
row["version_name"] = ""
|
||||
row["version_candidates_json"] = "[]"
|
||||
self._save(row)
|
||||
return len(cands)
|
||||
|
||||
def reopen_match(self, row: dict) -> None:
|
||||
"""The human says a matched row is the WRONG game: clear the match
|
||||
and immediately re-search so the card comes back as a ballot — the
|
||||
machine re-OFFERS, only the human re-decides (run_resolve never
|
||||
touches unmatched rows, so without this the row would just sit).
|
||||
If BGG is unreachable the row still reopens, bare: re-search and
|
||||
manual id remain available on the card."""
|
||||
row["match_status"] = "unmatched"
|
||||
row["bgg_id"] = ""
|
||||
row["bgg_name"] = ""
|
||||
row["year"] = ""
|
||||
row["type"] = ""
|
||||
row["candidates_json"] = "[]"
|
||||
row["version_status"] = ""
|
||||
row["version_id"] = ""
|
||||
row["version_name"] = ""
|
||||
row["version_candidates_json"] = "[]"
|
||||
entry = self.cues_for(row["title_raw"], row["source_photos"]) or TitleEntry(
|
||||
title_raw=row["title_raw"],
|
||||
title_normalized=normalize_title(row["title_raw"]),
|
||||
)
|
||||
try:
|
||||
# the FULL cascade resolve uses — board games, truncation heads,
|
||||
# then RPGGeek — so a row parked as "local" gets the same look
|
||||
cands = find_candidates(self.client, entry)[:8]
|
||||
if cands:
|
||||
stats = {
|
||||
t.bgg_id: t
|
||||
for t in self.client.things([c.bgg_id for c in cands], stats=True)
|
||||
}
|
||||
for c in cands:
|
||||
if c.bgg_id in stats:
|
||||
c.owned = stats[c.bgg_id].owned
|
||||
c.rank = stats[c.bgg_id].rank
|
||||
c.publishers = list(stats[c.bgg_id].publishers)
|
||||
row["match_status"] = "ambiguous"
|
||||
row["candidates_json"] = json.dumps(
|
||||
[c.as_json() for c in cands], ensure_ascii=False
|
||||
)
|
||||
except _BGG_ERRORS as err:
|
||||
self._warn(
|
||||
f"re-search unavailable ({err}) — the row is reopened; use "
|
||||
"(f) re-search or (m) manual id on its card"
|
||||
)
|
||||
self._save(row)
|
||||
|
||||
def open_version_ballot(self, row: dict) -> int:
|
||||
"""The human knows which printing a box is even when the photo
|
||||
showed no cues: put EVERY published version on the row's ballot
|
||||
(cue-scored when cues exist) and mark it version_ambiguous so the
|
||||
normal edition pass presents it. Returns the ballot size."""
|
||||
if not row["bgg_id"]:
|
||||
raise ValueError("no BGG match on this row yet")
|
||||
entry = self.cues_for(row["title_raw"], row["source_photos"])
|
||||
things = self.client.things([int(row["bgg_id"])], versions=True)
|
||||
if not things or not things[0].versions:
|
||||
raise ValueError("BGG lists no versions for this game")
|
||||
scored = sorted(
|
||||
((v, _score_version(entry, v) if entry else 0) for v in things[0].versions),
|
||||
key=lambda pair: (-pair[1], str(pair[0].year or "")),
|
||||
)
|
||||
row["version_candidates_json"] = json.dumps(
|
||||
[
|
||||
{
|
||||
"version_id": v.version_id,
|
||||
"name": v.name,
|
||||
"year": v.year,
|
||||
"publishers": list(v.publishers),
|
||||
"languages": list(v.languages),
|
||||
"score": s,
|
||||
}
|
||||
for v, s in scored
|
||||
],
|
||||
ensure_ascii=False,
|
||||
)
|
||||
row["version_status"] = "version_ambiguous"
|
||||
row["version_id"] = ""
|
||||
row["version_name"] = ""
|
||||
self._save(row)
|
||||
return len(scored)
|
||||
|
||||
def decide_version(self, row: dict, version_id: int | None) -> None:
|
||||
"""Pick a version from the row's stored candidates, or None -> unknown."""
|
||||
if version_id is None:
|
||||
@@ -404,10 +570,10 @@ class ReviewSession:
|
||||
while True:
|
||||
self._show_item(row, candidates)
|
||||
prompt = (
|
||||
"[1-N] pick (s)kip (r)eject (m <id>) manual BGG id "
|
||||
"(f <text>) re-search (q)uit > "
|
||||
"[1-N] pick (s)kip (r)eject (l)ocal — not on BGG "
|
||||
"(m <id>) manual BGG id (f <text>) re-search (q)uit > "
|
||||
if row["match_status"] == "unmatched"
|
||||
else "[1-N] pick (s)kip (r)eject (q)uit > "
|
||||
else "[1-N] pick (s)kip (r)eject (l)ocal — not on BGG (q)uit > "
|
||||
)
|
||||
answer = self._ask(prompt)
|
||||
lowered = answer.lower()
|
||||
@@ -416,6 +582,9 @@ class ReviewSession:
|
||||
if lowered == "r":
|
||||
self.decide_reject(row)
|
||||
return
|
||||
if lowered == "l":
|
||||
self.decide_local(row)
|
||||
return
|
||||
if answer.isdigit() and 1 <= int(answer) <= len(candidates):
|
||||
self.decide_pick(row, candidates[int(answer) - 1])
|
||||
return
|
||||
@@ -449,6 +618,9 @@ class ReviewSession:
|
||||
def _research(self, query: str) -> list[dict]:
|
||||
try:
|
||||
results = self.client.search(query)
|
||||
if not results:
|
||||
# board games first, then RPGGeek — same database, same token
|
||||
results = self.client.search(query, "rpgitem")
|
||||
except _BGG_ERRORS as err:
|
||||
self.console.print(f"[yellow]search unavailable: {err}[/yellow]")
|
||||
return []
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
/* bggpipe design system — one stylesheet, two pages (dashboard, review).
|
||||
/* bggpipe design system — one stylesheet for the whole app.
|
||||
*
|
||||
* The visual world is Juniper's mascot drawing: a sky-blue day, flat cel
|
||||
* color inside confident dark outlines, a cream game board with a rainbow
|
||||
@@ -30,7 +30,12 @@
|
||||
--gold: #f2c04b; /* pipe fittings: rings, trim, hovers */
|
||||
--gold-ink: #8a6414;
|
||||
--ticket: #fdf3d2; /* reshoot work-orders */
|
||||
--focus: #8330c2;
|
||||
--go-tint: #e2f2e4; /* pale state washes of the role colors */
|
||||
--stop-tint: #fbe3da;
|
||||
--gold-tint: #fdeebb;
|
||||
--accent-tint: #ece5f7;
|
||||
--navy-tint: #e3ecf3;
|
||||
--focus: var(--accent);
|
||||
--path: linear-gradient(90deg,
|
||||
#f767b8, #f79a3e, #f2c04b, #6fce6f, #5aa7f0, #9a5be0);
|
||||
--path-v: linear-gradient(180deg,
|
||||
@@ -59,7 +64,6 @@ body {
|
||||
}
|
||||
main { max-width: 62rem; margin: 0 auto; padding: 1.4rem 1.2rem 6rem; }
|
||||
h2 {
|
||||
color: var(--ink);
|
||||
font-family: var(--font-display);
|
||||
font-weight: 700; font-size: 1.1rem; letter-spacing: .02em;
|
||||
margin: 2rem 0 .8rem;
|
||||
@@ -121,6 +125,7 @@ nav[aria-label="Primary"] a[aria-current="page"] {
|
||||
display: block; margin: 0 auto;
|
||||
}
|
||||
.piperbox figcaption { font-size: .72rem; opacity: .8; margin-top: .45rem; }
|
||||
.navtoggle { display: none; }
|
||||
.piperbox .legal { font-size: .62rem; opacity: .6; line-height: 1.45; text-align: left; }
|
||||
.skip {
|
||||
position: absolute; left: -999px; top: 0; z-index: 10;
|
||||
@@ -163,7 +168,7 @@ kbd {
|
||||
border: var(--line);
|
||||
background: var(--board);
|
||||
}
|
||||
.banner.error { border-color: var(--stop); color: var(--stop-ink); background: #fbe3da; }
|
||||
.banner.error { border-color: var(--stop); color: var(--stop-ink); background: var(--stop-tint); }
|
||||
.banner.warn { border-color: var(--gold-ink); color: var(--gold-ink); background: var(--ticket); }
|
||||
|
||||
/* -- buttons ----------------------------------------------------------- */
|
||||
@@ -192,6 +197,17 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
display: flex; gap: 1.1rem;
|
||||
scroll-margin-top: 5rem;
|
||||
}
|
||||
.card.prose { display: block; }
|
||||
.card.prose p, .card.prose ol { margin: .55rem 0; line-height: 1.55; }
|
||||
.card.prose > :first-child { margin-top: 0; }
|
||||
.card.prose > :last-child { margin-bottom: 0; }
|
||||
.artcard { text-align: center; }
|
||||
.artcard img {
|
||||
width: 100%; max-width: 15rem;
|
||||
border: var(--line); border-radius: var(--radius-lg);
|
||||
background: var(--sky);
|
||||
}
|
||||
.artcard .legal { font-size: .74rem; color: var(--ink-soft); max-width: 30rem; margin: .6rem auto 0; }
|
||||
.card.active {
|
||||
border-color: var(--accent);
|
||||
box-shadow: var(--shadow-raised);
|
||||
@@ -231,11 +247,11 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
.cname { font-weight: 600; }
|
||||
.cmeta { color: var(--ink-soft); font-size: .8rem; margin-left: .4rem; }
|
||||
.rowactions { margin-top: .7rem; display: flex; gap: 1rem; align-items: center; flex-wrap: wrap; font-size: .85rem; }
|
||||
.rowactions button {
|
||||
font: inherit; border: var(--line); background: #fff;
|
||||
border-radius: var(--radius); padding: .25rem .7rem; cursor: pointer;
|
||||
}
|
||||
.rowactions button { padding: .25rem .7rem; box-shadow: none; }
|
||||
.rowactions button.reject { color: var(--stop-ink); border-color: var(--stop); }
|
||||
.research { display: flex; gap: .35rem; align-items: center; flex-wrap: wrap; }
|
||||
.research input[type=text] { width: 12em; }
|
||||
.research button { font-size: .78rem; }
|
||||
.rowactions input[type=text] {
|
||||
font: inherit; width: 8.5em; padding: .25rem .5rem;
|
||||
border: 2px solid var(--board-edge); border-radius: var(--radius);
|
||||
@@ -264,10 +280,9 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
.ticket .partial { font-family: var(--font-mono); font-size: .85rem; }
|
||||
.ticket .notes { color: var(--gold-ink); font-size: .85rem; margin-top: .2rem; }
|
||||
.ticket button {
|
||||
font: inherit; font-size: .8rem; margin-top: .5rem;
|
||||
font-size: .8rem; margin-top: .5rem;
|
||||
background: none; border: 1px solid var(--gold-ink); color: var(--gold-ink);
|
||||
border-radius: var(--radius); padding: .2rem .6rem; cursor: pointer;
|
||||
box-shadow: none;
|
||||
padding: .2rem .6rem; box-shadow: none;
|
||||
}
|
||||
|
||||
/* -- all-done celebration ---------------------------------------------- */
|
||||
@@ -280,7 +295,7 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
border-radius: var(--radius-lg); border: var(--line);
|
||||
background: var(--sky);
|
||||
}
|
||||
.done h2 { color: var(--ink); margin-top: 0; }
|
||||
.done h2 { margin-top: 0; }
|
||||
.done .nums { display: flex; justify-content: center; gap: 2rem; margin: 1rem 0; }
|
||||
.done .nums div { font-size: 1.6rem; font-weight: 700; font-family: var(--font-display); }
|
||||
.done .nums span { display: block; font-size: .72rem; text-transform: uppercase; letter-spacing: .08em; color: var(--ink-soft); }
|
||||
@@ -310,12 +325,14 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
text-underline-offset: 2px;
|
||||
}
|
||||
.catalog a:hover { color: var(--accent-ink); }
|
||||
.catalog td.actions { text-align: right; white-space: nowrap; }
|
||||
.catalog td.actions button {
|
||||
font: inherit; font-size: .78rem; border: var(--line); background: #fff;
|
||||
border-radius: var(--radius); padding: .2rem .6rem; cursor: pointer;
|
||||
box-shadow: none;
|
||||
.catalog td.actions { text-align: right; }
|
||||
.catalog td.actions button { margin: .1rem 0 .1rem .3rem; white-space: nowrap; }
|
||||
.catalog td.actions .ballotlink {
|
||||
display: inline-block; margin: .1rem 0 .1rem .3rem; white-space: nowrap;
|
||||
font-size: .78rem; color: var(--accent-ink);
|
||||
text-decoration: underline dotted; text-underline-offset: 3px;
|
||||
}
|
||||
.catalog td.actions button { font-size: .78rem; padding: .2rem .6rem; box-shadow: none; }
|
||||
.catalog td.actions button:hover { background: var(--board); }
|
||||
.catalog tr.editrow td { background: #fff; border-top: none; padding: .2rem .5rem .7rem; }
|
||||
.editform { display: flex; gap: .7rem; align-items: end; flex-wrap: wrap; }
|
||||
@@ -336,21 +353,28 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
border-radius: 999px; padding: .1rem .55rem; white-space: nowrap;
|
||||
border: 1px solid currentColor;
|
||||
}
|
||||
.chip.ok { background: #e2f2e4; color: var(--go-ink); }
|
||||
.card code {
|
||||
font-family: var(--font-mono); font-size: .82em;
|
||||
background: #fff; border: 1px solid var(--board-edge);
|
||||
border-radius: 4px; padding: 0 .3em;
|
||||
}
|
||||
.card ol { padding-left: 1.3rem; }
|
||||
.card ol li { margin: .35rem 0; }
|
||||
.chip.ok { background: var(--go-tint); color: var(--go-ink); }
|
||||
.chip.wait { background: var(--ticket); color: var(--gold-ink); }
|
||||
.chip.no { background: #fbe3da; color: var(--stop-ink); }
|
||||
.chip.open { background: #ece5f7; color: var(--accent-ink); }
|
||||
.chip.merged { background: #e3ecf3; color: var(--navy); }
|
||||
.chip.no { background: var(--stop-tint); color: var(--stop-ink); }
|
||||
.chip.open { background: var(--accent-tint); color: var(--accent-ink); }
|
||||
.chip.merged { background: var(--navy-tint); color: var(--navy); }
|
||||
.chip.shaky { background: var(--ticket); color: var(--gold-ink); border: 1px dashed var(--gold-ink); }
|
||||
|
||||
/* -- merge notices: slim, undoable ------------------------------------- */
|
||||
.card.merge { padding: .55rem 1rem; align-items: center; }
|
||||
.card.merge .body { display: flex; align-items: center; gap: .8rem; flex-wrap: wrap; }
|
||||
.card.merge .arrow { color: var(--ink-soft); }
|
||||
.card.merge button {
|
||||
font: inherit; font-size: .8rem; margin-left: auto;
|
||||
font-size: .8rem; margin-left: auto;
|
||||
background: none; border: 1px solid var(--board-edge);
|
||||
border-radius: var(--radius); padding: .2rem .6rem; cursor: pointer;
|
||||
box-shadow: none;
|
||||
padding: .2rem .6rem; box-shadow: none;
|
||||
}
|
||||
.card.merge button:hover { border-color: var(--stop); color: var(--stop-ink); }
|
||||
|
||||
@@ -385,7 +409,9 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
font: inherit; cursor: pointer;
|
||||
box-shadow: none;
|
||||
}
|
||||
#dropzone.hot, #dropzone:hover { border-style: solid; background: #fdeebb; }
|
||||
#dropzone.hot, #dropzone:hover { border-style: solid; background: var(--gold-tint); }
|
||||
#dropzone[aria-busy="true"] { cursor: progress; opacity: .8; }
|
||||
#dropzone.ok { border-style: solid; border-color: var(--go-ink); color: var(--go-ink); background: var(--go-tint); }
|
||||
#joblog {
|
||||
background: var(--navy); color: #eaf1fb;
|
||||
font-family: var(--font-mono); font-size: .78rem;
|
||||
@@ -394,13 +420,11 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
max-height: 20rem; overflow-y: auto; white-space: pre-wrap;
|
||||
}
|
||||
#jobstate { font-size: .85rem; color: var(--ink); margin-bottom: .4rem; }
|
||||
#jobstate .running { color: var(--accent-ink); font-weight: 600; }
|
||||
#jobstate .failed { color: var(--stop-ink); font-weight: 600; }
|
||||
#jobstate .s-running { color: var(--accent-ink); font-weight: 600; }
|
||||
#jobstate .s-done { color: var(--go-ink); font-weight: 600; }
|
||||
#jobstate .s-failed { color: var(--stop-ink); font-weight: 600; }
|
||||
|
||||
@media (max-width: 700px) {
|
||||
.card, .ticket { flex-direction: column; }
|
||||
.shots { flex-basis: auto; }
|
||||
}
|
||||
/* (phone-width rules live in the responsive section at the end) */
|
||||
|
||||
/* -- photos page: gallery + status ------------------------------------- */
|
||||
.gallery { display: grid; grid-template-columns: repeat(auto-fill, minmax(11rem, 1fr)); gap: .9rem; }
|
||||
@@ -439,6 +463,41 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
.game .info { padding: .55rem .7rem .7rem; }
|
||||
.game .gname { font-weight: 700; font-family: var(--font-display); }
|
||||
.game .gmeta { font-size: .78rem; color: var(--ink-soft); margin-top: .2rem; line-height: 1.5; }
|
||||
/* -- library: card grid links + one game's detail ---------------------- */
|
||||
a.game { text-decoration: none; color: inherit; }
|
||||
a.game:hover { border-color: var(--accent); box-shadow: var(--shadow-raised); }
|
||||
a.game:focus-visible { outline: 3px solid var(--focus); outline-offset: 2px; }
|
||||
.gamedetail { display: flex; gap: 1.4rem; align-items: flex-start; flex-wrap: wrap; }
|
||||
.gameartcol { flex: 0 0 clamp(12rem, 28vw, 18rem); }
|
||||
.gameart {
|
||||
width: 100%; border: var(--line); border-radius: var(--radius-lg);
|
||||
box-shadow: var(--shadow-card); display: block; background: var(--board);
|
||||
}
|
||||
.gameart.noart {
|
||||
aspect-ratio: 1; display: flex; align-items: center; justify-content: center;
|
||||
background: var(--sky-deep); color: var(--navy);
|
||||
font-family: var(--font-display); font-size: 3rem; font-weight: 700;
|
||||
}
|
||||
.gamefacts {
|
||||
flex: 1 1 18rem; background: var(--board); border: var(--line);
|
||||
border-radius: var(--radius-lg); box-shadow: var(--shadow-card);
|
||||
padding: .9rem 1.1rem;
|
||||
}
|
||||
.factrow { margin: .45rem 0; line-height: 1.5; }
|
||||
.factrow b {
|
||||
font-size: .72rem; text-transform: uppercase; letter-spacing: .06em;
|
||||
color: var(--ink-soft); margin-right: .4rem;
|
||||
}
|
||||
.chiplist { display: inline-flex; flex-wrap: wrap; gap: .3rem; vertical-align: middle; }
|
||||
.gdesc { white-space: pre-wrap; line-height: 1.6; }
|
||||
.artbtn { margin-top: .6rem; width: 100%; font-size: .8rem; }
|
||||
.editform label.wide { flex: 1 1 100%; }
|
||||
.editform textarea {
|
||||
font: inherit; font-size: .85rem; color: var(--ink); width: 100%;
|
||||
border: 2px solid var(--board-edge); border-radius: var(--radius);
|
||||
padding: .35rem .45rem; background: #fff; resize: vertical;
|
||||
}
|
||||
|
||||
.empty {
|
||||
background: var(--board); border: 2px dashed var(--board-edge);
|
||||
border-radius: var(--radius-lg); padding: 2rem; text-align: center;
|
||||
@@ -457,18 +516,108 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
|
||||
border: var(--line); border-radius: var(--radius); background: #fff;
|
||||
}
|
||||
|
||||
/* -- responsive: rail collapses to a top strip -------------------------- */
|
||||
/* -- toast: viewport-pinned transient confirmations --------------------- */
|
||||
#toast {
|
||||
position: fixed; bottom: 1.1rem; left: 50%; transform: translateX(-50%);
|
||||
z-index: 20; max-width: min(92vw, 34rem);
|
||||
background: var(--navy); color: #fff;
|
||||
border: var(--line); border-radius: var(--radius-lg);
|
||||
box-shadow: var(--shadow-raised);
|
||||
padding: .7rem 1rem; font-size: .9rem; line-height: 1.45;
|
||||
}
|
||||
#toast a { color: var(--gold); }
|
||||
#toast button {
|
||||
font-size: .8rem; padding: .25rem .7rem; margin-left: .5rem;
|
||||
box-shadow: none;
|
||||
}
|
||||
|
||||
/* -- responsive -------------------------------------------------------- */
|
||||
|
||||
/* The rail collapses to a top bar: brand + hamburger; the nav drops
|
||||
* down as a stacked menu (a scrolling strip lost its position on every
|
||||
* page load and hid the far entries). */
|
||||
@media (max-width: 900px) {
|
||||
body { display: block; }
|
||||
.sidebar {
|
||||
position: sticky; height: auto; flex-direction: row; align-items: center;
|
||||
flex-wrap: wrap; gap: 0 .5rem; z-index: 5;
|
||||
position: sticky; top: 0; height: auto; z-index: 5;
|
||||
display: flex; flex-direction: row; align-items: center; flex-wrap: wrap;
|
||||
border-right: none; border-bottom: 4px solid transparent;
|
||||
border-image: var(--path) 1;
|
||||
}
|
||||
.brand { padding: .5rem .8rem; }
|
||||
nav[aria-label="Primary"] { flex-direction: row; flex-wrap: wrap; }
|
||||
nav[aria-label="Primary"] a { border-left: none; border-bottom: 3px solid transparent; padding: .35rem .6rem; }
|
||||
nav[aria-label="Primary"] a[aria-current="page"] { border-bottom-color: var(--gold); }
|
||||
.brand { padding: .5rem .8rem; flex: 1; }
|
||||
.brand img { width: 30px; height: 30px; }
|
||||
.wordmark { font-size: 1.1rem; }
|
||||
.wordmark small { display: inline; margin-left: .5rem; font-size: .68rem; }
|
||||
.navtoggle {
|
||||
display: inline-flex; align-items: center; justify-content: center;
|
||||
width: 44px; height: 40px; margin-right: .6rem;
|
||||
background: none; border: 2px solid rgba(255, 255, 255, .75);
|
||||
border-radius: var(--radius); box-shadow: none; padding: 0;
|
||||
}
|
||||
.navtoggle .bars {
|
||||
display: block; position: relative;
|
||||
width: 18px; height: 2px; background: #fff; border-radius: 2px;
|
||||
}
|
||||
.navtoggle .bars::before, .navtoggle .bars::after {
|
||||
content: ""; position: absolute; left: 0;
|
||||
width: 18px; height: 2px; background: #fff; border-radius: 2px;
|
||||
}
|
||||
.navtoggle .bars::before { top: -6px; }
|
||||
.navtoggle .bars::after { top: 6px; }
|
||||
body.navopen .navtoggle { background: rgba(255, 255, 255, .15); }
|
||||
nav[aria-label="Primary"] {
|
||||
display: none; flex-basis: 100%; flex-direction: column;
|
||||
padding: 0 0 .4rem;
|
||||
}
|
||||
body.navopen nav[aria-label="Primary"] { display: flex; }
|
||||
nav[aria-label="Primary"] a { padding: .6rem 1rem; }
|
||||
.piperbox { display: none; }
|
||||
}
|
||||
|
||||
/* Phone-width layout: stacked cards, stacked tables, touch targets. */
|
||||
@media (max-width: 700px) {
|
||||
main { padding: 1rem .9rem 4rem; }
|
||||
h1 { font-size: 1.3rem; margin: .9rem 0 .7rem; }
|
||||
.keyhelp { display: none; } /* no keyboard on a phone */
|
||||
button, .linkbtn { padding: .45rem .9rem; } /* finger-sized */
|
||||
/* component buttons set their own padding at higher specificity —
|
||||
* they need the touch bump spelled out */
|
||||
.rowactions button, .ticket button, .card.merge button { padding: .4rem .8rem; }
|
||||
|
||||
.card, .ticket { flex-direction: column; }
|
||||
.shots { flex-basis: auto; }
|
||||
.ticket .stencil {
|
||||
writing-mode: horizontal-tb; letter-spacing: .35em;
|
||||
border-right: none; border-bottom: 1px solid var(--gold-ink);
|
||||
padding: 0 0 .3rem;
|
||||
}
|
||||
.done { padding: 1.1rem .9rem; }
|
||||
.done .nums { flex-wrap: wrap; gap: .8rem 1.6rem; }
|
||||
|
||||
.filterbar input[type=search] { min-width: 0; flex: 1 1 12rem; }
|
||||
.editform { flex-direction: column; align-items: stretch; }
|
||||
.editform input { width: 100%; }
|
||||
.editactions { flex-wrap: wrap; }
|
||||
.editactions button { flex: 1 1 auto; }
|
||||
|
||||
/* the titles/photo tables become stacked line-cards: one bordered
|
||||
* block per game, cells flowing top to bottom, empty cells gone */
|
||||
.catalog table, .catalog tbody, .catalog tr, .catalog td { display: block; }
|
||||
.catalog tr { padding: .6rem 0; border-top: 1px solid var(--board-edge); }
|
||||
.catalog tr:first-child { border-top: none; }
|
||||
.catalog tr:hover td { background: none; }
|
||||
.catalog td { border-top: none; padding: .12rem 0; }
|
||||
.catalog td:empty { display: none; }
|
||||
.catalog .t { max-width: none; font-size: .95rem; }
|
||||
.catalog td.actions {
|
||||
text-align: left; white-space: normal;
|
||||
display: flex; gap: .5rem; flex-wrap: wrap; padding-top: .4rem;
|
||||
}
|
||||
.catalog td.actions button { padding: .4rem .8rem; }
|
||||
.catalog tr.editrow { padding-top: 0; border-top: none; }
|
||||
|
||||
/* queue/library ledgers keep their columns but scroll inside their
|
||||
* own card; long photo lists wrap within their cell */
|
||||
.ledger { overflow-x: auto; }
|
||||
.ledger td { overflow-wrap: anywhere; min-width: 6rem; }
|
||||
}
|
||||
|
||||
@@ -10,9 +10,25 @@ const esc = s => String(s ?? "").replace(/[&<>"']/g,
|
||||
* pass through esc() at the call site. The API serves only local pipeline
|
||||
* data, but photo names and BGG titles still count as untrusted. */
|
||||
function showBanner(html) {
|
||||
const el = document.getElementById("banner");
|
||||
document.getElementById("banner").innerHTML = html;
|
||||
}
|
||||
|
||||
/* Transient confirmation pinned to the viewport: the #banner region sits
|
||||
* at the top of the page, which a user deep in a long list never sees.
|
||||
* The toast may carry buttons — wire them before the timer clears it. */
|
||||
function showToast(html, ms = 8000) {
|
||||
let el = document.getElementById("toast");
|
||||
if (!el) {
|
||||
el = document.createElement("div");
|
||||
el.id = "toast";
|
||||
el.setAttribute("role", "status");
|
||||
document.body.appendChild(el);
|
||||
}
|
||||
el.innerHTML = html;
|
||||
el.hidden = false;
|
||||
clearTimeout(showToast._timer);
|
||||
showToast._timer = setTimeout(() => { el.hidden = true; }, ms);
|
||||
return el;
|
||||
}
|
||||
|
||||
function errorBanner(detail) {
|
||||
@@ -49,6 +65,21 @@ async function apiPost(url, body) {
|
||||
return res;
|
||||
}
|
||||
|
||||
/* Per-page change gate for poll loops: render only when the payload
|
||||
* changed, and record it only after render returns — a throw leaves the
|
||||
* page stale, so the next poll retries instead of freezing on "current". */
|
||||
function changeGate() {
|
||||
let seen = null;
|
||||
const gate = (value, render) => {
|
||||
const key = JSON.stringify(value);
|
||||
if (key === seen) return;
|
||||
render();
|
||||
seen = key;
|
||||
};
|
||||
gate.reset = () => { seen = null; };
|
||||
return gate;
|
||||
}
|
||||
|
||||
/* Poll `fn` every `ms`; after 3 consecutive failures show the lost-contact
|
||||
* banner, and clear it (via `recovered`) on the next success. */
|
||||
function pollLoop(fn, ms, recovered) {
|
||||
@@ -66,6 +97,16 @@ function pollLoop(fn, ms, recovered) {
|
||||
}, ms);
|
||||
}
|
||||
|
||||
/* Mobile: the top bar's hamburger opens the nav; navigation reloads the
|
||||
* page, so each page starts with the menu closed. */
|
||||
{
|
||||
const toggle = document.querySelector(".navtoggle");
|
||||
if (toggle) toggle.addEventListener("click", () => {
|
||||
const open = document.body.classList.toggle("navopen");
|
||||
toggle.setAttribute("aria-expanded", String(open));
|
||||
});
|
||||
}
|
||||
|
||||
/* Sidebar badges: the counts that mean "something wants your attention". */
|
||||
async function refreshBadges() {
|
||||
const p = await fetchJSON("/api/pipeline");
|
||||
@@ -74,6 +115,7 @@ async function refreshBadges() {
|
||||
if (el) el.textContent = n > 0 ? String(n) : "";
|
||||
};
|
||||
set("photos", p.reshoot);
|
||||
set("titles", p.shaky_reads);
|
||||
set("review", p.pending_review);
|
||||
set("queue", p.to_add + p.to_update);
|
||||
return p;
|
||||
@@ -81,11 +123,70 @@ async function refreshBadges() {
|
||||
refreshBadges().catch(() => {});
|
||||
setInterval(() => refreshBadges().catch(() => {}), 5000);
|
||||
|
||||
/* One-line match summary for a titles/photo table row. Empty when the
|
||||
* row has no BGG data yet, so the mobile stacker can hide the cell. */
|
||||
/* Canonical outbound URL for a matched thing. RPG items live on RPGGeek
|
||||
* (same database, different site) — a /boardgame/ URL is the wrong home
|
||||
* for them. */
|
||||
function bggUrl(id, type) {
|
||||
return type === "rpgitem"
|
||||
? `https://rpggeek.com/rpgitem/${encodeURIComponent(id)}`
|
||||
: `https://boardgamegeek.com/boardgame/${encodeURIComponent(id)}`;
|
||||
}
|
||||
|
||||
function metaLine(c) {
|
||||
const name = !c.bgg_name ? "" : c.bgg_id
|
||||
? `<a href="${bggUrl(c.bgg_id, c.type)}" target="_blank" rel="noopener"
|
||||
title="open on ${c.type === "rpgitem" ? "RPGGeek" : "BGG"}">${esc(c.bgg_name)} ↗</a>`
|
||||
: esc(c.bgg_name);
|
||||
return [
|
||||
name,
|
||||
c.version_name ? esc(c.version_name) : "",
|
||||
c.type === "rpgitem" ? `<span class="chip open">RPG · local only</span>` : "",
|
||||
].filter(Boolean).join(" · ");
|
||||
}
|
||||
|
||||
/* Reshoot work-order card, shared by the photos gallery and the photo
|
||||
* detail page (which drops the thumbnail and source line — the photo is
|
||||
* right there). Dismiss buttons are wired by wireDismiss below. */
|
||||
function ticketCard(s, {showPhoto = true} = {}) {
|
||||
const img = showPhoto && s.photo_exists
|
||||
? `<a href="/photos/${encodeURIComponent(s.photo)}" target="_blank" tabindex="-1">
|
||||
<img src="/photos/${encodeURIComponent(s.photo)}" alt="photo ${esc(s.photo)}"></a>`
|
||||
: "";
|
||||
return `
|
||||
<section class="ticket"
|
||||
data-photo="${esc(s.photo)}" data-location="${esc(s.location)}"
|
||||
data-partial="${esc(s.partial_text)}" data-art="${esc(s.art_notes)}">
|
||||
<span class="stencil">reshoot</span>
|
||||
${img}
|
||||
<div>
|
||||
<div class="loc">${esc(s.location) || "somewhere in " + esc(s.photo)}</div>
|
||||
${s.partial_text ? `<div class="partial">text visible: ${esc(s.partial_text)}</div>` : ""}
|
||||
${s.art_notes ? `<div class="notes">${esc(s.art_notes)}</div>` : ""}
|
||||
${showPhoto ? `<div class="notes">from ${esc(s.photo)} — take a closer shot and drop it above</div>` : ""}
|
||||
<button class="dismiss">dismiss — found it / not a game</button>
|
||||
</div>
|
||||
</section>`;
|
||||
}
|
||||
|
||||
function wireDismiss(root, done) {
|
||||
root.querySelectorAll(".dismiss").forEach(b => b.addEventListener("click", async () => {
|
||||
const t = b.closest(".ticket");
|
||||
const res = await apiPost("/api/dismiss", {
|
||||
photo: t.dataset.photo, location: t.dataset.location,
|
||||
partial_text: t.dataset.partial, art_notes: t.dataset.art,
|
||||
});
|
||||
if (res) done();
|
||||
}));
|
||||
}
|
||||
|
||||
/* Status chip for a catalog entry — shared by the catalog and photo pages. */
|
||||
function statusChip(c) {
|
||||
if (c.status === "awaiting_resolve") return `<span class="chip wait">awaiting BGG</span>`;
|
||||
if (c.status === "awaiting_resolve") return `<span class="chip wait">awaiting resolve</span>`;
|
||||
if (c.status === "auto" || c.status === "approved") return `<span class="chip ok">${esc(c.status)}</span>`;
|
||||
if (c.status === "rejected") return `<span class="chip no">rejected</span>`;
|
||||
if (c.status === "local") return `<span class="chip open">local — not on BGG</span>`;
|
||||
if (c.status === "merged") return `<span class="chip merged" title="merged into ${esc(c.merged_into)}">merged → ${esc(c.merged_into)}</span>`;
|
||||
return `<span class="chip open">${esc(c.status)}</span>`;
|
||||
}
|
||||
|
||||
|
After Width: | Height: | Size: 39 KiB |
@@ -1,125 +0,0 @@
|
||||
<h1>Catalog</h1>
|
||||
<div class="pagebar"><span id="catcount"></span></div>
|
||||
<div class="filterbar">
|
||||
<input type="search" id="catsearch" placeholder="filter titles…" aria-label="filter catalog titles">
|
||||
</div>
|
||||
<div id="catbody"><p class="empty">Nothing extracted yet — start on the <a href="/photos">photos page</a>.</p></div>
|
||||
<script>
|
||||
"use strict";
|
||||
let CATALOG = [];
|
||||
let EDITING = null; // lineKey of the row whose editor is open
|
||||
|
||||
function lineKey(c) { return c.title_raw + "|" + c.photos.join(";"); }
|
||||
|
||||
function editorRow(c) {
|
||||
const cue = c.cues || {};
|
||||
return `
|
||||
<tr class="editrow"><td colspan="5">
|
||||
<form class="editform" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}">
|
||||
<label>Title <input name="title" value="${esc(c.title_raw)}" required></label>
|
||||
<label>Publisher <input name="publisher" value="${esc(cue.publisher || "")}"></label>
|
||||
<label>Edition <input name="edition" value="${esc(cue.edition || "")}"></label>
|
||||
<label>Year <input name="year" value="${esc(cue.year ?? "")}" inputmode="numeric" size="6"></label>
|
||||
<label>Language <input name="language" value="${esc(cue.language || "")}"></label>
|
||||
<span class="editactions">
|
||||
<button type="submit" class="primary">save</button>
|
||||
<button type="button" class="canceledit">cancel</button>
|
||||
</span>
|
||||
<span class="edithint">saving re-queues this title for resolve with the corrected data</span>
|
||||
</form>
|
||||
</td></tr>`;
|
||||
}
|
||||
|
||||
function render() {
|
||||
const q = document.getElementById("catsearch").value.trim().toLowerCase();
|
||||
const sorted = [...CATALOG].sort((a, b) =>
|
||||
a.title_raw.localeCompare(b.title_raw, undefined, { sensitivity: "base" }));
|
||||
const rows = q
|
||||
? sorted.filter(c => (c.title_raw + " " + c.bgg_name).toLowerCase().includes(q))
|
||||
: sorted;
|
||||
document.getElementById("catcount").innerHTML =
|
||||
`<b>${rows.length}</b> of <b>${CATALOG.length}</b> title(s)`;
|
||||
document.getElementById("catbody").innerHTML = rows.length
|
||||
? `<div class="catalog"><table>` + rows.map(c => `
|
||||
<tr>
|
||||
<td class="t">${esc(c.title_raw)}
|
||||
${c.split_copy ? `<span class="chip merged">copy</span>` : ""}</td>
|
||||
<td>${statusChip(c)}</td>
|
||||
<td class="meta">${c.bgg_name ? esc(c.bgg_name) + (c.bgg_id ? " · " + esc(c.bgg_id) : "") : ""}
|
||||
${c.version_name ? " · " + esc(c.version_name) : ""}
|
||||
${c.type === "rpgitem" ? ` <span class="chip open">RPG · local only</span>` : ""}</td>
|
||||
<td class="meta">${c.photos.map(p =>
|
||||
`<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>`
|
||||
).join(", ")}</td>
|
||||
<td class="actions">${c.can_split
|
||||
? `<button class="split" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}"
|
||||
data-rowix="${c.row_ix ?? ""}"
|
||||
title="one line, several boxes? make each photo its own copy">
|
||||
split into copies</button>`
|
||||
: ""}
|
||||
<button class="edit" data-key="${esc(lineKey(c))}"
|
||||
title="fix a misread title or add cues you already know">edit</button>
|
||||
</td>
|
||||
</tr>` + (EDITING === lineKey(c) ? editorRow(c) : "")).join("") + `</table></div>`
|
||||
: `<p class="empty">${CATALOG.length
|
||||
? "No titles match that filter."
|
||||
: `Nothing extracted yet — start on the <a href="/photos">photos page</a>.`}</p>`;
|
||||
}
|
||||
|
||||
let LAST = null;
|
||||
async function refresh() {
|
||||
const state = await fetchJSON("/api/state");
|
||||
if (EDITING) return; // never repaint under an open editor
|
||||
const payload = JSON.stringify(state.catalog);
|
||||
if (payload === LAST) return;
|
||||
LAST = payload;
|
||||
CATALOG = state.catalog;
|
||||
render();
|
||||
}
|
||||
|
||||
document.getElementById("catbody").addEventListener("click", async e => {
|
||||
const cancel = e.target.closest("button.canceledit");
|
||||
if (cancel) { EDITING = null; LAST = null; render(); refresh().catch(() => {}); return; }
|
||||
const edit = e.target.closest("button.edit");
|
||||
if (edit) {
|
||||
EDITING = EDITING === edit.dataset.key ? null : edit.dataset.key;
|
||||
render();
|
||||
return;
|
||||
}
|
||||
const b = e.target.closest("button.split");
|
||||
if (!b) return;
|
||||
const n = b.dataset.photos.split(";").length;
|
||||
if (!confirm(`Split "${b.dataset.title}" into ${n} separate copies (one per photo)? ` +
|
||||
`Each picks its own edition afterward.`)) return;
|
||||
const res = await apiPost("/api/split", {
|
||||
title_raw: b.dataset.title,
|
||||
source_photos: b.dataset.photos,
|
||||
row_ix: b.dataset.rowix === "" ? null : Number(b.dataset.rowix),
|
||||
});
|
||||
if (res) refresh().catch(() => {});
|
||||
});
|
||||
|
||||
document.getElementById("catbody").addEventListener("submit", async e => {
|
||||
const f = e.target.closest("form.editform");
|
||||
if (!f) return;
|
||||
e.preventDefault();
|
||||
const orig = CATALOG.find(c => lineKey(c) === EDITING) || {};
|
||||
const cue = orig.cues || {};
|
||||
const v = name => f.elements[name].value;
|
||||
const body = { title_raw: f.dataset.title, source_photos: f.dataset.photos };
|
||||
if (v("title").trim() !== f.dataset.title) body.title_new = v("title").trim();
|
||||
if (v("publisher") !== (cue.publisher || "")) body.publisher = v("publisher");
|
||||
if (v("edition") !== (cue.edition || "")) body.edition = v("edition");
|
||||
if (v("year") !== String(cue.year ?? "")) body.year = v("year");
|
||||
if (v("language") !== (cue.language || "")) body.language = v("language");
|
||||
if (Object.keys(body).length <= 2) { EDITING = null; render(); return; }
|
||||
const res = await apiPost("/api/edit-title", body);
|
||||
if (res) { EDITING = null; LAST = null; refresh().catch(() => {}); }
|
||||
});
|
||||
|
||||
document.getElementById("catsearch").addEventListener("input", render);
|
||||
refresh().catch(err => errorBanner(err.message || err));
|
||||
pollLoop(refresh, 5000, () => showBanner(""));
|
||||
</script>
|
||||
@@ -0,0 +1,80 @@
|
||||
<h1>Help</h1>
|
||||
<div class="pagebar">
|
||||
<a href="#flow">The flow</a>
|
||||
<a href="#pages">Pages</a>
|
||||
<a href="#curation">Fixing the titles</a>
|
||||
<a href="#statuses">Statuses</a>
|
||||
<a href="#keys">Keyboard</a>
|
||||
<a href="#files">Your data</a>
|
||||
</div>
|
||||
|
||||
<h2 id="flow">The flow: shelves → collection</h2>
|
||||
<div class="card prose">
|
||||
<p>Six pipeline stages and two checkpoints that are yours. Stages run from the <a href="/">Pipeline</a> page (or the CLI — both share all state and either can pick up where the other left off):</p>
|
||||
<ol>
|
||||
<li><b>extract</b> — every photo goes to the vision model once (Claude by default; a local model works too — see the README); what it reads (titles plus edition cues: publisher, edition wording, year, language) lands on the Titles page. Boxes it can see but can't read become <b>reshoot tickets</b> on the Photos page.</li>
|
||||
<li><b>proofread</b> <i>(you, on <a href="/titles">Titles</a>)</i> — fix misread titles, add cues you know, split multi-box lines, remove non-games, <a href="#curation">details below</a>. Worth doing <i>before</i> resolve: a fix made now is one BGG search done right; a fix made later sends the title back through resolve again.</li>
|
||||
<li><b>resolve</b> — titles are matched to BoardGameGeek games and editions. Anything uncertain is flagged, never guessed. Without a BGG API token, titles wait as <i>awaiting BGG</i> and are picked up automatically once the token exists.</li>
|
||||
<li><b>review</b> <i>(you, on <a href="/review">Review</a>)</i> — decide the flagged ones: which game, which edition, whether two reads are one box. Every decision saves immediately.</li>
|
||||
<li><b>diff</b> — your existing BGG collection is fetched and compared, per copy. What's genuinely new lands in the <a href="/queue">Queue</a>.</li>
|
||||
<li><b>upload</b> — a real browser logs into BGG and adds each queued game, slowly and politely. Always dry-run first; the buttons enforce that order.</li>
|
||||
<li><b>enrich</b> — full metadata (players, weight, rank, artwork) fills the <a href="/library">Library</a>.</li>
|
||||
</ol>
|
||||
<p>It's a loop, not a line: new photos, edits, and splits feed both checkpoints again, and the sidebar badges show when a page has work for you. Every stage is resumable — stop anything mid-run and nothing is lost; re-runs skip work already done.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="pages">What each page is for</h2>
|
||||
<div class="card prose">
|
||||
<p><b><a href="/">Pipeline</a></b> — run stages one at a time and watch their live output. Shows what's blocking (missing keys, stub data) and the counts at every step.</p>
|
||||
<p><b><a href="/photos">Photos</a></b> — drag photos in, drop them in the <code>photos/</code> folder, or (on a paired phone) tap the drop zone and shoot straight from the camera; camera captures get unique <code>shelf-…</code> names so they can never overwrite each other. Each photo has its own page listing every title read from it and any reshoot tickets — boxes seen but not identified. Photograph those up close, drop the new shot in, and extract again. Re-uploading a photo under the same <i>file name</i> deliberately replaces it, and the next extract run re-reads it.</p>
|
||||
<p><b><a href="/titles">Titles</a></b> — every read off your shelves, alphabetized, with its status and photos. This is the proofread checkpoint: <a href="#curation">edit, split, remove</a>. Its badge counts <span class="chip shaky">shaky read</span> lines — the model wasn't sure and nothing has verified them; filter to them, then press <b>✓ looks right</b> or edit each one.</p>
|
||||
<p><b><a href="/review">Review</a></b> — the decisions only you can make: which game a title is, which edition a copy is, whether two same-game reads are really one box (merges show a veto), and whether an unmatched title is a real game BGG simply doesn't have (<b>keep locally</b>: it joins the Library, never uploads). Keyboard-first; see <a href="#keys">shortcuts</a>.</p>
|
||||
<p><b><a href="/queue">Queue</a></b> — exactly what upload will do (new entries and version upgrades) and the log of everything it has done. Nothing reaches BGG that isn't visible here first. A job that fails is skipped by later runs (so one broken game can't loop forever); when any exist, the Pipeline's upload card offers a <b>retry N failed</b> checkbox. Each queued row shows what upload did with it — <span class="chip open">pending</span>, <span class="chip ok">done</span>, <span class="chip no">failed</span>, or <span class="chip no">retired</span> (a review decision since the last diff withdrew it). Finished rows stay listed until the next <b>diff</b> rebuilds the queue; the log below them is the permanent record.</p>
|
||||
<p><b><a href="/library">Library</a></b> — your enriched collection. Search titles, designers, mechanics and categories at once; filter by kind (board games, RPGs, off-BGG) or by how many people are playing tonight; sort by name, year, BGG rank, weight, or playing time. Click any game for its full detail: art, the usual stats, designers and mechanics, <b>your</b> edition, the shelf photos it was read from, and a link to its BGG page. RPG and off-BGG games live here too — identified and enriched, never uploaded. RPGs pull their designers, publishers and genres from RPGGeek; an off-BGG game's detail page lets you write its facts yourself and add a cover photo, since nothing else will ever have them (both are saved under <code>data/</code> and folded in by the next <b>enrich</b>).</p>
|
||||
</div>
|
||||
|
||||
<h2 id="curation">Fixing the titles: edit, split, remove</h2>
|
||||
<div class="card prose">
|
||||
<p>Vision reads aren't perfect, and you know things the photos don't show. Every line on the Titles page has curation actions, and every one of them is <b>durable</b>: the decision is saved in a small committed file and replayed on every rebuild, so re-running extract or resolve can never undo it.</p>
|
||||
<p><b>edit</b> — fix a misread title or add cues you already know (publisher, edition, year, language). A corrected misspelling automatically merges with a correctly-read sighting of the same game from another photo. If the line already had a BGG match, saving re-queues it so resolve searches again with the corrected data.</p>
|
||||
<p><b>split into copies</b> — one line, several physical boxes? Splitting makes each photo its own copy, and each copy picks its own edition afterward. Appears on any line whose title was seen in more than one photo. Splitting one game never affects a same-named different edition.</p>
|
||||
<p><b>pick edition</b> — a matched game with no legible edition cues stays version-less by design (never guess) — but you know which printing your box is. This fetches the game's complete version list into a Review ballot; pick yours there.</p>
|
||||
<p><b>wrong match</b> (inside the edit panel) — an auto-match landed on the wrong game (same-name impostors happen). This clears the match, re-searches immediately, and returns the title to Review as a fresh ballot of candidates (including same-named sibling editions); manual-id entry is there too for games BGG's search can't find. Note BGG sometimes splits one game's lineage across entries — Wiz-War's early editions and its FFG remake are separate games — so a copy whose edition isn't on the ballot may belong to the sibling entry.</p>
|
||||
<p><b>add a game</b> (top of the Titles page) — a game no photo shows: an expansion stored inside a base box, a game away from the shelves. It joins the list like any read (BGG wants base game and expansion as separate collection entries, so boxes that hold both need this for the hidden half), and if a later photo shows it, the sighting merges instead of duplicating.</p>
|
||||
<p><b>remove</b> (inside the edit panel) — for lines that shouldn't exist at all: a book read as a game, box art misread as a title, or a <i>duplicate read</i> — the same physical copy read differently from two photos, leaving two lines for one box (remove the worse read; the survivor keeps its own photos). This is different from <i>reject</i> on the Review page, which keeps the line visible as "no BGG match" — right for real games BGG doesn't know. And the mirror case — ONE line that's really several physical copies — wants <b>split</b>, not remove.</p>
|
||||
<p>Undo: each decision is one record in <code>data/title_edits.json</code>, <code>data/title_splits.json</code>, or <code>data/title_removals.json</code> — delete the record and the next rebuild restores the old state.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="statuses">What the status chips mean</h2>
|
||||
<div class="card prose">
|
||||
<p><span class="chip wait">awaiting resolve</span> extracted (or re-queued by an edit) but not yet matched — run <b>resolve</b> from the Pipeline page.</p>
|
||||
<p><span class="chip ok">auto</span> matched confidently, no review needed. <span class="chip ok">approved</span> you picked the match yourself.</p>
|
||||
<p><span class="chip open">ambiguous</span> several plausible games — needs your pick on Review. <span class="chip open">unmatched</span> nothing plausible found — enter a BGG id or re-search on Review.</p>
|
||||
<p><span class="chip merged">merged</span> two reads judged to be the same physical box; the merge is veto-able on Review. <span class="chip merged">copy</span> one copy of a title you split.</p>
|
||||
<p><span class="chip open">local — not on BGG</span> you ruled it's a real game BGG doesn't have: it joins the Library from its own photo reads (fill in its facts and add a cover photo on its Library page), and never uploads. Changed your mind — or suspect it's on RPGGeek after all? <b>look it up</b> on its Titles row searches again, RPGGeek included.</p>
|
||||
<p><span class="chip no">rejected</span> you ruled it's a bad read or not worth matching; it stays listed but goes no further.</p>
|
||||
<p><span class="chip shaky">shaky read</span> the vision model wasn't sure of this transcription and nothing has verified it yet — these are what the Titles badge counts. Clear one by pressing its <b>✓ looks right</b> (the read is fine as-is) or by editing it (you fixed it). A BGG match also clears it: a wrong read wouldn't have matched.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="keys">Keyboard shortcuts</h2>
|
||||
<div class="card prose">
|
||||
<p><b>Review:</b> <kbd>j</kbd>/<kbd>k</kbd> move between cards · <kbd>1</kbd>–<kbd>9</kbd> pick a candidate · <kbd>r</kbd> reject · <kbd>l</kbd> keep local (not on BGG) · <kbd>m</kbd> manual BGG id · <kbd>u</kbd> edition unknown · <kbd>v</kbd> veto a merge.</p>
|
||||
<p><b>Photo pages:</b> <kbd>←</kbd>/<kbd>→</kbd> move between photos.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="files">Your data, on disk</h2>
|
||||
<div class="card prose">
|
||||
<p>Everything lives in flat files under <code>data/</code> — inspectable, hand-editable, and git-friendly. The pipeline artifacts: <code>titles.json</code> (what was read), <code>matches.csv</code> (what it matched), <code>to_add.csv</code>/<code>to_update.csv</code> (what upload will do), <code>upload_log.csv</code> (what it did), <code>games.json</code> (the library). Your curation: <code>title_edits.json</code>, <code>title_splits.json</code>, <code>title_removals.json</code>, <code>unidentified_dismissed.json</code>.</p>
|
||||
<p>Credentials never live in files — only environment variables, set up by <code>bggpipe init</code>. The app serves localhost only, unless started with <code>--lan</code> — that opens it to your network behind an access key: scan the QR code the server prints (or open the printed link) once per device, and a year-long cookie keeps it paired. Delete <code>data/.lan_key</code> to revoke every device. No login beyond the key; trusted networks only.</p>
|
||||
<p>More depth: the README covers setup and photo technique; <code>docs/bgg-upload-flow.md</code> documents the upload automation.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="mascot">The piper</h2>
|
||||
<div class="card prose artcard">
|
||||
<img src="/static/logo-full.jpg"
|
||||
alt="the bggpipe piper — a bagpiper whose bag is a board game box">
|
||||
<p>Mascot art by Juniper, used with pride.</p>
|
||||
<p class="legal">BoardGameGeek and BGG are trademarks of BoardGameGeek, LLC.
|
||||
bggpipe is an independent project, not affiliated with or endorsed by
|
||||
BoardGameGeek.</p>
|
||||
</div>
|
||||
@@ -1,10 +1,25 @@
|
||||
<h1>Library</h1>
|
||||
<div class="pagebar"><span id="libcount"></span></div>
|
||||
<div class="filterbar" role="group" aria-label="filter by kind">
|
||||
<input type="search" id="libsearch" placeholder="search your games…" aria-label="search library">
|
||||
<input type="search" id="libsearch" placeholder="search titles, designers, mechanics…"
|
||||
aria-label="search library">
|
||||
<button type="button" data-kind="" aria-pressed="true">All</button>
|
||||
<button type="button" data-kind="boardgame" aria-pressed="false">Board games</button>
|
||||
<button type="button" data-kind="rpgitem" aria-pressed="false">RPGs</button>
|
||||
<button type="button" data-kind="localgame" aria-pressed="false">Off BGG</button>
|
||||
</div>
|
||||
<div class="filterbar">
|
||||
<label>plays with
|
||||
<input type="number" id="libplayers" min="1" max="20" placeholder="any"
|
||||
style="width:5em" aria-label="filter by player count"></label>
|
||||
<label>sort
|
||||
<select id="libsort" aria-label="sort library">
|
||||
<option value="name">name</option>
|
||||
<option value="year">year (newest)</option>
|
||||
<option value="rank">BGG rank</option>
|
||||
<option value="weight">weight (lightest)</option>
|
||||
<option value="playtime">playing time</option>
|
||||
</select></label>
|
||||
</div>
|
||||
<div id="libbody"></div>
|
||||
<script>
|
||||
@@ -14,39 +29,72 @@ let KIND = ""; // "" = all; "boardgame" also covers expansions
|
||||
|
||||
function gameCard(g) {
|
||||
const art = g.thumbnail || g.image;
|
||||
const players = g.min_players
|
||||
? (g.min_players === g.max_players ? `${g.min_players}` : `${g.min_players}–${g.max_players}`) + " players"
|
||||
const lo = g.min_players ?? g.max_players, hi = g.max_players ?? g.min_players;
|
||||
const players = lo
|
||||
? (lo === hi ? `${lo}` : `${lo}–${hi}`) + " players"
|
||||
: "";
|
||||
const time = g.playtime ? `${g.playtime} min` : "";
|
||||
const weight = g.weight ? `weight ${g.weight.toFixed(1)}` : "";
|
||||
const rank = g.rank ? `BGG rank ${g.rank}` : "";
|
||||
return `
|
||||
<article class="game">
|
||||
<a class="game" href="/library/game/${encodeURIComponent(g.key)}">
|
||||
${art
|
||||
? `<img src="${esc(art)}" alt="" loading="lazy">`
|
||||
: `<div class="noart" aria-hidden="true">${esc((g.name || "?").charAt(0).toUpperCase())}</div>`}
|
||||
<div class="info">
|
||||
<div class="gname">${esc(g.name)}${g.year ? ` <span class="meta">(${esc(g.year)})</span>` : ""}
|
||||
${g.type === "rpgitem" ? `<span class="chip open">RPG · local only</span>` : ""}</div>
|
||||
${g.type === "rpgitem" ? `<span class="chip open">RPG · local only</span>` : ""}
|
||||
${g.type === "localgame" ? `<span class="chip open">not on BGG · local</span>` : ""}</div>
|
||||
<div class="gmeta">${[players, time, weight, rank].filter(Boolean).map(esc).join(" · ")}</div>
|
||||
${g.version ? `<div class="gmeta">${esc(g.version.name || "")}</div>` : ""}
|
||||
</div>
|
||||
</article>`;
|
||||
</a>`;
|
||||
}
|
||||
|
||||
function matchesKind(g) {
|
||||
if (!KIND) return true;
|
||||
if (KIND === "boardgame") return g.type !== "rpgitem";
|
||||
return g.type === "rpgitem";
|
||||
if (KIND === "boardgame") return !["rpgitem", "localgame"].includes(g.type);
|
||||
return g.type === KIND;
|
||||
}
|
||||
|
||||
// null sorts last whichever way the column runs: an unranked game is not
|
||||
// "rank 0", and a game with no weight is not the lightest
|
||||
function by(field, dir = 1) {
|
||||
return (a, b) => {
|
||||
const x = a[field], y = b[field];
|
||||
if (x == null && y == null) return 0;
|
||||
if (x == null) return 1;
|
||||
if (y == null) return -1;
|
||||
return (x - y) * dir;
|
||||
};
|
||||
}
|
||||
|
||||
const SORTS = {
|
||||
name: (a, b) => (a.name || "").localeCompare(b.name || "", undefined, {sensitivity: "base"}),
|
||||
year: by("year", -1),
|
||||
rank: by("rank"),
|
||||
weight: by("weight"),
|
||||
playtime: by("playtime"),
|
||||
};
|
||||
|
||||
function render() {
|
||||
const q = document.getElementById("libsearch").value.trim().toLowerCase();
|
||||
const seats = Number(document.getElementById("libplayers").value) || 0;
|
||||
let rows = GAMES.filter(matchesKind);
|
||||
if (q) rows = rows.filter(g =>
|
||||
(g.name + " " + (g.designers || []).join(" ")).toLowerCase().includes(q));
|
||||
if (q) rows = rows.filter(g => [
|
||||
g.name,
|
||||
(g.designers || []).join(" "),
|
||||
(g.mechanics || []).join(" "),
|
||||
(g.categories || []).join(" "),
|
||||
g.version && g.version.name,
|
||||
].filter(Boolean).join(" ").toLowerCase().includes(q));
|
||||
if (seats) rows = rows.filter(g =>
|
||||
(g.min_players || g.max_players || 0) <= seats
|
||||
&& seats <= (g.max_players || g.min_players || 0));
|
||||
rows = [...rows].sort(SORTS[document.getElementById("libsort").value] || SORTS.name);
|
||||
document.getElementById("libcount").innerHTML =
|
||||
`<b>${rows.length}</b> of <b>${GAMES.length}</b> game(s)`;
|
||||
`<b>${rows.length}</b> of <b>${GAMES.length}</b> game(s)`
|
||||
+ (seats ? ` playable with <b>${seats}</b>` : "");
|
||||
document.getElementById("libbody").innerHTML = rows.length
|
||||
? `<div class="shelfgrid">${rows.map(gameCard).join("")}</div>`
|
||||
: `<p class="empty">${GAMES.length
|
||||
@@ -56,17 +104,15 @@ function render() {
|
||||
Run the pipeline through <code>enrich</code> to fill these shelves.`}</p>`;
|
||||
}
|
||||
|
||||
let LAST = null;
|
||||
const GATE = changeGate();
|
||||
async function refresh() {
|
||||
const games = await fetchJSON("/api/library");
|
||||
const payload = JSON.stringify(games);
|
||||
if (payload === LAST) return;
|
||||
LAST = payload;
|
||||
GAMES = games;
|
||||
render();
|
||||
GATE(games, () => { GAMES = games; render(); });
|
||||
}
|
||||
|
||||
document.getElementById("libsearch").addEventListener("input", render);
|
||||
document.getElementById("libplayers").addEventListener("input", render);
|
||||
document.getElementById("libsort").addEventListener("change", render);
|
||||
document.querySelectorAll("[data-kind]").forEach(b => b.addEventListener("click", () => {
|
||||
KIND = b.dataset.kind;
|
||||
document.querySelectorAll("[data-kind]").forEach(o =>
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
<h1 id="gname">Loading…</h1>
|
||||
<div class="pagebar">
|
||||
<a href="/library">← library</a>
|
||||
<span id="gcrumb"></span>
|
||||
</div>
|
||||
<div id="gbody"><p class="empty">Loading…</p></div>
|
||||
<script>
|
||||
"use strict";
|
||||
const KEY = decodeURIComponent(location.pathname.replace(/^\/library\/game\//, ""));
|
||||
|
||||
function chips(label, values) {
|
||||
if (!values || !values.length) return "";
|
||||
return `<div class="factrow"><b>${esc(label)}</b>
|
||||
<span class="chiplist">${values.map(v =>
|
||||
`<span class="cue">${esc(v)}</span>`).join("")}</span></div>`;
|
||||
}
|
||||
|
||||
function fact(label, value) {
|
||||
return value === null || value === undefined || value === ""
|
||||
? ""
|
||||
: `<div class="factrow"><b>${esc(label)}</b> ${esc(value)}</div>`;
|
||||
}
|
||||
|
||||
function players(g) {
|
||||
const lo = g.min_players, hi = g.max_players;
|
||||
if (!lo && !hi) return "";
|
||||
const range = lo === hi ? `${lo}` : `${lo ?? "?"}–${hi ?? "?"}`;
|
||||
const best = (g.best_player_counts || []).length
|
||||
? ` (best at ${esc(g.best_player_counts.join(", "))})`
|
||||
: "";
|
||||
return `${range} players${best}`;
|
||||
}
|
||||
|
||||
function playtime(g) {
|
||||
if (!g.playtime && !g.min_playtime) return "";
|
||||
if (g.min_playtime && g.max_playtime && g.min_playtime !== g.max_playtime)
|
||||
return `${g.min_playtime}–${g.max_playtime} min`;
|
||||
return `${g.playtime || g.min_playtime} min`;
|
||||
}
|
||||
|
||||
function localForm(g) {
|
||||
const v = (x) => (x === null || x === undefined ? "" : x);
|
||||
return `
|
||||
<h2>Your notes</h2>
|
||||
<div class="card prose">
|
||||
<p class="meta">BGG has no entry for this game, so what you type here is
|
||||
all it will ever know. Saved to <code>data/local_games.json</code>.</p>
|
||||
<form class="editform" id="localform">
|
||||
<label>Title <input name="name" value="${esc(v(g.name))}" required></label>
|
||||
<label>Year <input name="year" value="${esc(v(g.year))}" inputmode="numeric" size="6"></label>
|
||||
<label>Players from <input name="min_players" value="${esc(v(g.min_players))}" size="3"></label>
|
||||
<label>to <input name="max_players" value="${esc(v(g.max_players))}" size="3"></label>
|
||||
<label>Minutes <input name="playtime" value="${esc(v(g.playtime))}" size="5"></label>
|
||||
<label>Publishers <input name="publishers" value="${esc((g.publishers || []).join(", "))}"></label>
|
||||
<label>Designers <input name="designers" value="${esc((g.designers || []).join(", "))}"></label>
|
||||
<label class="wide">Notes
|
||||
<textarea name="description" rows="4">${esc(v(g.description))}</textarea></label>
|
||||
<span class="editactions"><button type="submit" class="primary">save</button></span>
|
||||
</form>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function wireLocal(g) {
|
||||
const form = document.getElementById("localform");
|
||||
if (form) form.addEventListener("submit", async e => {
|
||||
e.preventDefault();
|
||||
const f = Object.fromEntries(new FormData(form).entries());
|
||||
const res = await apiPost(`/api/local-game/${encodeURIComponent(KEY)}`, f);
|
||||
if (res) {
|
||||
showToast("saved — run <b>enrich</b> to fold this into the library");
|
||||
refresh();
|
||||
}
|
||||
});
|
||||
const btn = document.getElementById("artbtn");
|
||||
const file = document.getElementById("artfile");
|
||||
if (btn) btn.addEventListener("click", () => file.click());
|
||||
if (file) file.addEventListener("change", async () => {
|
||||
if (!file.files.length) return;
|
||||
const body = new FormData();
|
||||
body.append("file", file.files[0]);
|
||||
btn.disabled = true;
|
||||
btn.textContent = "uploading…";
|
||||
let res = null;
|
||||
try {
|
||||
res = await fetch(`/api/local-art/${encodeURIComponent(KEY)}`, {method: "POST", body});
|
||||
} catch (err) {
|
||||
alert("Upload failed: " + err);
|
||||
}
|
||||
btn.disabled = false;
|
||||
if (res && res.ok) {
|
||||
showToast("photo saved — run <b>enrich</b> to fold it into the library");
|
||||
refresh();
|
||||
} else if (res) {
|
||||
const detail = await res.json().then(d => d.detail).catch(() => null);
|
||||
alert("Upload failed: " + (detail ?? res.statusText));
|
||||
refresh();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function render(g) {
|
||||
document.getElementById("gname").textContent = g.name || "(unnamed)";
|
||||
document.title = `${g.name} · bggpipe`;
|
||||
const local = g.type === "localgame";
|
||||
const rpg = g.type === "rpgitem";
|
||||
document.getElementById("gcrumb").innerHTML = [
|
||||
g.year ? `<b>${esc(g.year)}</b>` : "",
|
||||
local ? `<span class="chip open">not on BGG · local</span>` : "",
|
||||
rpg ? `<span class="chip open">RPG · local only</span>` : "",
|
||||
g.bgg_id
|
||||
? `<a href="${bggUrl(g.bgg_id, g.type)}" target="_blank" rel="noopener">
|
||||
view on ${rpg ? "RPGGeek" : "BGG"} ↗</a>`
|
||||
: "",
|
||||
].filter(Boolean).join(" · ");
|
||||
|
||||
const art = g.image
|
||||
? `<img class="gameart" src="${esc(g.image)}" alt="box art for ${esc(g.name)}">`
|
||||
: `<div class="gameart noart">${esc((g.name || "?")[0])}</div>`;
|
||||
// an off-BGG game has no publisher art and no API to fetch any: the
|
||||
// owner's own photo is the only cover it will ever have
|
||||
const artAdd = local
|
||||
? `<button id="artbtn" class="artbtn">${g.image ? "replace" : "add"} a photo</button>
|
||||
<input id="artfile" type="file" accept=".jpg,.jpeg,.png,.heic" hidden
|
||||
aria-label="cover photo for ${esc(g.name)}">`
|
||||
: "";
|
||||
|
||||
const facts = [
|
||||
fact("players", players(g)),
|
||||
fact("playing time", playtime(g)),
|
||||
fact("ages", g.min_age ? `${g.min_age}+` : ""),
|
||||
fact("weight", g.weight ? `${g.weight.toFixed(2)} / 5` : ""),
|
||||
fact("BGG rank", g.rank || ""),
|
||||
fact("BGG rating", g.rating ? g.rating.toFixed(2) : ""),
|
||||
fact("owned by", g.users_owned ? `${g.users_owned.toLocaleString()} people` : ""),
|
||||
chips("designers", g.designers),
|
||||
chips("artists", (g.artists || []).slice(0, 8)),
|
||||
chips("publishers", (g.publishers || []).slice(0, 6)),
|
||||
chips("categories", g.categories),
|
||||
chips("mechanics", g.mechanics),
|
||||
].filter(Boolean).join("");
|
||||
|
||||
const version = g.version
|
||||
? `<h2>Your edition</h2>
|
||||
<div class="card prose">
|
||||
<div class="factrow"><b>${esc(g.version.name || "—")}</b>
|
||||
${g.version.year ? ` · ${esc(g.version.year)}` : ""}</div>
|
||||
${chips("publishers", g.version.publishers)}
|
||||
${chips("languages", g.version.languages)}
|
||||
</div>`
|
||||
: `<h2>Your edition</h2>
|
||||
<div class="card prose"><p class="meta">No specific edition recorded —
|
||||
set one from the <a href="/titles">Titles</a> page if you know it.</p></div>`;
|
||||
|
||||
const photos = (g.photos || []).length
|
||||
? `<h2>Seen on your shelves</h2>
|
||||
<div class="card prose"><p>${g.photos.map(p =>
|
||||
`<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>`
|
||||
).join(" · ")}</p></div>`
|
||||
: "";
|
||||
|
||||
document.getElementById("gbody").innerHTML = `
|
||||
<div class="gamedetail">
|
||||
<div class="gameartcol">${art}${artAdd}</div>
|
||||
<div class="gamefacts">${facts}</div>
|
||||
</div>
|
||||
${local ? localForm(g) : ""}
|
||||
${version}
|
||||
${photos}
|
||||
${g.description && !local ? `<h2>About</h2>
|
||||
<div class="card prose"><p class="gdesc">${esc(g.description)}</p></div>` : ""}`;
|
||||
wireLocal(g);
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
try {
|
||||
render(await fetchJSON(`/api/library/${encodeURIComponent(KEY)}`));
|
||||
} catch (err) {
|
||||
document.getElementById("gbody").innerHTML =
|
||||
`<p class="empty">This game isn't in the library
|
||||
(${esc(err.message || err)}) — it may need <b>enrich</b> to run.
|
||||
Back to the <a href="/library">library</a>.</p>`;
|
||||
}
|
||||
}
|
||||
refresh();
|
||||
</script>
|
||||
@@ -13,22 +13,7 @@
|
||||
const NAME = decodeURIComponent(location.pathname.split("/").pop());
|
||||
document.getElementById("photoname").textContent = NAME;
|
||||
document.getElementById("rawlink").href = `/photos/${encodeURIComponent(NAME)}`;
|
||||
document.title = `bggpipe — ${NAME}`;
|
||||
|
||||
function ticket(s) {
|
||||
return `
|
||||
<section class="ticket"
|
||||
data-photo="${esc(s.photo)}" data-location="${esc(s.location)}"
|
||||
data-partial="${esc(s.partial_text)}" data-art="${esc(s.art_notes)}">
|
||||
<span class="stencil">reshoot</span>
|
||||
<div>
|
||||
<div class="loc">${esc(s.location) || "somewhere in this photo"}</div>
|
||||
${s.partial_text ? `<div class="partial">text visible: ${esc(s.partial_text)}</div>` : ""}
|
||||
${s.art_notes ? `<div class="notes">${esc(s.art_notes)}</div>` : ""}
|
||||
<button class="dismiss">dismiss — found it / not a game</button>
|
||||
</div>
|
||||
</section>`;
|
||||
}
|
||||
document.title = `${NAME} · bggpipe`;
|
||||
|
||||
function render(state, photos) {
|
||||
const info = photos.find(p => p.name === NAME);
|
||||
@@ -64,8 +49,7 @@ function render(state, photos) {
|
||||
<tr>
|
||||
<td class="t">${esc(c.title_raw)}</td>
|
||||
<td>${statusChip(c)}</td>
|
||||
<td class="meta">${c.bgg_name ? esc(c.bgg_name) + (c.bgg_id ? " · " + esc(c.bgg_id) : "") : ""}
|
||||
${c.version_name ? " · " + esc(c.version_name) : ""}</td>
|
||||
<td class="meta">${metaLine(c)}</td>
|
||||
</tr>`).join("") + `</table></div>`
|
||||
: `<p class="empty">${info.extracted
|
||||
? "No titles were read from this photo."
|
||||
@@ -73,30 +57,19 @@ function render(state, photos) {
|
||||
|
||||
if (tickets.length) {
|
||||
html += `<h2>Reshoot tickets <span class="count">— boxes seen here but not identified</span></h2>`;
|
||||
html += tickets.map(ticket).join("");
|
||||
html += tickets.map(s => ticketCard(s, {showPhoto: false})).join("");
|
||||
}
|
||||
body.innerHTML = html;
|
||||
|
||||
body.querySelectorAll(".dismiss").forEach(b => b.addEventListener("click", async () => {
|
||||
const t = b.closest(".ticket");
|
||||
const res = await apiPost("/api/dismiss", {
|
||||
photo: t.dataset.photo, location: t.dataset.location,
|
||||
partial_text: t.dataset.partial, art_notes: t.dataset.art,
|
||||
});
|
||||
if (res) refresh().catch(() => {});
|
||||
}));
|
||||
wireDismiss(body, () => refresh().catch(() => {}));
|
||||
}
|
||||
|
||||
let LAST = null;
|
||||
const GATE = changeGate();
|
||||
async function refresh() {
|
||||
const [state, photos] = await Promise.all([
|
||||
fetchJSON("/api/state"),
|
||||
fetchJSON("/api/photos-list"),
|
||||
]);
|
||||
const payload = JSON.stringify([state.catalog, state.unidentified, photos]);
|
||||
if (payload === LAST) return;
|
||||
LAST = payload;
|
||||
render(state, photos);
|
||||
GATE([state.catalog, state.unidentified, photos], () => render(state, photos));
|
||||
}
|
||||
|
||||
document.addEventListener("keydown", e => {
|
||||
|
||||
@@ -11,40 +11,12 @@
|
||||
<script>
|
||||
"use strict";
|
||||
|
||||
function ticket(s) {
|
||||
const img = s.photo_exists
|
||||
? `<a href="/photos/${encodeURIComponent(s.photo)}" target="_blank" tabindex="-1">
|
||||
<img src="/photos/${encodeURIComponent(s.photo)}" alt="photo ${esc(s.photo)}"></a>`
|
||||
: "";
|
||||
return `
|
||||
<section class="ticket"
|
||||
data-photo="${esc(s.photo)}" data-location="${esc(s.location)}"
|
||||
data-partial="${esc(s.partial_text)}" data-art="${esc(s.art_notes)}">
|
||||
<span class="stencil">reshoot</span>
|
||||
${img}
|
||||
<div>
|
||||
<div class="loc">${esc(s.location) || "somewhere in " + esc(s.photo)}</div>
|
||||
${s.partial_text ? `<div class="partial">text visible: ${esc(s.partial_text)}</div>` : ""}
|
||||
${s.art_notes ? `<div class="notes">${esc(s.art_notes)}</div>` : ""}
|
||||
<div class="notes">from ${esc(s.photo)} — take a closer shot and drop it above</div>
|
||||
<button class="dismiss">dismiss — found it / not a game</button>
|
||||
</div>
|
||||
</section>`;
|
||||
}
|
||||
|
||||
function render(state, photos) {
|
||||
const tickets = document.getElementById("tickets");
|
||||
tickets.innerHTML = state.unidentified.length
|
||||
? state.unidentified.map(ticket).join("")
|
||||
? state.unidentified.map(s => ticketCard(s)).join("")
|
||||
: `<p class="empty">No open reshoot tickets.</p>`;
|
||||
tickets.querySelectorAll(".dismiss").forEach(b => b.addEventListener("click", async () => {
|
||||
const t = b.closest(".ticket");
|
||||
const res = await apiPost("/api/dismiss", {
|
||||
photo: t.dataset.photo, location: t.dataset.location,
|
||||
partial_text: t.dataset.partial, art_notes: t.dataset.art,
|
||||
});
|
||||
if (res) refresh();
|
||||
}));
|
||||
wireDismiss(tickets, () => refresh().catch(() => {}));
|
||||
|
||||
document.getElementById("gallerycount").textContent = `— ${photos.length} on file`;
|
||||
document.getElementById("shots").innerHTML = photos.map(p => `
|
||||
@@ -59,17 +31,13 @@ function render(state, photos) {
|
||||
</figure>`).join("");
|
||||
}
|
||||
|
||||
let LAST = null;
|
||||
const GATE = changeGate(); // rebuilding innerHTML re-renders every <img>
|
||||
async function refresh() {
|
||||
const [state, photos] = await Promise.all([
|
||||
fetchJSON("/api/state"),
|
||||
fetchJSON("/api/photos-list"),
|
||||
]);
|
||||
// re-render only on change: rebuilding innerHTML re-renders every <img>
|
||||
const payload = JSON.stringify([state.unidentified, state.warnings, photos]);
|
||||
if (payload === LAST) return;
|
||||
LAST = payload;
|
||||
render(state, photos);
|
||||
GATE([state.unidentified, state.warnings, photos], () => render(state, photos));
|
||||
}
|
||||
|
||||
const zone = document.getElementById("dropzone");
|
||||
@@ -83,25 +51,54 @@ zone.addEventListener("drop", e => {
|
||||
});
|
||||
pick.addEventListener("change", () => sendPhotos(pick.files));
|
||||
|
||||
const ZONE_IDLE = "drop shelf photos here, or click to choose";
|
||||
let UPLOADING = false;
|
||||
|
||||
async function sendPhotos(files) {
|
||||
if (!files.length) return;
|
||||
if (!files.length || UPLOADING) return;
|
||||
const form = new FormData();
|
||||
for (const f of files) form.append("files", f);
|
||||
let res;
|
||||
const mb = ([...files].reduce((total, f) => total + f.size, 0) / 1048576).toFixed(1);
|
||||
UPLOADING = true;
|
||||
zone.disabled = true;
|
||||
zone.setAttribute("aria-busy", "true");
|
||||
zone.classList.remove("ok");
|
||||
zone.textContent = `uploading ${files.length} photo(s) — ${mb} MB… keep this page open`;
|
||||
let res = null;
|
||||
try {
|
||||
res = await fetch("/api/photos", {method: "POST", body: form});
|
||||
} catch (err) {
|
||||
alert("Upload failed (no response from the server): " + err);
|
||||
return;
|
||||
}
|
||||
UPLOADING = false;
|
||||
zone.disabled = false;
|
||||
zone.removeAttribute("aria-busy");
|
||||
pick.value = ""; // re-picking the same photo must fire change again
|
||||
if (!res) { zone.textContent = ZONE_IDLE; return; }
|
||||
if (!res.ok) {
|
||||
const detail = await res.json().then(d => d.detail).catch(() => null);
|
||||
zone.textContent = ZONE_IDLE;
|
||||
alert("Upload failed: " + (detail ?? res.statusText));
|
||||
return;
|
||||
}
|
||||
const saved = await res.json().then(d => d.saved || []).catch(() => []);
|
||||
zone.classList.add("ok");
|
||||
zone.textContent = `✓ saved ${saved.join(", ")} — add another?`;
|
||||
setTimeout(() => {
|
||||
if (!UPLOADING && zone.classList.contains("ok")) {
|
||||
zone.classList.remove("ok");
|
||||
zone.textContent = ZONE_IDLE;
|
||||
}
|
||||
}, 6000);
|
||||
GATE.reset(); // the new photo must appear even if nothing else changed
|
||||
refresh().catch(() => {}); // the next poll self-heals a refresh hiccup
|
||||
}
|
||||
|
||||
// leaving mid-upload silently loses the photo — make the browser ask
|
||||
window.addEventListener("beforeunload", e => {
|
||||
if (UPLOADING) e.preventDefault();
|
||||
});
|
||||
|
||||
refresh().catch(err => errorBanner(err.message || err));
|
||||
pollLoop(refresh, 5000, () => showBanner(""));
|
||||
</script>
|
||||
|
||||
@@ -3,11 +3,11 @@
|
||||
|
||||
<h2 id="activity">Activity</h2>
|
||||
<div id="jobstate" aria-live="polite">idle</div>
|
||||
<div id="joblog" aria-label="stage output">(stage output appears here)</div>
|
||||
<div id="joblog" role="log" aria-label="stage output">(stage output appears here)</div>
|
||||
<script>
|
||||
"use strict";
|
||||
let P = null;
|
||||
let running = false;
|
||||
let RUNNING = false;
|
||||
|
||||
async function runStage(stage, body) {
|
||||
const res = await apiPost(`/api/run/${stage}`, body);
|
||||
@@ -23,12 +23,12 @@ function stageCard(num, name, facts, actions) {
|
||||
}
|
||||
|
||||
function runBtn(stage, label) {
|
||||
return `<button class="primary" data-run="${stage}" ${running ? "disabled" : ""}>${esc(label ?? "Run")}</button>`;
|
||||
return `<button class="primary" data-run="${stage}" ${RUNNING ? "disabled" : ""}>${esc(label ?? "Run")}</button>`;
|
||||
}
|
||||
|
||||
function render() {
|
||||
const m = P.matches, log = P.upload_log;
|
||||
running = P.job.status === "running"; // buttons below depend on this
|
||||
RUNNING = P.job.status === "running"; // buttons below depend on this
|
||||
|
||||
const banners = [];
|
||||
if (P.stub_data) banners.push(
|
||||
@@ -42,35 +42,56 @@ function render() {
|
||||
|
||||
const uploadFacts = `<b>${P.to_add}</b> to add · <b>${P.to_update}</b> version updates
|
||||
${log.added || log.added_no_version ? `· <b>${(log.added ?? 0) + (log.added_no_version ?? 0)}</b> added` : ""}
|
||||
${log.failed ? `· <b>${log.failed}</b> failed` : ""}
|
||||
· <a href="/queue">inspect the queue</a>`;
|
||||
${P.upload_failed ? `· <b>${P.upload_failed}</b> failed` : ""}
|
||||
· <a href="/queue">inspect the queue</a>
|
||||
${!P.to_add && !P.to_update && P.queued_total
|
||||
? `<br><span class="meta">${P.queued_total} queued row(s) already applied —
|
||||
the next <b>diff</b> clears them (BGG's collection export can lag the
|
||||
site by a while)</span>`
|
||||
: ""}`;
|
||||
|
||||
document.getElementById("stages").innerHTML = [
|
||||
stageCard(1, "extract", `read titles off <b>${P.photos}</b> <a href="/photos">photo(s)</a> — <b>${P.titles}</b> so far`, runBtn("extract")),
|
||||
stageCard(1, "extract", `read titles off <b>${P.photos}</b> <a href="/photos">photo(s)</a> — <b>${P.titles}</b> so far`
|
||||
+ (P.titles > 0
|
||||
? P.shaky_reads > 0
|
||||
? ` · <b>${P.shaky_reads}</b> shaky read(s) worth a <a href="/titles">proofread</a> before resolving`
|
||||
: ` · <a href="/titles">proofread the titles</a> before resolving`
|
||||
: ""), runBtn("extract")),
|
||||
stageCard(2, "resolve", `match titles to BGG ids — <b>${(m.auto ?? 0)}</b> auto · <b>${m.ambiguous ?? 0}</b> ambiguous · <b>${m.unmatched ?? 0}</b> unmatched`, runBtn("resolve")),
|
||||
stageCard(3, "review", `<b>${P.pending_review}</b> item(s) waiting for your call`, `<a class="linkbtn" href="/review">Open review</a>`),
|
||||
stageCard(4, "diff", `compare against your BGG collection`, runBtn("diff")),
|
||||
stageCard(5, "upload", uploadFacts,
|
||||
`${runBtn("upload", "Dry run")}
|
||||
<button class="danger" data-upload-real ${running || P.stub_data ? "disabled" : ""}>Upload</button>
|
||||
<label>limit <input type="number" id="uplimit" min="1" placeholder="all"></label>`),
|
||||
<button class="danger" data-upload-real ${RUNNING || P.stub_data ? "disabled" : ""}>Upload</button>
|
||||
<label>limit <input type="number" id="uplimit" min="1" placeholder="all"></label>
|
||||
${P.upload_failed
|
||||
? `<label title="failed jobs are skipped on normal runs so a broken one can't loop">
|
||||
<input type="checkbox" id="upretry"> retry ${P.upload_failed} failed</label>`
|
||||
: ""}`),
|
||||
stageCard(6, "enrich", `<b>${P.games}</b> game(s) in the <a href="/library">library</a>`, runBtn("enrich")),
|
||||
].join("");
|
||||
|
||||
document.querySelectorAll("[data-run]").forEach(b =>
|
||||
b.addEventListener("click", () => runStage(b.dataset.run,
|
||||
b.dataset.run === "upload" ? {dry_run: true} : {})));
|
||||
b.dataset.run === "upload" ? {dry_run: true, ...uploadOpts()} : {})));
|
||||
const uploadOpts = () => ({
|
||||
limit: Number(document.getElementById("uplimit").value) || null,
|
||||
retry_failed: !!document.getElementById("upretry")?.checked,
|
||||
});
|
||||
const real = document.querySelector("[data-upload-real]");
|
||||
if (real) real.addEventListener("click", () => {
|
||||
const limit = Number(document.getElementById("uplimit").value) || null;
|
||||
if (!confirm(`Really add games to your live BGG collection${limit ? ` (limit ${limit})` : ""}? A browser window will open.`)) return;
|
||||
runStage("upload", {dry_run: false, limit});
|
||||
const {limit, retry_failed} = uploadOpts();
|
||||
if (!confirm(`Really add games to your live BGG collection${limit ? ` (limit ${limit})` : ""}` +
|
||||
`${retry_failed ? ", retrying previously failed jobs" : ""}? A browser window will open.`)) return;
|
||||
runStage("upload", {dry_run: false, limit, retry_failed});
|
||||
});
|
||||
|
||||
const job = P.job;
|
||||
const state = document.getElementById("jobstate");
|
||||
if (job.status === "idle") state.textContent = "idle";
|
||||
else state.innerHTML = `<span class="${esc(job.status)}">${esc(job.stage)}: ${esc(job.status)}</span>` +
|
||||
// s- prefix: a bare status class collides with page classes (.done is
|
||||
// the review celebration card)
|
||||
else state.innerHTML = `<span class="s-${esc(job.status)}">${esc(job.stage)}: ${esc(job.status)}</span>` +
|
||||
(job.error ? ` — ${esc(job.error)}` : "");
|
||||
const logEl = document.getElementById("joblog");
|
||||
const atBottom = logEl.scrollTop + logEl.clientHeight >= logEl.scrollHeight - 4;
|
||||
@@ -78,14 +99,10 @@ function render() {
|
||||
if (atBottom) logEl.scrollTop = logEl.scrollHeight;
|
||||
}
|
||||
|
||||
let LAST = null;
|
||||
const GATE = changeGate();
|
||||
async function refresh() {
|
||||
const p = await fetchJSON("/api/pipeline");
|
||||
const payload = JSON.stringify(p);
|
||||
if (payload === LAST) return;
|
||||
LAST = payload;
|
||||
P = p;
|
||||
render();
|
||||
GATE(p, () => { P = p; render(); });
|
||||
}
|
||||
|
||||
refresh().catch(err => errorBanner(err.message || err));
|
||||
|
||||
@@ -11,24 +11,50 @@ function table(headers, rows) {
|
||||
${rows.join("")}</table></div>`;
|
||||
}
|
||||
|
||||
// a queue row's state comes from the upload log: the CSVs are diff-time
|
||||
// snapshots and never shrink as work completes
|
||||
function stateChip(r) {
|
||||
if (r.stale) return `<span class="chip no" title="${esc(r.stale)}">retired</span>`;
|
||||
if (r.state === "done" && r.last_status === "added_no_version")
|
||||
return `<span class="chip ok" title="the add saved without an edition — set it on BGG by hand if it matters">done · no version</span>`;
|
||||
if (r.state === "done") return `<span class="chip ok">done</span>`;
|
||||
if (r.state === "failed") return `<span class="chip no">failed</span>`;
|
||||
return `<span class="chip open">pending</span>`;
|
||||
}
|
||||
|
||||
function tally(rows) {
|
||||
const n = s => rows.filter(r => !r.stale && r.state === s).length;
|
||||
const parts = [`<b>${n("")}</b> pending`];
|
||||
if (n("done")) parts.push(`<b>${n("done")}</b> done`);
|
||||
if (n("failed")) parts.push(`<b>${n("failed")}</b> failed`);
|
||||
const retired = rows.filter(r => r.stale).length;
|
||||
if (retired) parts.push(`<b>${retired}</b> retired by review`);
|
||||
return parts.join(" · ");
|
||||
}
|
||||
|
||||
function render(q) {
|
||||
let html = "";
|
||||
|
||||
html += `<h2>To add <span class="count">— ${q.to_add.length} new collection entr${q.to_add.length === 1 ? "y" : "ies"}</span></h2>`;
|
||||
html += `<h2>To add <span class="count">— ${tally(q.to_add)}</span></h2>`;
|
||||
html += q.to_add.length
|
||||
? table(["game", "version", "seen in"], q.to_add.map(r => `
|
||||
? table(["game", "version", "seen in", ""], q.to_add.map(r => `
|
||||
<tr><td class="t">${esc(r.bgg_name)} <span class="meta">· ${esc(r.bgg_id)}</span></td>
|
||||
<td>${r.version_name ? esc(r.version_name) : `<span class="meta">no version</span>`}</td>
|
||||
<td class="meta">${esc(r.source_photos)}</td></tr>`))
|
||||
<td class="meta">${esc((r.source_photos ?? "").split(";").join(", "))}</td>
|
||||
<td>${stateChip(r)}</td></tr>`))
|
||||
: `<p class="empty">Nothing queued — run <b>diff</b> from the <a href="/">pipeline</a> first.</p>`;
|
||||
|
||||
html += `<h2>Version updates <span class="count">— ${q.to_update.length} existing entr${q.to_update.length === 1 ? "y" : "ies"} gaining a version</span></h2>`;
|
||||
html += `<h2>Version updates <span class="count">— ${tally(q.to_update)}</span></h2>`;
|
||||
html += q.to_update.length
|
||||
? table(["game", "version to set", "collection id"], q.to_update.map(r => `
|
||||
? table(["game", "version to set", "collection id", ""], q.to_update.map(r => `
|
||||
<tr><td class="t">${esc(r.bgg_name)} <span class="meta">· ${esc(r.bgg_id)}</span></td>
|
||||
<td>${esc(r.version_name)}</td>
|
||||
<td class="meta">${esc(r.collid)}</td></tr>`))
|
||||
<td class="meta">${esc(r.collid)}</td>
|
||||
<td>${stateChip(r)}</td></tr>`))
|
||||
: `<p class="empty">No version updates pending.</p>`;
|
||||
if (q.to_add.concat(q.to_update).some(r => r.state === "done"))
|
||||
html += `<p class="empty">Finished jobs stay listed until the next <b>diff</b>
|
||||
rebuilds the queue — the log below is the permanent record.</p>`;
|
||||
|
||||
html += `<h2>Upload log <span class="count">— every attempt ever made (${q.log.length})</span></h2>`;
|
||||
html += q.log.length
|
||||
@@ -43,13 +69,10 @@ function render(q) {
|
||||
document.getElementById("queuebody").innerHTML = html;
|
||||
}
|
||||
|
||||
let LAST = null;
|
||||
const GATE = changeGate();
|
||||
async function refresh() {
|
||||
const q = await fetchJSON("/api/queue");
|
||||
const payload = JSON.stringify(q);
|
||||
if (payload === LAST) return;
|
||||
LAST = payload;
|
||||
render(q);
|
||||
GATE(q, () => render(q));
|
||||
}
|
||||
|
||||
refresh().catch(err => errorBanner(err.message || err));
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
<span id="sectionlinks"></span>
|
||||
<span class="keyhelp">
|
||||
<kbd>j</kbd>/<kbd>k</kbd> move · <kbd>1</kbd>–<kbd>9</kbd> pick ·
|
||||
<kbd>r</kbd> reject · <kbd>m</kbd> manual id · <kbd>u</kbd> unknown ·
|
||||
<kbd>r</kbd> reject · <kbd>l</kbd> keep local · <kbd>m</kbd> manual id · <kbd>u</kbd> unknown ·
|
||||
<kbd>v</kbd> veto merge
|
||||
</span>
|
||||
</div>
|
||||
@@ -11,32 +11,16 @@
|
||||
<script>
|
||||
"use strict";
|
||||
let STATE = null;
|
||||
let active = 0;
|
||||
let ACTIVE = 0;
|
||||
|
||||
async function refresh() {
|
||||
const res = await fetch("/api/state");
|
||||
if (!res.ok) throw new Error(`${res.status}`);
|
||||
STATE = await res.json();
|
||||
STATE = await fetchJSON("/api/state");
|
||||
render();
|
||||
}
|
||||
|
||||
async function post(url, body) {
|
||||
let res;
|
||||
try {
|
||||
res = await fetch(url, {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
} catch (err) {
|
||||
alert("That didn't save (no response from the server): " + err);
|
||||
return;
|
||||
}
|
||||
if (!res.ok) {
|
||||
const detail = await res.json().then(d => d.detail).catch(() => null);
|
||||
alert("That didn't save: " + (detail ?? res.statusText));
|
||||
return;
|
||||
}
|
||||
const res = await apiPost(url, body);
|
||||
if (!res) return;
|
||||
try {
|
||||
STATE = await res.json();
|
||||
render();
|
||||
@@ -77,7 +61,9 @@ function matchCard(row, idx) {
|
||||
<kbd>${i + 1}</kbd> ${thumbHtml(c)}
|
||||
<span><span class="cname">${esc(c.name)}</span>
|
||||
<span class="cmeta">${esc(c.year ?? "—")} · ${esc(c.type || "?")}
|
||||
· rank ${esc(c.rank ?? "—")} · owned ${esc(c.owned ?? "—")}</span></span>
|
||||
· rank ${esc(c.rank ?? "—")} · owned ${esc(c.owned ?? "—")}
|
||||
· <a href="${bggUrl(c.bgg_id, c.type)}" target="_blank" rel="noopener"
|
||||
title="check this candidate before picking">view ↗</a></span></span>
|
||||
</li>`).join("");
|
||||
return `
|
||||
<section class="card actionable" data-kind="match" data-rowix="${row.row_ix}"
|
||||
@@ -91,7 +77,16 @@ function matchCard(row, idx) {
|
||||
<div class="rowactions">
|
||||
<span><kbd>m</kbd> <input type="text" inputmode="numeric" placeholder="BGG id, then ⏎"
|
||||
aria-label="manual BGG id"></span>
|
||||
<span class="research">
|
||||
<input type="text" class="rq" value="${esc(row.title_raw)}"
|
||||
aria-label="search text for ${esc(row.title_raw)}">
|
||||
<button class="dosearch" data-types="">search BGG</button>
|
||||
<button class="dosearch" data-types="rpgitem"
|
||||
title="BGG's board-game entries hide same-named RPGs from the automatic search">
|
||||
search RPGGeek</button>
|
||||
</span>
|
||||
<button class="reject" title="press r"><kbd>r</kbd> reject — not a game / bad read</button>
|
||||
<button class="golocal" title="press l"><kbd>l</kbd> not on BGG — keep locally</button>
|
||||
</div>
|
||||
</div>
|
||||
</section>`;
|
||||
@@ -109,12 +104,24 @@ function versionCard(row, idx) {
|
||||
return `
|
||||
<section class="card actionable" data-kind="version" data-rowix="${row.row_ix}"
|
||||
data-title="${esc(row.title_raw)}" data-photos="${esc(row.source_photos)}">
|
||||
${shots(row.photos || [])}
|
||||
<div class="body">
|
||||
<p class="title">${esc(row.bgg_name || row.title_raw)}</p>
|
||||
<p class="status">which edition? (skippable — <kbd>u</kbd> records "unknown", or just move on)</p>
|
||||
<p class="title">${esc(row.bgg_name || row.title_raw)}
|
||||
${row.bgg_id ? `<a class="cmeta" href="${bggUrl(row.bgg_id, row.type)}/versions"
|
||||
target="_blank" rel="noopener"
|
||||
title="BGG's own list of every printing">versions on BGG ↗</a>` : ""}</p>
|
||||
<p class="status">which edition${row.photos && row.photos.length
|
||||
? ` is the copy in ${esc(row.photos.join(", "))}`
|
||||
: ""}? (skippable — <kbd>u</kbd> records "unknown", or just move on)</p>
|
||||
<ol class="cands">${cands}</ol>
|
||||
<div class="rowactions">
|
||||
<button class="unknown" title="press u"><kbd>u</kbd> can't tell — leave version unset</button>
|
||||
<button class="allversions"
|
||||
title="the cues shortlisted these — fetch every published printing instead">
|
||||
list every printing</button>
|
||||
<button class="wronggame"
|
||||
title="none of these printings is your box: BGG may file it as a SEPARATE game — re-match it">
|
||||
wrong game — re-match</button>
|
||||
</div>
|
||||
</div>
|
||||
</section>`;
|
||||
@@ -142,7 +149,7 @@ function render() {
|
||||
<div class="done">
|
||||
<h2>${waiting
|
||||
? "Resolved set fully reviewed"
|
||||
: "All reviewed — the catalog is diff-ready"}</h2>
|
||||
: "All reviewed — every title is diff-ready"}</h2>
|
||||
<div class="nums">
|
||||
<div>${s.summary.extracted}<span>extracted</span></div>
|
||||
<div>${s.summary.recognized}<span>recognized</span></div>
|
||||
@@ -150,23 +157,34 @@ function render() {
|
||||
<div>${s.summary.rejected}<span>rejected</span></div>
|
||||
</div>
|
||||
${waiting
|
||||
? `<p class="waiting">${waiting} titles are extracted but not yet matched
|
||||
to BGG — they're waiting on the API token.</p>
|
||||
? s.summary.token_present
|
||||
? `<p class="waiting">${waiting} title(s) are extracted but not yet matched
|
||||
to BGG.</p>
|
||||
<p class="next">Run <b>resolve</b> from the <a href="/">pipeline</a> to match them.</p>`
|
||||
: `<p class="waiting">${waiting} title(s) are extracted but not yet matched
|
||||
to BGG — they're waiting on your API token
|
||||
(<a href="https://boardgamegeek.com/applications" target="_blank"
|
||||
rel="noopener">boardgamegeek.com/applications</a>).</p>
|
||||
<p class="next">When it arrives, run <b>resolve</b> from the <a href="/">pipeline</a>.</p>`
|
||||
: `<p class="next">Next: run <b>diff</b> from the <a href="/">pipeline</a>, then check the <a href="/queue">queue</a>.</p>`}
|
||||
</div>`;
|
||||
}
|
||||
// csv order is extraction order — alphabetize so a long session has a
|
||||
// findable, predictable shape (stable within a payload, so the keyboard
|
||||
// cursor stays put between polls)
|
||||
const byTitle = (a, b) =>
|
||||
a.title_raw.localeCompare(b.title_raw, undefined, { sensitivity: "base" });
|
||||
if (s.pending.length) {
|
||||
html += `<h2 id="matches">Matches <span class="count">— pick the game each photo shows</span></h2>`;
|
||||
html += s.pending.map(matchCard).join("");
|
||||
html += [...s.pending].sort(byTitle).map(matchCard).join("");
|
||||
}
|
||||
if (s.versions.length) {
|
||||
html += `<h2 id="editions">Editions <span class="count">— optional pass, never blocks uploads</span></h2>`;
|
||||
html += s.versions.map(versionCard).join("");
|
||||
html += [...s.versions].sort(byTitle).map(versionCard).join("");
|
||||
}
|
||||
if (s.merges.length) {
|
||||
html += `<h2 id="merges">Merges <span class="count">— duplicate reads folded into one game; veto if wrong</span></h2>`;
|
||||
html += s.merges.map(mg => `
|
||||
html += [...s.merges].sort(byTitle).map(mg => `
|
||||
<section class="card merge actionable" data-kind="merge"
|
||||
data-rowix="${mg.row_ix}"
|
||||
data-title="${esc(mg.title_raw)}" data-photos="${esc(mg.source_photos)}">
|
||||
@@ -182,9 +200,12 @@ function render() {
|
||||
m.innerHTML = html;
|
||||
|
||||
const cards = actionables();
|
||||
if (active >= cards.length) active = Math.max(0, cards.length - 1);
|
||||
if (ACTIVE >= cards.length) ACTIVE = Math.max(0, cards.length - 1);
|
||||
highlight();
|
||||
|
||||
// an outbound "view ↗" inside a pick row must not also cast the vote
|
||||
m.querySelectorAll("[data-pick] a, [data-pickver] a").forEach(a =>
|
||||
a.onclick = e => e.stopPropagation());
|
||||
m.querySelectorAll("[data-pick]").forEach(li => li.onclick = () => {
|
||||
const card = li.closest(".card");
|
||||
decide(card, "pick", Number(li.dataset.pick));
|
||||
@@ -195,6 +216,28 @@ function render() {
|
||||
});
|
||||
m.querySelectorAll(".reject").forEach(b => b.onclick = () =>
|
||||
decide(b.closest(".card"), "reject"));
|
||||
m.querySelectorAll(".golocal").forEach(b => b.onclick = () =>
|
||||
decide(b.closest(".card"), "local"));
|
||||
m.querySelectorAll(".dosearch").forEach(b => b.onclick = () => {
|
||||
const card = b.closest(".card");
|
||||
post("/api/research", {
|
||||
title_raw: card.dataset.title, source_photos: card.dataset.photos,
|
||||
row_ix: rowIx(card), query: card.querySelector(".rq").value,
|
||||
types: b.dataset.types || null,
|
||||
});
|
||||
});
|
||||
m.querySelectorAll(".wronggame").forEach(b => b.onclick = () =>
|
||||
post("/api/reopen-match", {
|
||||
title_raw: b.closest(".card").dataset.title,
|
||||
source_photos: b.closest(".card").dataset.photos,
|
||||
row_ix: rowIx(b.closest(".card")),
|
||||
}));
|
||||
m.querySelectorAll(".allversions").forEach(b => b.onclick = () =>
|
||||
post("/api/open-versions", {
|
||||
title_raw: b.closest(".card").dataset.title,
|
||||
source_photos: b.closest(".card").dataset.photos,
|
||||
row_ix: rowIx(b.closest(".card")),
|
||||
}));
|
||||
m.querySelectorAll(".unknown").forEach(b => b.onclick = () =>
|
||||
version(b.closest(".card"), "unknown"));
|
||||
m.querySelectorAll(".veto").forEach(b => b.onclick = () =>
|
||||
@@ -212,13 +255,15 @@ function render() {
|
||||
|
||||
const actionables = () => [...document.querySelectorAll(".actionable")];
|
||||
|
||||
function highlight() {
|
||||
actionables().forEach((el, i) => el.classList.toggle("active", i === active));
|
||||
const el = actionables()[active];
|
||||
if (el) el.scrollIntoView({block: "nearest", behavior: "auto"});
|
||||
function highlight(scroll = false) {
|
||||
actionables().forEach((el, i) => el.classList.toggle("active", i === ACTIVE));
|
||||
// scroll only for KEYBOARD moves: a mouse user's cursor idles on card
|
||||
// one, and scrolling to it on every re-render yanks them to the top
|
||||
const el = actionables()[ACTIVE];
|
||||
if (el && scroll) el.scrollIntoView({block: "nearest", behavior: "auto"});
|
||||
}
|
||||
|
||||
const rowIx = card => card.dataset.rowix === undefined ? null : Number(card.dataset.rowix);
|
||||
const rowIx = card => Number(card.dataset.rowix); // every card template sets it
|
||||
const decide = (card, action, bgg_id = null) => post("/api/decision", {
|
||||
title_raw: card.dataset.title, source_photos: card.dataset.photos,
|
||||
row_ix: rowIx(card), action, bgg_id,
|
||||
@@ -241,10 +286,10 @@ document.addEventListener("keydown", e => {
|
||||
if (e.target.tagName === "INPUT") return;
|
||||
const cards = actionables();
|
||||
if (!cards.length) return;
|
||||
const card = cards[active];
|
||||
const card = cards[ACTIVE];
|
||||
const kind = card?.dataset.kind;
|
||||
if (e.key === "j" || e.key === "ArrowDown") { active = Math.min(active + 1, cards.length - 1); highlight(); }
|
||||
else if (e.key === "k" || e.key === "ArrowUp") { active = Math.max(active - 1, 0); highlight(); }
|
||||
if (e.key === "j" || e.key === "ArrowDown") { ACTIVE = Math.min(ACTIVE + 1, cards.length - 1); highlight(true); }
|
||||
else if (e.key === "k" || e.key === "ArrowUp") { ACTIVE = Math.max(ACTIVE - 1, 0); highlight(true); }
|
||||
else if (/^[1-9]$/.test(e.key) && kind === "match") {
|
||||
const li = card.querySelectorAll("[data-pick]")[Number(e.key) - 1];
|
||||
if (li) decide(card, "pick", Number(li.dataset.pick));
|
||||
@@ -254,6 +299,7 @@ document.addEventListener("keydown", e => {
|
||||
if (li) version(card, "pick", Number(li.dataset.pickver));
|
||||
}
|
||||
else if (e.key === "r" && kind === "match") decide(card, "reject");
|
||||
else if (e.key === "l" && kind === "match") decide(card, "local");
|
||||
else if (e.key === "u" && kind === "version") version(card, "unknown");
|
||||
else if (e.key === "v" && kind === "merge") vetoMerge(card);
|
||||
else if (e.key === "m" && kind === "match") { card.querySelector("input")?.focus(); e.preventDefault(); }
|
||||
@@ -264,7 +310,6 @@ refresh().catch(err => errorBanner(err.message || err));
|
||||
// Live-follow the data files; only re-render on an actual change (keeps
|
||||
// the keyboard cursor stable) and never mid-typing. Stale poll responses
|
||||
// (answered before a decision landed) are discarded by revision.
|
||||
let lastGood = null;
|
||||
pollLoop(async () => {
|
||||
const fresh = await fetchJSON("/api/state");
|
||||
// discard stale responses — but only within one server lifetime: a
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
<h1>Titles</h1>
|
||||
<div class="pagebar"><span id="catcount"></span></div>
|
||||
<div class="filterbar">
|
||||
<input type="search" id="catsearch" placeholder="filter titles…" aria-label="filter titles">
|
||||
<button id="shakyfilter" aria-pressed="false" hidden
|
||||
title="reads the model wasn't sure of — confirm or fix each one">shaky reads</button>
|
||||
<button id="addtitle" aria-expanded="false"
|
||||
title="a game no photo shows — an expansion inside a base box, a game away from the shelves">add a game</button>
|
||||
</div>
|
||||
<form id="addform" class="editform card" hidden>
|
||||
<label>Title <input name="title" required></label>
|
||||
<label>Publisher <input name="publisher"></label>
|
||||
<label>Edition <input name="edition"></label>
|
||||
<label>Year <input name="year" inputmode="numeric" size="6"></label>
|
||||
<label>Language <input name="language"></label>
|
||||
<span class="editactions">
|
||||
<button type="submit" class="primary">add</button>
|
||||
<button type="button" id="addcancel">cancel</button>
|
||||
</span>
|
||||
<span class="edithint">joins the titles list like a photo read — resolve matches it next run</span>
|
||||
</form>
|
||||
<div id="catbody"><p class="empty">Nothing extracted yet — start on the <a href="/photos">photos page</a>.</p></div>
|
||||
<script>
|
||||
"use strict";
|
||||
let CATALOG = [];
|
||||
let EDITING = null; // lineKey of the row whose editor is open
|
||||
|
||||
function lineKey(c) { return c.title_raw + "|" + c.photos.join(";"); }
|
||||
|
||||
function editorRow(c) {
|
||||
const cue = c.cues || {};
|
||||
return `
|
||||
<tr class="editrow"><td colspan="5">
|
||||
<form class="editform" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}" data-rowix="${c.row_ix ?? ""}">
|
||||
<label>Title <input name="title" value="${esc(c.title_raw)}" required></label>
|
||||
<label>Publisher <input name="publisher" value="${esc(cue.publisher || "")}"></label>
|
||||
<label>Edition <input name="edition" value="${esc(cue.edition || "")}"></label>
|
||||
<label>Year <input name="year" value="${esc(cue.year ?? "")}" inputmode="numeric" size="6"></label>
|
||||
<label>Language <input name="language" value="${esc(cue.language || "")}"></label>
|
||||
<span class="editactions">
|
||||
<button type="submit" class="primary">save</button>
|
||||
<button type="button" class="canceledit">cancel</button>
|
||||
${c.bgg_id && ["auto", "approved"].includes(c.status)
|
||||
? `<button type="button" class="wrongmatch danger">wrong match — re-search</button>`
|
||||
: ""}
|
||||
<button type="button" class="removetitle danger"
|
||||
title="not a game, a duplicate read of a copy already listed, or otherwise doesn't belong">
|
||||
remove this line</button>
|
||||
</span>
|
||||
<span class="edithint">saving re-queues this title for resolve with the corrected data</span>
|
||||
</form>
|
||||
</td></tr>`;
|
||||
}
|
||||
|
||||
let SHAKY_ONLY = false;
|
||||
|
||||
function render() {
|
||||
const q = document.getElementById("catsearch").value.trim().toLowerCase();
|
||||
const shakyCount = CATALOG.filter(c => c.shaky).length;
|
||||
const fbtn = document.getElementById("shakyfilter");
|
||||
fbtn.hidden = shakyCount === 0 && !SHAKY_ONLY;
|
||||
fbtn.textContent = `shaky reads — ${shakyCount}`;
|
||||
fbtn.setAttribute("aria-pressed", String(SHAKY_ONLY));
|
||||
const sorted = [...CATALOG].sort((a, b) =>
|
||||
a.title_raw.localeCompare(b.title_raw, undefined, { sensitivity: "base" }));
|
||||
const searched = q
|
||||
? sorted.filter(c => (c.title_raw + " " + c.bgg_name).toLowerCase().includes(q))
|
||||
: sorted;
|
||||
const rows = SHAKY_ONLY ? searched.filter(c => c.shaky) : searched;
|
||||
document.getElementById("catcount").innerHTML =
|
||||
`<b>${rows.length}</b> of <b>${CATALOG.length}</b> title(s)`;
|
||||
document.getElementById("catbody").innerHTML = rows.length
|
||||
? `<div class="catalog"><table>` + rows.map(c => `
|
||||
<tr>
|
||||
<td class="t">${esc(c.title_raw)}
|
||||
${c.split_copy ? `<span class="chip merged">copy</span>` : ""}
|
||||
${c.shaky ? `<span class="chip shaky"
|
||||
title="the model wasn't sure of this read — press ✓ if it's right, or edit it">shaky read</span>` : ""}</td>
|
||||
<td>${statusChip(c)}</td>
|
||||
<td class="meta">${metaLine(c)}</td>
|
||||
<td class="meta">${c.photos.length
|
||||
? c.photos.map(p =>
|
||||
`<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>`
|
||||
).join(", ")
|
||||
: `<span class="chip merged">added by hand</span>`}</td>
|
||||
<td class="actions">${c.can_split
|
||||
? `<button class="split" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}"
|
||||
data-rowix="${c.row_ix ?? ""}"
|
||||
title="one line, several boxes? make each photo its own copy">
|
||||
split into copies</button>`
|
||||
: ""}
|
||||
${c.status === "local"
|
||||
? `<button class="wrongmatch" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}" data-rowix="${c.row_ix ?? ""}"
|
||||
title="search BGG and RPGGeek again — RPGs and expansions often are listed">
|
||||
look it up</button>`
|
||||
: ""}
|
||||
${c.version_status === "version_ambiguous"
|
||||
? `<a class="ballotlink" href="/review#editions"
|
||||
title="this copy's edition ballot is open">ballot in Review →</a>`
|
||||
: ""}
|
||||
${c.bgg_id && ["auto", "approved"].includes(c.status)
|
||||
&& ["version_unknown", "version_error", ""].includes(c.version_status || "")
|
||||
? `<button class="pickedition" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}" data-rowix="${c.row_ix ?? ""}"
|
||||
title="you know which printing this box is — fetch the full edition list into Review">
|
||||
pick edition</button>`
|
||||
: ""}
|
||||
${c.shaky ? `<button class="confirmread" data-title="${esc(c.title_raw)}"
|
||||
data-photos="${esc(c.photos.join(";"))}"
|
||||
title="the read is correct as-is — mark it verified">✓ looks right</button>` : ""}
|
||||
<button class="edit" data-key="${esc(lineKey(c))}"
|
||||
title="fix a misread title or add cues you already know">edit</button>
|
||||
</td>
|
||||
</tr>` + (EDITING === lineKey(c) ? editorRow(c) : "")).join("") + `</table></div>`
|
||||
: `<p class="empty">${CATALOG.length
|
||||
? "No titles match that filter."
|
||||
: `Nothing extracted yet — start on the <a href="/photos">photos page</a>.`}</p>`;
|
||||
}
|
||||
|
||||
const GATE = changeGate();
|
||||
async function refresh() {
|
||||
const state = await fetchJSON("/api/state");
|
||||
if (EDITING) return; // never repaint under an open editor
|
||||
GATE(state.catalog, () => { CATALOG = state.catalog; render(); });
|
||||
}
|
||||
|
||||
document.getElementById("catbody").addEventListener("click", async e => {
|
||||
const cancel = e.target.closest("button.canceledit");
|
||||
if (cancel) { EDITING = null; GATE.reset(); render(); refresh().catch(() => {}); return; }
|
||||
const wm = e.target.closest("button.wrongmatch");
|
||||
if (wm) {
|
||||
// in the edit panel it means "wrong match"; on a local row it means
|
||||
// "look it up" — same reopen, different starting point
|
||||
const f = wm.closest("form.editform");
|
||||
const src = f ? f.dataset : wm.dataset;
|
||||
if (f && !confirm(`"${src.title}" matched the wrong game? This clears the ` +
|
||||
`match and sends it back to Review for a re-search.`)) return;
|
||||
const res = await apiPost("/api/reopen-match", {
|
||||
title_raw: src.title,
|
||||
source_photos: src.photos,
|
||||
row_ix: src.rowix === "" ? null : Number(src.rowix),
|
||||
});
|
||||
if (res) {
|
||||
EDITING = null; GATE.reset(); refresh().catch(() => {});
|
||||
showToast(`searching BGG and RPGGeek for <b>${esc(src.title)}</b> —
|
||||
<a href="/review#matches">pick a match in Review</a>`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
const pe = e.target.closest("button.pickedition");
|
||||
if (pe) {
|
||||
const res = await apiPost("/api/open-versions", {
|
||||
title_raw: pe.dataset.title,
|
||||
source_photos: pe.dataset.photos,
|
||||
row_ix: pe.dataset.rowix === "" ? null : Number(pe.dataset.rowix),
|
||||
});
|
||||
if (res) {
|
||||
showToast(`edition ballot ready for <b>${esc(pe.dataset.title)}</b> —
|
||||
<a href="/review#editions">pick it in Review</a>`);
|
||||
GATE.reset(); refresh().catch(() => {});
|
||||
}
|
||||
return;
|
||||
}
|
||||
const ok = e.target.closest("button.confirmread");
|
||||
if (ok) {
|
||||
const res = await apiPost("/api/edit-title", {
|
||||
title_raw: ok.dataset.title,
|
||||
source_photos: ok.dataset.photos,
|
||||
confirm: true,
|
||||
});
|
||||
if (res) { GATE.reset(); refresh().catch(() => {}); }
|
||||
return;
|
||||
}
|
||||
const rm = e.target.closest("button.removetitle");
|
||||
if (rm) {
|
||||
const f = rm.closest("form.editform");
|
||||
if (!confirm(`Remove "${f.dataset.title}" from the titles list? ` +
|
||||
`Right for a line that shouldn't exist: not a game, or a duplicate ` +
|
||||
`read of a copy that's already its own line. ` +
|
||||
`(One line for several REAL boxes? Use split instead.) ` +
|
||||
`Re-running extract won't bring it back — the removal is saved ` +
|
||||
`in data/title_removals.json (delete its record there to undo).`)) return;
|
||||
const res = await apiPost("/api/remove-title", {
|
||||
title_raw: f.dataset.title,
|
||||
source_photos: f.dataset.photos,
|
||||
});
|
||||
if (res) { EDITING = null; GATE.reset(); refresh().catch(() => {}); }
|
||||
return;
|
||||
}
|
||||
const edit = e.target.closest("button.edit");
|
||||
if (edit) {
|
||||
EDITING = EDITING === edit.dataset.key ? null : edit.dataset.key;
|
||||
render();
|
||||
return;
|
||||
}
|
||||
const b = e.target.closest("button.split");
|
||||
if (!b) return;
|
||||
const n = b.dataset.photos.split(";").length;
|
||||
if (!confirm(`Split "${b.dataset.title}" into ${n} separate copies (one per photo)? ` +
|
||||
`Each picks its own edition afterward.`)) return;
|
||||
const res = await apiPost("/api/split", {
|
||||
title_raw: b.dataset.title,
|
||||
source_photos: b.dataset.photos,
|
||||
row_ix: b.dataset.rowix === "" ? null : Number(b.dataset.rowix),
|
||||
});
|
||||
if (res) refresh().catch(() => {});
|
||||
});
|
||||
|
||||
document.getElementById("catbody").addEventListener("submit", async e => {
|
||||
const f = e.target.closest("form.editform");
|
||||
if (!f) return;
|
||||
e.preventDefault();
|
||||
const orig = CATALOG.find(c => lineKey(c) === EDITING) || {};
|
||||
const cue = orig.cues || {};
|
||||
const v = name => f.elements[name].value;
|
||||
const body = { title_raw: f.dataset.title, source_photos: f.dataset.photos };
|
||||
if (v("title").trim() !== f.dataset.title) body.title_new = v("title").trim();
|
||||
if (v("publisher") !== (cue.publisher || "")) body.publisher = v("publisher");
|
||||
if (v("edition") !== (cue.edition || "")) body.edition = v("edition");
|
||||
if (v("year") !== String(cue.year ?? "")) body.year = v("year");
|
||||
if (v("language") !== (cue.language || "")) body.language = v("language");
|
||||
if (Object.keys(body).length <= 2) {
|
||||
// saving unchanged: on a shaky line that means "it's right as-is"
|
||||
if (orig.shaky) {
|
||||
const res = await apiPost("/api/edit-title", { ...body, confirm: true });
|
||||
if (res) { EDITING = null; GATE.reset(); refresh().catch(() => {}); return; }
|
||||
}
|
||||
EDITING = null; render(); return;
|
||||
}
|
||||
const res = await apiPost("/api/edit-title", body);
|
||||
if (res) {
|
||||
EDITING = null; GATE.reset(); refresh().catch(() => {});
|
||||
// an edit re-queues any matched row: without a resolve run the line
|
||||
// sits "awaiting resolve" — say so, and offer the run right here
|
||||
const toast = showToast(`saved — a previously matched title re-matches
|
||||
on the next resolve run. <button id="resolvenow">run resolve now</button>`);
|
||||
const btn = toast.querySelector("#resolvenow");
|
||||
if (btn) btn.addEventListener("click", async () => {
|
||||
const r = await apiPost("/api/run/resolve");
|
||||
if (r) showToast(`resolve running — progress on the
|
||||
<a href="/">Pipeline</a> page; this list updates itself`);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
const addForm = document.getElementById("addform");
|
||||
const addBtn = document.getElementById("addtitle");
|
||||
addBtn.addEventListener("click", () => {
|
||||
addForm.hidden = !addForm.hidden;
|
||||
addBtn.setAttribute("aria-expanded", String(!addForm.hidden));
|
||||
if (!addForm.hidden) addForm.elements.title.focus();
|
||||
});
|
||||
document.getElementById("addcancel").addEventListener("click", () => {
|
||||
addForm.hidden = true;
|
||||
addBtn.setAttribute("aria-expanded", "false");
|
||||
addForm.reset();
|
||||
});
|
||||
addForm.addEventListener("submit", async e => {
|
||||
e.preventDefault();
|
||||
const v = name => addForm.elements[name].value;
|
||||
const res = await apiPost("/api/add-title", {
|
||||
title: v("title"), publisher: v("publisher"), edition: v("edition"),
|
||||
year: v("year"), language: v("language"),
|
||||
});
|
||||
if (res) {
|
||||
addForm.reset();
|
||||
addForm.hidden = true;
|
||||
addBtn.setAttribute("aria-expanded", "false");
|
||||
GATE.reset();
|
||||
refresh().catch(() => {});
|
||||
}
|
||||
});
|
||||
|
||||
document.getElementById("catsearch").addEventListener("input", render);
|
||||
document.getElementById("shakyfilter").addEventListener("click", () => {
|
||||
SHAKY_ONLY = !SHAKY_ONLY;
|
||||
render();
|
||||
});
|
||||
refresh().catch(err => errorBanner(err.message || err));
|
||||
pollLoop(refresh, 5000, () => showBanner(""));
|
||||
</script>
|
||||
@@ -5,6 +5,8 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title><!--TITLE--></title>
|
||||
<link rel="icon" type="image/png" href="/static/favicon.png">
|
||||
<link rel="apple-touch-icon" href="/static/apple-touch-icon.png">
|
||||
<meta name="apple-mobile-web-app-title" content="bggpipe">
|
||||
<link rel="stylesheet" href="/static/app.css">
|
||||
</head>
|
||||
<body>
|
||||
@@ -14,7 +16,9 @@
|
||||
<img src="/static/logo.jpg" alt="">
|
||||
<span class="wordmark">bggpipe<small>shelf → BGG pipeline</small></span>
|
||||
</a>
|
||||
<nav aria-label="Primary">
|
||||
<button class="navtoggle" aria-expanded="false" aria-controls="primary-nav"
|
||||
aria-label="Menu"><span class="bars"></span></button>
|
||||
<nav id="primary-nav" aria-label="Primary">
|
||||
<!--NAV-->
|
||||
</nav>
|
||||
<figure class="piperbox">
|
||||
@@ -27,7 +31,7 @@
|
||||
</figure>
|
||||
</div>
|
||||
<div class="content">
|
||||
<div id="banner"></div>
|
||||
<div id="banner" role="status"></div>
|
||||
<script src="/static/app.js"></script>
|
||||
<main id="main">
|
||||
<!--PAGE-->
|
||||
|
||||
@@ -39,7 +39,8 @@ import typer
|
||||
from bggpipe.bgg_client import BGGAuthError, BGGClient, client_for
|
||||
from bggpipe.config import Config
|
||||
from bggpipe.fsio import atomic_write_text
|
||||
from bggpipe.models import CollectionItem
|
||||
from bggpipe.models import CollectionItem, is_recognized
|
||||
from bggpipe.resolve import read_matches
|
||||
|
||||
BGG = "https://boardgamegeek.com"
|
||||
UPLOAD_LOG_COLUMNS = [
|
||||
@@ -57,6 +58,16 @@ DONE_STATUSES = {"added", "added_no_version", "updated", "already_present"}
|
||||
MAX_VERSION_PAGES = 40
|
||||
|
||||
|
||||
_YEARISH = re.compile(r"[\s(]*\d{4}(?:-\d+)?[\s)]*")
|
||||
|
||||
|
||||
def _loose_version_text(text: str) -> str:
|
||||
"""Version names as displayed differ from the API's by punctuation,
|
||||
parens and year qualifiers — compare only the words that name it."""
|
||||
text = _YEARISH.sub(" ", text or "")
|
||||
return " ".join(re.sub(r"[^\w\s]", " ", text.casefold()).split())
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UploadJob:
|
||||
action: str # "add" | "update"
|
||||
@@ -89,6 +100,116 @@ def _job_key(row: dict) -> tuple[str, str, str]:
|
||||
return _key(row["action"], row["bgg_id"], row["collid"], row["version_id"])
|
||||
|
||||
|
||||
def annotate_queue(
|
||||
queue_rows: list[dict], action: str, log_rows: list[dict]
|
||||
) -> list[dict]:
|
||||
"""Each queue row plus the outcome of its LAST upload attempt, in a
|
||||
`state` field: "" (pending), "done", or "failed". to_add.csv and
|
||||
to_update.csv are diff-time snapshots — nothing removes a row once its
|
||||
job succeeds — so a reader without the log sees finished work as
|
||||
outstanding forever."""
|
||||
last: dict[tuple[str, str, str], str] = {}
|
||||
done_count: Counter[tuple[str, str, str]] = Counter()
|
||||
for row in log_rows:
|
||||
last[_job_key(row)] = row["status"]
|
||||
if row["status"] in DONE_STATUSES:
|
||||
done_count[_job_key(row)] += 1
|
||||
# vetoed duplicate copies share a key: one success must mark ONE row
|
||||
# done, not both, or the queue reports a copy uploaded that never was
|
||||
claimed: Counter[tuple[str, str, str]] = Counter()
|
||||
out = []
|
||||
for row in queue_rows:
|
||||
key = _key(
|
||||
action,
|
||||
row.get("bgg_id", ""),
|
||||
row.get("collid", ""),
|
||||
row.get("version_id", ""),
|
||||
)
|
||||
status = last.get(key, "")
|
||||
if status in DONE_STATUSES and claimed[key] >= done_count[key]:
|
||||
status = "" # completions exhausted: this copy is still pending
|
||||
if status in DONE_STATUSES:
|
||||
claimed[key] += 1
|
||||
out.append(
|
||||
{
|
||||
**row,
|
||||
"state": "done"
|
||||
if status in DONE_STATUSES
|
||||
else "failed"
|
||||
if status == "failed"
|
||||
else "",
|
||||
"last_status": status,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def stale_jobs(queue_rows: list[dict], match_rows: list[dict]) -> dict[str, str]:
|
||||
"""bgg_id -> why, for queued games the CURRENT matches.csv no longer
|
||||
endorses. to_add.csv is a snapshot from the last diff; a review
|
||||
decision taken afterwards (marking a game local, rejecting it, calling
|
||||
a match wrong) must not still upload."""
|
||||
if not match_rows:
|
||||
# nothing to compare against (no matches.csv, or a test harness
|
||||
# driving the queue directly): absence is not a verdict
|
||||
return {}
|
||||
live: dict[str, list[dict]] = {}
|
||||
endorsed: Counter[tuple[str, str]] = Counter()
|
||||
for row in match_rows:
|
||||
if row.get("bgg_id"):
|
||||
live.setdefault(row["bgg_id"], []).append(row)
|
||||
if is_recognized(row):
|
||||
endorsed[(row["bgg_id"], row.get("version_id", ""))] += 1
|
||||
queued: Counter[tuple[str, str]] = Counter()
|
||||
for row in queue_rows:
|
||||
queued[(row.get("bgg_id") or "", row.get("version_id", ""))] += 1
|
||||
stale = {}
|
||||
for row in queue_rows:
|
||||
bgg_id = row.get("bgg_id") or ""
|
||||
rows = live.get(bgg_id, [])
|
||||
if not rows:
|
||||
stale[bgg_id] = "no longer matched to this game in matches.csv"
|
||||
continue
|
||||
if not any(is_recognized(r) for r in rows):
|
||||
statuses = sorted({r["match_status"] for r in rows})
|
||||
stale[bgg_id] = f"now {', '.join(statuses)} in matches.csv"
|
||||
continue
|
||||
# the game survives, but does THIS copy? Rejecting one of two
|
||||
# editions must not ride along on the other's endorsement.
|
||||
pair = (bgg_id, row.get("version_id", ""))
|
||||
if queued[pair] > endorsed[pair]:
|
||||
stale[bgg_id] = (
|
||||
f"queued with version {row.get('version_id') or '(none)'} "
|
||||
"but matches.csv no longer endorses that copy — re-run diff"
|
||||
)
|
||||
return stale
|
||||
|
||||
|
||||
def outstanding_failures(
|
||||
log_rows: list[dict],
|
||||
queue_rows: list[dict] | None = None,
|
||||
match_rows: list[dict] | None = None,
|
||||
) -> int:
|
||||
"""Failures still worth retrying: the job's LATEST attempt failed AND
|
||||
it is still queued and still endorsed by matches.csv.
|
||||
|
||||
Two ways to over-count. upload_log.csv is append-only, so a successful
|
||||
retry leaves its old 'failed' line behind; and a failed job whose
|
||||
review decision has since changed (marked local, rejected) is not
|
||||
pending work at all — it will never run again, so offering to retry it
|
||||
is a lie."""
|
||||
last: dict[tuple[str, str, str], tuple[str, str]] = {}
|
||||
for row in log_rows:
|
||||
last[_job_key(row)] = (row["status"], row.get("bgg_id", ""))
|
||||
failed = [bgg_id for status, bgg_id in last.values() if status == "failed"]
|
||||
if queue_rows is None:
|
||||
return len(failed)
|
||||
live = {r.get("bgg_id", "") for r in queue_rows} - set(
|
||||
stale_jobs(queue_rows, match_rows or [])
|
||||
)
|
||||
return sum(1 for bgg_id in failed if bgg_id in live)
|
||||
|
||||
|
||||
def _read_csv(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
@@ -114,10 +235,9 @@ def build_queue(
|
||||
log_rows: list[dict],
|
||||
*,
|
||||
retry_failed: bool = False,
|
||||
) -> tuple[list[UploadJob], int, int, list[UploadJob]]:
|
||||
) -> tuple[list[UploadJob], int, int]:
|
||||
"""Turn the diff outputs into pending jobs, minus work the log says is
|
||||
done. Returns (jobs, skipped_done, skipped_failed, deferred) — deferred
|
||||
being same-game updates held for a later run.
|
||||
done. Returns (jobs, skipped_done, skipped_failed).
|
||||
|
||||
Completions are COUNTED per key, not looked up: two vetoed duplicate
|
||||
copies share a key, and one logged success must complete exactly one
|
||||
@@ -161,8 +281,6 @@ def build_queue(
|
||||
|
||||
jobs: list[UploadJob] = []
|
||||
skipped_done = skipped_failed = 0
|
||||
deferred: list[UploadJob] = []
|
||||
update_game_seen: set[str] = set()
|
||||
seen: Counter[tuple[str, str, str]] = Counter()
|
||||
queued_versions: dict[str, set[str]] = {}
|
||||
for job in candidates:
|
||||
@@ -179,11 +297,13 @@ def build_queue(
|
||||
prior = set()
|
||||
if prior and job.version_id not in prior:
|
||||
typer.echo(
|
||||
f" note: {job.name} was previously {job.action}ed with a "
|
||||
f" skipping {job.name}: previously {job.action}ed with a "
|
||||
f"different version ({', '.join(sorted(prior)) or 'none'}) — "
|
||||
"if re-review changed the version, the BGG entry needs a "
|
||||
"manual correction (additive-only rule)"
|
||||
"the entry exists on BGG, so re-adding can only duplicate "
|
||||
"it; correct the version by hand (additive-only rule)"
|
||||
)
|
||||
skipped_done += 1
|
||||
continue
|
||||
occurrence = seen[job.key]
|
||||
seen[job.key] += 1
|
||||
if not job.name:
|
||||
@@ -198,27 +318,23 @@ def build_queue(
|
||||
skipped_done += 1
|
||||
elif last_status.get(job.key) == "failed" and not retry_failed:
|
||||
skipped_failed += 1
|
||||
elif job.action == "update" and job.bgg_id in update_game_seen:
|
||||
# The row-edit flow finds rows by game name, not collid — a
|
||||
# second same-game update this run could reopen the copy the
|
||||
# first one just versioned and overwrite it. One per run; the
|
||||
# next run (after --verify) picks up the rest.
|
||||
deferred.append(job)
|
||||
else:
|
||||
if job.action == "update":
|
||||
update_game_seen.add(job.bgg_id)
|
||||
# same-game updates coexist in one run: update_entry addresses
|
||||
# the copy by collid and the edition by radio value, so a
|
||||
# second update cannot reopen what the first just saved
|
||||
jobs.append(job)
|
||||
return jobs, skipped_done, skipped_failed, deferred
|
||||
return jobs, skipped_done, skipped_failed
|
||||
|
||||
|
||||
def _scrub(text: str) -> str:
|
||||
"""Credentials must never reach the log, even via a selector error that
|
||||
echoes filled form values."""
|
||||
echoes filled form values. Playwright errors carry multi-line call
|
||||
logs; a CSV cell keeps one line."""
|
||||
for key in ("BGG_PASSWORD", "BGG_USERNAME"):
|
||||
value = os.environ.get(key)
|
||||
if value:
|
||||
text = text.replace(value, "***")
|
||||
return text
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
class LoginError(RuntimeError):
|
||||
@@ -236,10 +352,11 @@ class Uploader(Protocol):
|
||||
class PlaywrightUploader:
|
||||
"""Drives the real site. Selector documentation: docs/bgg-upload-flow.md.
|
||||
|
||||
Verified selectors (2026-08-01): the login form (#inputUsername /
|
||||
#inputPassword / "Sign In") and the Add-to-Collection dialog structure.
|
||||
UNVERIFIED: version-picker pagination, post-save behavior, and the whole
|
||||
collection-row edit flow for updates — expect first-real-run adjustments.
|
||||
All flows verified against the live site 2026-08-06 (adds, the version
|
||||
picker with pagination, and the collection-row version edit). The
|
||||
remaining unknown is the second-copy add: BGG may edit the existing
|
||||
entry instead of creating one, which run_upload reports as a copy-count
|
||||
shortfall during --verify.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -275,11 +392,31 @@ class PlaywrightUploader:
|
||||
# plus per-element auto-waiting is the reliable pattern.
|
||||
self._page.goto(url, wait_until="domcontentloaded")
|
||||
|
||||
# BGG's Sign In control is an <a class="btn"> with NO href, so it has
|
||||
# no implicit link role — role-based queries cannot find it.
|
||||
_SIGN_IN = 'a:has-text("Sign In"), button:has-text("Sign In")'
|
||||
_SIGNED_IN = ':text-is("Sign Out"), :text-is("Log Out")'
|
||||
|
||||
def _signed_out(self) -> bool:
|
||||
# Heuristic: the header shows a "Sign In" link only when logged out.
|
||||
return (
|
||||
self._page.get_by_role("link", name=re.compile(r"^sign in$", re.I)).count()
|
||||
> 0
|
||||
"""True = signed out. Waits for the header to prove ONE state or
|
||||
the other: BGG hydrates it after domcontentloaded, so an early read
|
||||
finds NEITHER control — and any check resting on a single absence
|
||||
guesses, silently, in the unsafe direction. Undeterminable state
|
||||
raises rather than assumes."""
|
||||
page = self._page
|
||||
deadline = time.monotonic() + 30
|
||||
while time.monotonic() < deadline:
|
||||
out = page.locator(self._SIGN_IN).count()
|
||||
inside = page.locator(self._SIGNED_IN).count()
|
||||
if out and not inside:
|
||||
return True
|
||||
if inside and not out:
|
||||
return False
|
||||
page.wait_for_timeout(250)
|
||||
raise LoginError(
|
||||
"could not tell whether this browser is signed in (neither a "
|
||||
"Sign In nor a Sign Out control appeared in 30s) — BGG's header "
|
||||
"may have changed; see docs/bgg-upload-flow.md"
|
||||
)
|
||||
|
||||
def _ensure_logged_in(self) -> None:
|
||||
@@ -313,6 +450,15 @@ class PlaywrightUploader:
|
||||
"login did not complete (wrong credentials, or the login "
|
||||
"page changed — see docs/bgg-upload-flow.md)"
|
||||
) from err
|
||||
# Verify the TRANSITION: the Sign In control was visible a
|
||||
# moment ago, so its disappearance is evidence, not inference.
|
||||
self._goto(f"{BGG}/")
|
||||
if self._signed_out():
|
||||
raise LoginError(
|
||||
"the login form was submitted but the site still offers "
|
||||
"Sign In — credentials rejected, or a challenge is "
|
||||
"pending in the browser window"
|
||||
)
|
||||
self._context.storage_state(path=str(self._storage_state))
|
||||
self._storage_state.chmod(0o600) # session cookies: owner-only
|
||||
self._authed = True
|
||||
@@ -332,14 +478,49 @@ class PlaywrightUploader:
|
||||
raise
|
||||
return dialog
|
||||
|
||||
def _select_version(self, dialog, version_name: str) -> bool:
|
||||
"""Page through the version sub-view matching the canonical version
|
||||
NAME (the list has no search box). Returns False — with the sub-view
|
||||
cancelled — when the name never shows up."""
|
||||
# Verified 2026-08-06 against the live picker: version rows are the
|
||||
# <li>s carrying a thumbnail (the paging <li>s are not), and paging is
|
||||
# an AngularJS <ul class="pagination"> of anchors, not buttons.
|
||||
_VERSION_ROWS = "li:has(.summary-item-thumbnail)"
|
||||
# Paging controls render TWICE — a desktop set and a mobile set marked
|
||||
# visible-xs-* — so every one of these selectors matches hidden nodes
|
||||
# too; clicking one waits forever. Always pick the visible match.
|
||||
_NEXT_PAGE = 'ul.pagination a[title="Next Page"]'
|
||||
_PAGE_LINKS = "ul.pagination a"
|
||||
# Rows read "<game name> (<version name>) (<year>)", so the version name
|
||||
# is matched inside its parentheses — bare substrings would let
|
||||
# "English edition" match "English edition, second printing".
|
||||
_TRAILING_YEAR = re.compile(r"\s+\d{4}(?:-\d+)?$")
|
||||
|
||||
def _rows_on_page(self, dialog) -> list[str]:
|
||||
return [
|
||||
" ".join(t.split())
|
||||
for t in dialog.locator(self._VERSION_ROWS).all_text_contents()
|
||||
]
|
||||
|
||||
def _visible(self, dialog, selector):
|
||||
"""The first VISIBLE match, or None — see _NEXT_PAGE."""
|
||||
loc = dialog.locator(selector)
|
||||
for i in range(loc.count()):
|
||||
candidate = loc.nth(i)
|
||||
if candidate.is_visible():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _has_next_page(self, dialog) -> bool:
|
||||
nxt = self._visible(dialog, self._NEXT_PAGE)
|
||||
if nxt is None:
|
||||
return False # single-page list: the container is ng-show'd off
|
||||
return "disabled" not in (nxt.evaluate("e => e.closest('li').className") or "")
|
||||
|
||||
def _select_version(self, dialog, version_name: str) -> tuple[bool, str]:
|
||||
"""Pick the version whose name matches, paging the whole list first
|
||||
so a decision is made against every candidate. Returns (picked,
|
||||
reason); on failure the sub-view is cancelled and the caller adds
|
||||
the game version-less — never a guessed edition (spec)."""
|
||||
dialog.get_by_role("button", name="Set version/edition").click()
|
||||
pattern = re.compile(re.escape(version_name), re.I)
|
||||
try:
|
||||
dialog.get_by_role("listitem").first.wait_for(timeout=15_000)
|
||||
dialog.locator(self._VERSION_ROWS).first.wait_for(timeout=15_000)
|
||||
except self._timeout_error as err:
|
||||
# a version resolve found on BGG can't be missing from the
|
||||
# picker: an unrendered list means a slow page or changed markup
|
||||
@@ -347,18 +528,28 @@ class PlaywrightUploader:
|
||||
"version picker never rendered — site slow or markup "
|
||||
"changed; attempt is retryable"
|
||||
) from err
|
||||
for _ in range(MAX_VERSION_PAGES):
|
||||
items = dialog.get_by_role("listitem").filter(has_text=pattern)
|
||||
if items.count():
|
||||
items.first.click()
|
||||
return True
|
||||
# Pagination controls are UNVERIFIED (docs/bgg-upload-flow.md);
|
||||
# best guess is a next-page button, stopping when absent/disabled.
|
||||
nxt = dialog.get_by_role("button", name=re.compile("next|›|»", re.I)).first
|
||||
if nxt.count() == 0 or nxt.is_disabled():
|
||||
break # genuine end of list: added_no_version is honest
|
||||
nxt.click()
|
||||
self._page.wait_for_timeout(2_000) # etiquette: paginating hits BGG too
|
||||
|
||||
wanted = f"({version_name})".casefold()
|
||||
# BGG's API name can carry a printing qualifier the picker omits
|
||||
# ("English edition 2018-2" vs "(English edition) (2018)")
|
||||
relaxed = f"({self._TRAILING_YEAR.sub('', version_name).strip()})".casefold()
|
||||
self._goto_first_page(dialog) # a fresh sub-view opens on page 1,
|
||||
# but never assume it: paging state outlives a close/reopen
|
||||
exact: list[tuple[int, str]] = []
|
||||
loose: list[tuple[int, str]] = []
|
||||
pages = 0
|
||||
for page_ix in range(MAX_VERSION_PAGES):
|
||||
pages = page_ix + 1
|
||||
for text in self._rows_on_page(dialog):
|
||||
folded = text.casefold()
|
||||
if wanted in folded:
|
||||
exact.append((page_ix, text))
|
||||
elif relaxed != wanted and relaxed in folded:
|
||||
loose.append((page_ix, text))
|
||||
if not self._has_next_page(dialog):
|
||||
break
|
||||
self._visible(dialog, self._NEXT_PAGE).click()
|
||||
self._page.wait_for_timeout(400) # client-side paging: no request
|
||||
else:
|
||||
# never saw the end of the list: "not in picker" would be a false
|
||||
# verdict frozen into DONE_STATUSES
|
||||
@@ -366,61 +557,182 @@ class PlaywrightUploader:
|
||||
f"hit MAX_VERSION_PAGES ({MAX_VERSION_PAGES}) without "
|
||||
"finding the version or the end of the list — retryable"
|
||||
)
|
||||
|
||||
hits = exact or loose
|
||||
if len(hits) == 1:
|
||||
_, text = hits[0]
|
||||
if not self._click_row(dialog, text):
|
||||
raise RuntimeError(
|
||||
f"{text!r} was listed a moment ago but vanished on the "
|
||||
"second pass — retryable"
|
||||
)
|
||||
how = "" if exact else f" (matched loosely as {text!r})"
|
||||
return True, how
|
||||
|
||||
# Two-level dismissal: the sub-view has its own Cancel distinct from
|
||||
# the main dialog's.
|
||||
dialog.get_by_role("button", name="Cancel").first.click()
|
||||
if not hits:
|
||||
return False, f"not offered in the picker ({pages} page(s) scanned)"
|
||||
return False, (
|
||||
f"{len(hits)} versions match {version_name!r} "
|
||||
f"({', '.join(t for _, t in hits[:3])}…) — refusing to guess"
|
||||
)
|
||||
|
||||
def _goto_first_page(self, dialog) -> None:
|
||||
"""Back to page 1 via the numbered "1" anchor. Closing and
|
||||
reopening the sub-view does NOT reset Angular's paging state (it
|
||||
reopens wherever it was left), and First/Prev render in a
|
||||
mobile-only variant that a desktop viewport can never click."""
|
||||
pager = dialog.locator(self._PAGE_LINKS)
|
||||
for i in range(pager.count()):
|
||||
anchor = pager.nth(i)
|
||||
if anchor.is_visible() and (
|
||||
" ".join((anchor.text_content() or "").split()) == "1"
|
||||
):
|
||||
anchor.click()
|
||||
self._page.wait_for_timeout(400)
|
||||
return
|
||||
# no pagination rendered: a single-page list is already page 1
|
||||
|
||||
def _click_row(self, dialog, text: str) -> bool:
|
||||
"""Find the row again by its NORMALIZED text and click it by index.
|
||||
Playwright's has_text regex matches raw textContent — whose tabs and
|
||||
newlines a normalized capture will never equal — so the comparison
|
||||
happens in Python, and the click addresses a position."""
|
||||
self._goto_first_page(dialog)
|
||||
for _ in range(MAX_VERSION_PAGES):
|
||||
for i, row_text in enumerate(self._rows_on_page(dialog)):
|
||||
if row_text == text:
|
||||
dialog.locator(self._VERSION_ROWS).nth(i).click()
|
||||
return True
|
||||
if not self._has_next_page(dialog):
|
||||
return False
|
||||
self._visible(dialog, self._NEXT_PAGE).click()
|
||||
self._page.wait_for_timeout(400)
|
||||
return False
|
||||
|
||||
def _owned_button(self):
|
||||
"""The owned-game page replaces "Add To" with "In Collections" —
|
||||
positive evidence the collection already holds this game."""
|
||||
return self._page.get_by_role("button", name="In Collections")
|
||||
|
||||
def add_game(self, job: UploadJob) -> tuple[str, str]:
|
||||
self._ensure_logged_in()
|
||||
page = self._page
|
||||
# /boardgame/<id> redirects to the canonical slug for any subtype.
|
||||
self._goto(f"{BGG}/boardgame/{job.bgg_id}/")
|
||||
add_btn = page.get_by_role("button", name="Add To").first
|
||||
dialog = self._open_dialog(add_btn)
|
||||
# Content settles when the game-name heading replaces "Loading...".
|
||||
dialog.get_by_role(
|
||||
"heading", name=re.compile(re.escape(job.name), re.I)
|
||||
).wait_for(timeout=15_000)
|
||||
dialog.get_by_label("Own").check()
|
||||
add_btn = page.get_by_role("button", name="Add To")
|
||||
for _ in range(60): # the header hydrates late: poll for EITHER state
|
||||
if add_btn.count() and add_btn.first.is_visible():
|
||||
break
|
||||
owned = self._owned_button()
|
||||
if owned.count() and owned.first.is_visible():
|
||||
if job.second_copy:
|
||||
# the second-copy path goes through the In Collections
|
||||
# dialog, which this code has never driven live
|
||||
raise RuntimeError(
|
||||
"second copy of an owned game — the In Collections "
|
||||
"add-a-copy flow is unverified; add this copy by hand"
|
||||
)
|
||||
return (
|
||||
"already_present",
|
||||
"the site already lists this game as owned — a previous "
|
||||
"attempt likely landed without reaching the log",
|
||||
)
|
||||
page.wait_for_timeout(500)
|
||||
dialog = self._open_dialog(add_btn.first)
|
||||
# Wait for the form itself, NOT for a heading matching our stored
|
||||
# name: a match made through an ALTERNATE name (BGG 140509 is
|
||||
# "Dungeons & Dragons" to search, "Dragones Y Mazmorras" on the
|
||||
# page) would never show it. The /boardgame/<id>/ URL already
|
||||
# guarantees which game this is.
|
||||
dialog.get_by_role("checkbox", name="Own", exact=True).wait_for(timeout=15_000)
|
||||
# exact: "Own" is a substring of "Prev. Owned", and a loose label
|
||||
# match resolves to both checkboxes (strict-mode violation)
|
||||
dialog.get_by_role("checkbox", name="Own", exact=True).check()
|
||||
status, note = "added", ""
|
||||
if job.version_name and not self._select_version(dialog, job.version_name):
|
||||
if job.version_name:
|
||||
picked, why = self._select_version(dialog, job.version_name)
|
||||
if picked:
|
||||
note = why.strip()
|
||||
else:
|
||||
status = "added_no_version"
|
||||
note = f"version {job.version_name!r} not in picker; added without version"
|
||||
note = f"version {job.version_name!r}: {why}; added without version"
|
||||
dialog.get_by_role("button", name="Save").click()
|
||||
# The dialog is hidden after save, not removed from the DOM.
|
||||
try:
|
||||
dialog.wait_for(state="hidden", timeout=15_000)
|
||||
except self._timeout_error:
|
||||
# a slow hide is not a failed save: reload and ask the page.
|
||||
# Logging "failed" for a landed add would double-add on retry.
|
||||
self._goto(f"{BGG}/boardgame/{job.bgg_id}/")
|
||||
owned = self._owned_button()
|
||||
try:
|
||||
owned.first.wait_for(timeout=15_000)
|
||||
except self._timeout_error as err:
|
||||
raise RuntimeError(
|
||||
"the save dialog never closed and the page does not "
|
||||
"show the game as owned — the add may not have landed"
|
||||
) from err
|
||||
note = (note + "; " if note else "") + "save confirmed via page reload"
|
||||
return status, note
|
||||
|
||||
def update_entry(self, job: UploadJob) -> tuple[str, str]:
|
||||
"""Set the version on an EXISTING entry — strictly additive.
|
||||
|
||||
UNVERIFIED FLOW: the collection table can't target a collid directly,
|
||||
so this filters by game and opens the row's status link. With several
|
||||
copies of one game the wrong row could open — acceptable only while
|
||||
every 2018 entry is version-less; revisit after the first real run.
|
||||
Verified 2026-08-06. The collection table's version cell carries its
|
||||
own collid in an onclick and opens an inline editor whose radios
|
||||
carry VERSION IDS as their values, so both the copy and the edition
|
||||
are addressed exactly — no name matching, no dialog, no pagination.
|
||||
Clicking a radio fires CE_SaveData itself; there is no Save button.
|
||||
"""
|
||||
self._ensure_logged_in()
|
||||
page = self._page
|
||||
self._goto(
|
||||
f"{BGG}/collection/user/{self._username}?objectid={job.bgg_id}&own=1"
|
||||
)
|
||||
row = (
|
||||
page.get_by_role("row")
|
||||
.filter(has_text=re.compile(re.escape(job.name), re.I))
|
||||
.first
|
||||
)
|
||||
opener = row.get_by_role("link", name=re.compile("own", re.I)).first
|
||||
dialog = self._open_dialog(opener)
|
||||
if not self._select_version(dialog, job.version_name):
|
||||
# Never guess a version: close without touching the entry.
|
||||
dialog.get_by_role("button", name="Cancel").first.click()
|
||||
cell = page.locator(f'td.collection_version[onclick*="{job.collid}"]')
|
||||
if cell.count() == 0:
|
||||
raise RuntimeError(
|
||||
f"version {job.version_name!r} not in picker — entry left untouched"
|
||||
f"collid {job.collid} has no version cell on the collection "
|
||||
"page — the entry may have been removed; re-run diff"
|
||||
)
|
||||
cell.first.click()
|
||||
radio = page.locator(
|
||||
f'form[id^="form_version"] input[type="radio"][value="{job.version_id}"]'
|
||||
)
|
||||
try:
|
||||
radio.first.wait_for(timeout=15_000)
|
||||
except self._timeout_error as err:
|
||||
# the editor lists every version of the game: an absent id means
|
||||
# the wrong game's editor opened, or BGG retired that version
|
||||
page.keyboard.press("Escape")
|
||||
raise RuntimeError(
|
||||
f"version {job.version_id} not offered for collid "
|
||||
f"{job.collid} — entry left untouched"
|
||||
) from err
|
||||
radio.first.click() # fires CE_SaveData: no separate Save button
|
||||
want = _loose_version_text(job.version_name)
|
||||
for _ in range(40):
|
||||
settled = " ".join((cell.first.text_content() or "").split())
|
||||
if settled and "editing" not in settled.casefold():
|
||||
got = _loose_version_text(settled)
|
||||
if want and want not in got and got not in want:
|
||||
# the editor closed but re-rendered its OLD content:
|
||||
# the AJAX save failed server-side. "updated" here
|
||||
# would mark the job done forever without evidence.
|
||||
raise RuntimeError(
|
||||
f"the version cell settled on {settled!r}, not the "
|
||||
f"chosen {job.version_name!r} — the save did not "
|
||||
"land; safe to retry (same radio, same result)"
|
||||
)
|
||||
return "updated", ""
|
||||
page.wait_for_timeout(500)
|
||||
raise RuntimeError(
|
||||
"the version cell never left its editing state — the save may "
|
||||
"not have landed; re-run with --verify"
|
||||
)
|
||||
dialog.get_by_role("button", name="Save").click()
|
||||
dialog.wait_for(state="hidden", timeout=15_000)
|
||||
return "updated", "row-edit flow is unverified; confirm with --verify"
|
||||
|
||||
|
||||
def _process(
|
||||
@@ -468,7 +780,7 @@ def _process(
|
||||
suffix = f" — {note}" if note else ""
|
||||
typer.echo(f" {job.name}: {status}{suffix}")
|
||||
if status == "failed":
|
||||
kind = note.split(":", 1)[0] # exception type from _scrub format
|
||||
kind = note # IDENTICAL means the whole message, not the class
|
||||
consecutive = (kind, consecutive[1] + 1 if kind == consecutive[0] else 1)
|
||||
if consecutive[1] >= 3:
|
||||
typer.echo(
|
||||
@@ -603,7 +915,19 @@ def run_upload(
|
||||
to_update = _read_csv(cfg.to_update_path)
|
||||
log_rows = _read_csv(log_path)
|
||||
|
||||
jobs, skipped_done, skipped_failed, deferred = build_queue(
|
||||
# The queue is a snapshot; review decisions since the last diff win.
|
||||
stale = stale_jobs(to_add + to_update, read_matches(cfg.matches_path))
|
||||
if stale:
|
||||
to_add = [r for r in to_add if r.get("bgg_id") not in stale]
|
||||
to_update = [r for r in to_update if r.get("bgg_id") not in stale]
|
||||
typer.echo(
|
||||
f"Skipping {len(stale)} queued game(s) whose review decision "
|
||||
"changed since the last diff — re-run diff to refresh the queue:"
|
||||
)
|
||||
for bgg_id, why in stale.items():
|
||||
typer.echo(f" {bgg_id}: {why}")
|
||||
|
||||
jobs, skipped_done, skipped_failed = build_queue(
|
||||
to_add, to_update, log_rows, retry_failed=retry_failed
|
||||
)
|
||||
if limit is not None:
|
||||
@@ -615,12 +939,6 @@ def run_upload(
|
||||
f" (skipping {skipped_done} already done, {skipped_failed} "
|
||||
"previously failed — use --retry-failed)"
|
||||
)
|
||||
if deferred:
|
||||
typer.echo(
|
||||
f"{len(deferred)} version update(s) deferred: one update per game "
|
||||
"per run (the row-edit flow can't target a collid) — re-run "
|
||||
"upload after --verify confirms this batch."
|
||||
)
|
||||
|
||||
results: list[dict] = []
|
||||
if dry_run:
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
{
|
||||
"title_raw": "Catan",
|
||||
"confidence": "high",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Wingspan",
|
||||
@@ -10,31 +12,113 @@
|
||||
"publisher_hint": "Stonemaier Games",
|
||||
"year_hint": 2019,
|
||||
"language_hint": "English",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Wingspan: European Expansion",
|
||||
"confidence": "high",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Wingspan Europe",
|
||||
"confidence": "medium",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Café International",
|
||||
"confidence": "high",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Citadels",
|
||||
"confidence": "high",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Blorvath: Quest of the Zzyzx",
|
||||
"confidence": "low",
|
||||
"source_photos": ["hand-typed-test-list"]
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "CIVILIZATION Game of the Heroic Age - The Dawn of History 8000 BC to 250 BC",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "The Avalon Hill Game Company, Baltimore, Maryland",
|
||||
"edition_hint": "Bookcase Game",
|
||||
"language_hint": "English",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "ADVANCED CIVILIZATION Game Expansion of the Heroic Age - Featuring New Civilization, Commodity, and Calamity Cards",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "The Avalon Hill Game Company, Baltimore, Maryland",
|
||||
"edition_hint": "Bookcase Game",
|
||||
"language_hint": "English",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "SORCERER The Game of Magical Conflict",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Simulations Publications Incorporated (SPI)",
|
||||
"edition_hint": "Designer's Edition",
|
||||
"language_hint": "English",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "STARFORCE ALPHA CENTAURI Interstellar Conflict in the 25th Century",
|
||||
"confidence": "high",
|
||||
"publisher_hint": "Simulations Publications Incorporated (SPI)",
|
||||
"edition_hint": "Designer's Edition",
|
||||
"language_hint": "English",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "FLAT TOP",
|
||||
"confidence": "high",
|
||||
"year_hint": 1942,
|
||||
"language_hint": "English",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "ALICE IS MISSING: A SILENT ROLE PLAYING GAME",
|
||||
"confidence": "high",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "Dungeon!",
|
||||
"confidence": "high",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title_raw": "WIZ-WAR",
|
||||
"confidence": "high",
|
||||
"source_photos": [
|
||||
"hand-typed-test-list"
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
This cache contains hand-written stub XML, not real BGG responses. Data resolved from it must not be uploaded.
|
||||
@@ -1 +0,0 @@
|
||||
<items total="1"><item type="boardgame" id="207830"><name type="primary" value="5-Minute Dungeon"/><yearpublished value="2017"/></item></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -0,0 +1 @@
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +1,9 @@
|
||||
<items total="1"><item type="boardgameexpansion" id="177"><name type="primary" value="Advanced Civilization"/><yearpublished value="1991"/></item></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="2" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> <item type="boardgame" id="177">
|
||||
<name type="primary" value="Advanced Civilization"/>
|
||||
<yearpublished value="1991" />
|
||||
</item>
|
||||
<item type="boardgameexpansion" id="177">
|
||||
<name type="primary" value="Advanced Civilization"/>
|
||||
<yearpublished value="1991" />
|
||||
</item>
|
||||
</items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="1"><item type="rpgitem" id="400001"><name type="primary" value="Alice Is Missing: A Silent Role Playing Game"/><yearpublished value="2020"/></item></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="1"><item type="rpgitem" id="400002"><name type="primary" value="Alice Is Missing: Silent Falls Expansion"/><yearpublished value="2023"/></item></items>
|
||||
@@ -0,0 +1 @@
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -0,0 +1 @@
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1,9 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="2" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> <item type="rpgitem" id="311654">
|
||||
<name type="primary" value="Alice is Missing"/>
|
||||
<yearpublished value="2020" />
|
||||
</item>
|
||||
<item type="rpgitem" id="380459">
|
||||
<name type="primary" value="Alice is Missing: Silent Falls"/>
|
||||
<yearpublished value="2023" />
|
||||
</item>
|
||||
</items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="1"><item type="boardgame" id="26"><name type="primary" value="Age of Renaissance"/><yearpublished value="1996"/></item></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -0,0 +1 @@
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -0,0 +1 @@
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="1"><item type="boardgame" id="240"><name type="primary" value="Britannia"/><yearpublished value="1986"/></item></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="1"><item type="boardgame" id="235096"><name type="primary" value="Cat Crimes"/><yearpublished value="2017"/></item></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +0,0 @@
|
||||
<items total="0"></items>
|
||||
@@ -1 +1 @@
|
||||
<items total="0"></items>
|
||||
<?xml version="1.0" encoding="utf-8"?><items total="0" termsofuse="https://boardgamegeek.com/xmlapi/termsofuse"> </items>
|
||||