Initial commit: spec, Claude Code setup, and project docs

Design spec for the bggpipe shelf-to-BGG pipeline, CLAUDE.md and
bgg-api skill capturing BGG API constraints, ruff format-on-edit
hook, README, LICENSE, and .gitignore.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Eric Wagoner
2026-08-01 12:11:14 -04:00
commit 6bd4222fc1
7 changed files with 330 additions and 0 deletions
+17
View File
@@ -0,0 +1,17 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "Write|Edit",
"hooks": [
{
"type": "command",
"command": "jq -r '.tool_input.file_path // .tool_response.filePath // empty' | { read -r f; case \"$f\" in *.py) uvx ruff format \"$f\" && uvx ruff check --fix \"$f\";; esac; } 2>/dev/null || true",
"timeout": 60,
"statusMessage": "ruff format + check"
}
]
}
]
}
}
+52
View File
@@ -0,0 +1,52 @@
---
name: bgg-api
description: Reference for BoardGameGeek's XML API2 and website automation — endpoints, 202 queueing, rate limits, collection quirks, upload etiquette. Use when writing or debugging any code that talks to boardgamegeek.com (resolve, diff, or upload stages).
---
# BoardGameGeek API & site automation reference
## Endpoints (XML API2 — the only sanctioned read API)
- Search: `https://boardgamegeek.com/xmlapi2/search?query=<title>&type=boardgame,boardgameexpansion`
- Thing (details/stats): `https://boardgamegeek.com/xmlapi2/thing?id=<id1,id2,...>&stats=1` — accepts comma-separated IDs; batch (~20) to reduce request count.
- Thing versions: `https://boardgamegeek.com/xmlapi2/thing?id=<id>&versions=1` — lists every published edition/version of a game, each with its own version id, name, publisher, year, and language. Used to match photo edition cues to a concrete version.
- Collection: `https://boardgamegeek.com/xmlapi2/collection?username=<user>&own=1` — add `&version=1` to include version info on collection items.
All responses are XML. There is no JSON API and **no write API** — writes automate the website via Playwright with a logged-in session.
## HTTP 202 queueing (collection endpoint)
The first `/collection` call typically returns **HTTP 202** with a "please retry" message: BGG queues the export and serves it on a later request. Retry schedule: **2s, 5s, 10s, 30s, give up after ~5 tries** with a clear error message. Treat 202 as normal flow, not an error.
## Collection endpoint quirks
- The default subtype **excludes expansions**. Make a second call with `&subtype=boardgameexpansion` and merge, or owned expansions will be invisible to `diff` and re-uploaded.
- `own=1` filters to owned items; other statuses (wishlist, previously owned) exist and must not be counted as owned.
- Each collection item has a **`collid`** (unique per copy) alongside `objectid` (the game). Owning two editions of one game = two items, same `objectid`, different `collid`s. Diff on (objectid, version) pairs, not bare objectid, when versions are known.
- With `&version=1`, items that have a version set include it; items without one simply don't — version-less entries are legal and common.
## Rate limiting
- **≤1 request every 2 seconds** to any BGG endpoint (API or website). Jittered exponential backoff on 429/503.
- Cache every API response on disk under `data/bgg_cache/` keyed by query/ID; check cache before hitting the network so re-runs are free.
- Upload stage is deliberately slower: **24 s randomized delay** between games — this is a real account on a community site.
## Search & matching heuristics
1. Exact normalized-name match → strong candidate.
2. Fuzzy match: `rapidfuzz` `token_sort_ratio ≥ 90` → good candidate.
3. Ties: fetch `/thing?stats=1` for top ~5 candidates; prefer higher owned-count / better BGG rank (well-known game beats obscure duplicate of the same name).
- Search results include `boardgameexpansion` as a distinct `type` — keep expansions but tag them so review catches base/expansion confusion.
- Classify every result: `auto` (single confident match) / `ambiguous` (store all candidates) / `unmatched`. When in doubt between editions or base-vs-expansion, choose `ambiguous`.
## Website automation (upload stage)
- Login with `BGG_USERNAME` / `BGG_PASSWORD` env vars; persist Playwright storage state locally (gitignored) so repeat runs skip login. Never write credentials to disk, logs, or error messages.
- Per game: navigate to the game page → "Add to Collection" flow → status **Owned** → if a version_id is known, set it in the collection item's version picker → save. Never guess a version — omit it when unknown. The version picker is the most fragile part of the UI: walk it manually once and document the selectors before automating.
- A second copy of an owned game must be a NEW collection entry (new collid), not an edit of the existing item.
- Expect UI fragility: wrap each game in its own try/except, log the failure to `upload_log.csv`, and continue. `--retry-failed` re-attempts failures; `--dry-run` logs without touching the site.
- Idempotency: skip IDs already logged `added`; `--verify` re-fetches the collection to confirm.
## Policy note
BGG changed API access policies in 2025 and broke older community tools. Do not use undocumented endpoints, scraped JSON blobs, or third-party mirrors — only XML API2 and the public website.
+23
View File
@@ -0,0 +1,23 @@
# Secrets & credential-adjacent state
.env
*.storage_state.json
playwright/.auth/
storage_state.json
# Local inputs & cache (CSV/JSON artifacts in data/ ARE committed)
photos/
data/bgg_cache/
# Python
__pycache__/
*.pyc
.venv/
.pytest_cache/
.ruff_cache/
# Personal Claude Code files
CLAUDE.local.md
.claude/settings.local.json
# macOS
.DS_Store
+33
View File
@@ -0,0 +1,33 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project
`bggpipe` — a resumable, idempotent CLI pipeline: shelf photos → Claude vision title + edition-cue extraction → BoardGameGeek ID and version matching → human review → collection diff → upload to BGG via Playwright → metadata enrichment (`games.json`). Full design: @bgg-shelf-pipeline-spec.md. **The repo is greenfield — no code exists yet.** Follow the spec's "Suggested Build Order" when scaffolding; re-run `/init` once code exists.
## Stack (decided, not yet scaffolded)
- Python 3.12+, deps via **uv** (`uv add`, `uv run`), CLI framework **typer**, tests **pytest**, lint/format **ruff**.
- Browser automation: **Playwright** (not Selenium). Needs `uv run playwright install chromium` after install.
- Vision: Anthropic API, latest Sonnet model.
## Hard rules (from spec — never violate)
- **≤1 request every 2 seconds** to any BGG endpoint; jittered backoff on 429/503. Upload stage: 24 s randomized delay between games.
- **Credentials never touch disk or logs.** `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD` come from env vars only. Playwright storage state is credential-adjacent — keep it gitignored.
- **Every stage is idempotent and resumable** — killing mid-run and restarting must lose no work; re-runs skip already-processed items.
- Use only the XML API2 and the public website — no undocumented BGG endpoints (BGG tightened access policies in 2025).
- BGG has **no write API**: writes drive the real website with a logged-in Playwright session.
## Domain gotchas
- Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan).
- Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry.
- Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance.
- Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift.
## Git
- Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social`), **not GitHub**`gh` CLI does not work here.
- Commit `data/matches.csv`, `data/to_add.csv`, `data/upload_log.csv`, `data/titles.json`, `data/games.json`. Never commit `data/bgg_cache/`, `photos/`, Playwright storage state, or `.env`.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Eric Wagoner
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+44
View File
@@ -0,0 +1,44 @@
# bggpipe — Shelf-to-BoardGameGeek Collection Pipeline
Photograph your board game shelves. End up with your whole collection — including which *edition* of each game you own — cataloged on [BoardGameGeek](https://boardgamegeek.com).
```
photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
→ [3 review] → matches.csv (approved) → [4 diff] → to_add.csv
→ [5 upload] → upload_log.csv
→ [6 enrich] → games.json
```
> **Status: pre-release.** The design is complete ([full spec](bgg-shelf-pipeline-spec.md)); the code is being built. Nothing below works yet.
## Why this exists
BGG has no bulk import and no write API. Cataloging a few hundred games by hand means hours of searching, clicking, and second-guessing which of five editions you own. This pipeline replaces that with: take photos, run a command, resolve a handful of ambiguous matches in a review step, done.
## How it works
1. **extract** — Shelf photos go to the Anthropic API (Claude vision), which reads game titles off spines and boxes along with edition cues: publisher, edition wording, print year, language.
2. **resolve** — Titles are matched to BGG game IDs via the [XML API2](https://boardgamegeek.com/wiki/page/BGG_XML_API2) (exact + fuzzy matching, popularity tiebreaks), then edition cues are matched against BGG's version list for each game. Anything uncertain is flagged rather than guessed.
3. **review** — A local review step for ambiguous matches: pick the right game/version, or leave the version blank. Wrong guesses never reach your collection.
4. **diff** — Your existing BGG collection is fetched and compared, per copy (owning one edition of a game doesn't hide a second edition you also own).
5. **upload** — A Playwright browser session logs into your BGG account and adds each game — with its version, when known — politely and slowly, with dry-run mode, per-game logging, and resumability.
6. **enrich** — Full metadata for every game (designers, player counts, weight, rank, mechanics, artwork URLs, version details) lands in `data/games.json`, ready to power whatever you build next.
Every stage is idempotent and resumable: kill it mid-run, restart, lose nothing. All artifacts are flat CSV/JSON files you can inspect and edit.
## Requirements
- macOS or Linux, Python 3.12+, [uv](https://docs.astral.sh/uv/)
- An [Anthropic API key](https://console.anthropic.com/) (vision extraction)
- A BoardGameGeek account
- Playwright Chromium: `uv run playwright install chromium`
Secrets come from environment variables only — `ANTHROPIC_API_KEY`, `BGG_USERNAME`, `BGG_PASSWORD` — and are never written to disk or logs.
## A note on being a good BGG citizen
This tool is **not affiliated with or supported by BoardGameGeek**. It uses only the sanctioned XML API2 for reads and drives the regular website for writes, deliberately slowly (one request every couple of seconds, slower for uploads). Please keep it that way: BGG is a community resource running on community goodwill. You are responsible for your own account — review the dry-run output before a real upload.
## License
MIT — see [LICENSE](LICENSE).
+140
View File
@@ -0,0 +1,140 @@
# Shelf-to-BGG Collection Pipeline — Build Spec
## Goal
Build a command-line pipeline that takes photos of my board game shelves and ends with every recognized game added to my BoardGameGeek collection. The pipeline must be resumable, idempotent, and leave a human-reviewable audit trail at every stage.
Beyond the bare game, capture **which edition/version I own** wherever the photos allow it — many of my games exist in multiple editions, and in some cases I own more than one edition of the same game (each is a separate collection entry). Also capture **all critical data about each game itself** (see Stage 6 — enrich): the BGG `/thing` metadata is cheap to fetch and will seed the future frontend. Purchase provenance (where/when acquired, price paid) is explicitly NOT tracked — I don't have that data.
**Out of scope (for now):** the web frontend for displaying the collection. That's a follow-up project. But keep the data artifacts (see Data Model) clean and structured so they can seed it later.
## Constraints & Context
- BGG has **no write API**. Reads go through the XML API2 (`https://boardgamegeek.com/xmlapi2/`); writes must automate the website itself with a real login session.
- BGG's XML API queues collection requests: a first call may return HTTP 202 ("try again"). Retry with backoff.
- BGG will throttle aggressive clients. Target ≤1 request every 2 seconds to any BGG endpoint, with jittered backoff on 429/503.
- BGG changed API access policies in 2025; some older community tools broke. Don't depend on undocumented endpoints beyond XML API2 and the public website.
- Vision extraction uses the **Anthropic API** (Claude with vision). Assume `ANTHROPIC_API_KEY` in the environment.
- Runs on macOS. Prefer **Python 3.12+** with `uv` for dependency management. Browser automation via **Playwright** (not Selenium).
## Pipeline Overview
Five stages, each a separate subcommand of one CLI (suggest `bggpipe`):
```
photos/ → [1 extract] → titles.json → [2 resolve] → matches.csv
→ [3 review] → matches.csv (approved) → [4 diff] → to_add.csv
→ [5 upload] → upload_log.csv
→ [6 enrich] → games.json (runs any time after review)
```
Each stage reads the previous stage's artifact and writes its own. Re-running a stage must be safe (skip already-processed items).
### Stage 1 — `extract`: Vision title extraction
- Input: a directory of shelf photos (JPEG/PNG/HEIC — convert HEIC to JPEG first via `sips` or Pillow-HEIF).
- For each photo, send to the Anthropic API (model: latest Sonnet) with a prompt that asks for:
- Every board game title visible (spines and face-out boxes), transcribed as printed.
- A confidence level per title (`high` / `medium` / `low`).
- **Edition/version cues**, each if legible: publisher name or logo, edition wording ("2nd Edition", "Deluxe", "Big Box", anniversary marks), copyright/print year, language, and distinctive box-art notes (colorway, artwork style). These drive version matching in Stage 2.
- If the SAME title appears more than once in the photos with visibly different boxes, report each as a separate entry — I own multiple editions of some games.
- Downscale images so the long edge is ≤1568px before sending (API sweet spot; keeps tokens down).
- Prompt for structured JSON output; parse defensively (strip code fences).
- Handle overlap: the same game may appear in two photos. Dedupe by normalized title (casefold, strip punctuation/articles) but **keep provenance** — record which photo(s) each title came from.
- Output: `titles.json` — list of `{title_raw, title_normalized, confidence, publisher_hint, edition_hint, year_hint, language_hint, art_notes, source_photos[]}`.
- Dedupe nuance: identical normalized titles from different photos collapse to one entry ONLY if their edition cues don't conflict; conflicting cues (different publisher/edition text) stay as separate entries.
- Support `--only <photo>` to re-run a single photo (e.g., after retaking a blurry shot).
### Stage 2 — `resolve`: Match titles to BGG IDs
- For each title, query `https://boardgamegeek.com/xmlapi2/search?query=<title>&type=boardgame,boardgameexpansion`.
- Scoring heuristic for candidates:
1. Exact normalized-name match → strong.
2. Fuzzy match (e.g., `rapidfuzz` token_sort_ratio ≥ 90) → good.
3. If multiple candidates tie, fetch `/thing?id=...&stats=1` for the top ~5 and prefer higher-owned/higher-ranked entries (obscure duplicates lose to the well-known game of the same name).
- Classify each result:
- `auto` — single confident match, no review needed.
- `ambiguous` — multiple plausible matches; store all candidates.
- `unmatched` — nothing plausible found.
- Expansions: BGG returns `boardgameexpansion` as a distinct type. Keep them — I own expansions and want them in the collection — but tag them so review can catch base-game/expansion confusion (a spine reading "Wingspan Europe" must not match base Wingspan).
- Cache all BGG responses on disk (keyed by query/ID) so re-runs don't re-hit the API.
- **Version resolution**: once a game ID is settled (auto or approved), fetch `/thing?id=<id>&versions=1` and score the version list against the extraction's edition cues (publisher, year, language, edition wording). Same three-way classification: a single clear winner is `version_auto`; multiple plausible → `version_ambiguous` (goes to review); no cues at all → `version_unknown` (acceptable — BGG allows collection entries with no version set, and guessing wrong is worse than leaving it blank).
- Output: `matches.csv` with columns: `title_raw, bgg_id, bgg_name, year, type, match_status (auto|ambiguous|unmatched|approved|rejected), version_id, version_name, version_status (version_auto|version_ambiguous|version_unknown|version_approved), candidates_json, version_candidates_json, source_photos`.
### Stage 3 — `review`: Human review of ambiguous/unmatched
- A minimal local review flow. A TUI is fine (e.g., `rich`/`textual`), or a tiny localhost web page — builder's choice, but keep it dependency-light.
- For each `ambiguous` item: show the raw title, source photo filename, and candidate list (name, year, type, BGG rank, owned count) → pick one, skip, or reject.
- For each `unmatched` item: allow manual BGG ID entry or a free-text re-search.
- For each `version_ambiguous` item: show the version candidates (version name, publisher, year, language) next to the edition cues from the photo → pick one, or mark `version_unknown`. Keep this pass optional/skippable — version review shouldn't block getting games uploaded.
- Show the source photo (or a crop) alongside if easy; otherwise the filename is acceptable.
- Decisions update `matches.csv` in place (`approved` with chosen `bgg_id`, or `rejected`).
- Must be resumable — quitting mid-review loses nothing.
### Stage 4 — `diff`: Compare against existing BGG collection
- Fetch my current collection: `https://boardgamegeek.com/xmlapi2/collection?username=<me>&own=1` (handle the 202-retry queue; also pass `&subtype=boardgameexpansion` in a second call — the collection endpoint excludes expansions from the default subtype).
- Output `to_add.csv`: approved/auto matches whose IDs are **not** already in the collection.
- **Multiple editions of the same game**: collection items are identified by `collid` (one per copy), not just `objectid`. If matches contain two entries for the same `bgg_id` with different `version_id`s, both belong in the collection as separate entries. Diff logic: a (bgg_id, version_id) pair is "already owned" only if a collection item matches both; a bare bgg_id with `version_unknown` is "already owned" if any copy of that game exists.
- Print a summary: N recognized, N already owned, N to add (including second editions), N rejected/unmatched.
### Stage 5 — `upload`: Add games via Playwright
- Log in to boardgamegeek.com with credentials from env vars (`BGG_USERNAME`, `BGG_PASSWORD`). Never write credentials to disk or logs. Persist the browser session/storage state locally so repeat runs don't re-login.
- For each row in `to_add.csv`: navigate to the game page, use the "Add to Collection" flow, set status **Owned**, and — when a `version_id` is present — set the specific version in the collection item's version picker before saving. Manually walk this flow once and document the selectors before automating; the version UI is the most fragile part.
- Adding a second copy of an already-owned game must create a NEW collection entry, not edit the existing one.
- Log every attempt to `upload_log.csv`: `bgg_id, name, status (added|already_present|failed), timestamp, error`.
- Idempotent: skip IDs already logged as `added`; re-verify against a fresh collection fetch on `--verify`.
- Deliberately slow: 24 s randomized delay between games. This is a real account on a community site — behave like a polite human.
- Expect UI fragility: fail gracefully per-game and continue; a `--retry-failed` flag re-attempts failures.
- Dry-run mode (`--dry-run`) that logs what it *would* add without touching the site.
### Stage 6 — `enrich`: Capture full game metadata
- For every approved/auto game ID, fetch `/thing?id=<batched,ids>&stats=1` (comma-separated batches of ~20) and store the critical data: name, year published, designers, artists, publishers, min/max players, community best-player-counts, playtime, min age, weight/complexity, BGG rank + rating, categories, mechanics, description, image + thumbnail URLs — plus the chosen version's details (version name, publisher, year, language) when known.
- Output: `data/games.json`, keyed by bgg_id (+ version_id where set). This file is the seed for the future web frontend, so keep it complete and stable.
- Idempotent and cheap: everything comes through the existing cache; `--refresh` forces a re-fetch (ranks and ratings drift over time).
## Data Model
All artifacts are flat files in a `data/` directory — human-readable, git-friendly, and reusable by the future frontend:
- `titles.json` — extraction output (stage 1)
- `bgg_cache/` — cached XML API responses
- `matches.csv` — the master matching table (stages 23)
- `to_add.csv` — upload queue (stage 4)
- `upload_log.csv` — audit trail (stage 5)
- `games.json` — full game + version metadata (stage 6); seed data for the future frontend
## Configuration
`config.toml` (or env) for: BGG username, photo directory, model name, rate-limit settings. Secrets only via env vars.
## Error Handling & Edge Cases
- **202 queue** on `/collection`: retry with backoff (2s, 5s, 10s, 30s; give up after ~5 tries with a clear message).
- **HEIC photos** from iPhone: convert transparently.
- **Duplicate copies**: a title appearing in multiple photos with consistent edition cues is one game (dedupe). But genuinely distinct editions of the same game ARE in scope — they stay separate entries end-to-end and become separate BGG collection entries. Identical duplicate copies of the same edition are out of scope (assume dedupe).
- **Non-game items** on shelves (books, card sleeves, storage boxes): the vision prompt should be instructed to include only board/card games; anything that slips through will fail resolution and land in review.
- **Special characters** in titles (é, colons, ampersands): normalize consistently on both sides of the match.
- **Base game vs. expansion vs. new edition**: the most common failure mode. Bias toward surfacing these as `ambiguous` rather than auto-matching.
## Acceptance Criteria
1. Given a directory of shelf photos, `bggpipe extract && bggpipe resolve` produces `matches.csv` with ≥90% of clearly legible titles auto-matched correctly.
2. Review flow lets me resolve every ambiguous/unmatched item without editing CSVs by hand.
3. `bggpipe upload --dry-run` shows exactly what would be added; the real run adds them, and a subsequent `bggpipe diff` reports zero remaining.
4. Killing any stage mid-run and restarting loses no work.
5. No BGG endpoint is hit faster than the rate limits above.
6. Credentials never appear in any file, log, or error message.
7. Where photos show legible edition cues, the matched version survives to the BGG collection entry; where they don't, the entry is added version-less rather than with a guessed version.
8. A game I own in two editions ends up as two distinct collection entries, and `games.json` contains full metadata for every game in the collection.
## Suggested Build Order
1. Scaffold CLI + config + BGG API client (with caching, 202 handling, rate limiting). Test against my real username read-only.
2. Stage 2 resolve with a hand-typed test title list — validates matching before spending vision tokens.
3. Stage 1 extract against 23 test photos; iterate on the prompt.
4. Stage 3 review + Stage 4 diff.
5. Stage 5 upload — test with `--dry-run`, then a single game, then small batches. Verify the version-picker flow manually on one game first.
6. Stage 6 enrich — mostly free once the cached API client exists; build alongside stage 4.