Eric Wagoner
3543005236
compute_diff is a pure function over matches + collection items: new games land in to_add.csv (carrying a confident version when matching produced one); owned version-less entries with version_auto/approved matches produce additive to_update.csv rows keyed by collid, consuming distinct collids when photos show two editions; entries that already carry a version are never touched — disagreements are reported in the summary. Unseen collection entries are listed informationally. Live API mode activates when BGG_API_TOKEN + username exist; otherwise the two hand-pulled snapshot XMLs (real 2018 collection, 79 unique copies after collid dedupe) are used, and they double as parsing fixtures in tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
70 lines
6.2 KiB
Markdown
70 lines
6.2 KiB
Markdown
---
|
||
name: bgg-api
|
||
description: Reference for BoardGameGeek's XML API2 and website automation — endpoints, 202 queueing, rate limits, collection quirks, upload etiquette. Use when writing or debugging any code that talks to boardgamegeek.com (resolve, diff, or upload stages).
|
||
---
|
||
|
||
# BoardGameGeek API & site automation reference
|
||
|
||
## Authentication (required since 2025)
|
||
|
||
Every XML API request must carry `Authorization: Bearer <token>` or BGG
|
||
returns **401 Unauthorized**. Tokens come from a registered application:
|
||
create one at `https://boardgamegeek.com/applications` (non-commercial
|
||
license is free; approval can take a week or more), then generate a token
|
||
under "Tokens". `bggpipe` reads it from the `BGG_API_TOKEN` env var — never
|
||
put it in config.toml, code, or logs. Requests must go to
|
||
`boardgamegeek.com` **without** a leading `www` or the token is ignored.
|
||
Exception: downloading your own collection while logged in on the website
|
||
needs no registration — relevant to the Playwright stages, not the API
|
||
client. Usage is monitored per-application at `/applications` → "Usage".
|
||
Open-source note: each user of this tool registers their OWN application
|
||
and supplies their own token — never ship or share a token in the repo.
|
||
Future frontend note: public-facing apps must display the "Powered by
|
||
BGG" logo linking back to boardgamegeek.com.
|
||
|
||
## Endpoints (XML API2 — the only sanctioned read API)
|
||
|
||
- Search: `https://boardgamegeek.com/xmlapi2/search?query=<title>&type=boardgame,boardgameexpansion`
|
||
- Thing (details/stats): `https://boardgamegeek.com/xmlapi2/thing?id=<id1,id2,...>&stats=1` — accepts comma-separated IDs; batch (~20) to reduce request count.
|
||
- Thing versions: `https://boardgamegeek.com/xmlapi2/thing?id=<id>&versions=1` — lists every published edition/version of a game, each with its own version id, name, publisher, year, and language. Used to match photo edition cues to a concrete version.
|
||
- Collection: `https://boardgamegeek.com/xmlapi2/collection?username=<user>&own=1` — add `&version=1` to include version info on collection items.
|
||
|
||
All responses are XML. There is no JSON API and **no write API** — writes automate the website via Playwright with a logged-in session.
|
||
|
||
## HTTP 202 queueing (collection endpoint)
|
||
|
||
The first `/collection` call typically returns **HTTP 202** with a "please retry" message: BGG queues the export and serves it on a later request. Retry schedule: **2s, 5s, 10s, 30s, give up after ~5 tries** with a clear error message. Treat 202 as normal flow, not an error.
|
||
|
||
## Collection endpoint quirks
|
||
|
||
- The default subtype **excludes expansions**. Make a second call with `&subtype=boardgameexpansion` and merge, or owned expansions will be invisible to `diff` and re-uploaded.
|
||
- `own=1` filters to owned items; other statuses (wishlist, previously owned) exist and must not be counted as owned.
|
||
- Each collection item has a **`collid`** (unique per copy) alongside `objectid` (the game). Owning two editions of one game = two items, same `objectid`, different `collid`s. Diff on (objectid, version) pairs, not bare objectid, when versions are known.
|
||
- With `&version=1`, items that have a version set include it; items without one simply don't — version-less entries are legal and common.
|
||
|
||
## Rate limiting
|
||
|
||
- **≤1 request every 2 seconds** to any BGG endpoint (API or website). Jittered exponential backoff on 429/503.
|
||
- Cache every API response on disk under `data/bgg_cache/` keyed by query/ID; check cache before hitting the network so re-runs are free.
|
||
- Upload stage is deliberately slower: **2–4 s randomized delay** between games — this is a real account on a community site.
|
||
|
||
## Search & matching heuristics
|
||
|
||
1. Exact normalized-name match → strong candidate.
|
||
2. Fuzzy match: `rapidfuzz` `token_sort_ratio ≥ 90` → good candidate.
|
||
3. Ties: fetch `/thing?stats=1` for top ~5 candidates; prefer higher owned-count / better BGG rank (well-known game beats obscure duplicate of the same name).
|
||
- Search results include `boardgameexpansion` as a distinct `type` — keep expansions but tag them so review catches base/expansion confusion.
|
||
- Classify every result: `auto` (single confident match) / `ambiguous` (store all candidates) / `unmatched`. When in doubt between editions or base-vs-expansion, choose `ambiguous`.
|
||
|
||
## Website automation (upload stage)
|
||
|
||
- Login with `BGG_USERNAME` / `BGG_PASSWORD` env vars; persist Playwright storage state locally (gitignored) so repeat runs skip login. Never write credentials to disk, logs, or error messages.
|
||
- Per game: navigate to the game page → "Add to Collection" flow → status **Owned** → if a version_id is known, set it in the collection item's version picker → save. Never guess a version — omit it when unknown. The flow has been walked and documented: see `docs/bgg-upload-flow.md` for the dialog structure, version-picker behavior (paginated, no search — match by canonical version NAME from the XML API), and observed automation gotchas (stale elements, hidden-not-removed dialogs, hydration races).
|
||
- A second copy of an owned game must be a NEW collection entry (new collid), not an edit of the existing item. Conversely, version UPGRADES from to_update.csv must edit the EXISTING item (same collid) — additive only: fill the empty version field, change nothing else, and skip any entry that already has a version.
|
||
- Expect UI fragility: wrap each game in its own try/except, log the failure to `upload_log.csv`, and continue. `--retry-failed` re-attempts failures; `--dry-run` logs without touching the site.
|
||
- Idempotency: skip IDs already logged `added`; `--verify` re-fetches the collection to confirm.
|
||
|
||
## Policy note
|
||
|
||
BGG's 2025 policy (version 2025-07-02, boardgamegeek.com/using_the_xml_api) locked the API behind registered applications and Bearer tokens, breaking older community tools. Do not use undocumented endpoints, scraped JSON blobs, or third-party mirrors — only the authenticated XML API2 and the public website. Third-party services that proxy BGG data to other applications are explicitly prohibited. The policy asks for server-side requests, aggressive caching, and minimal request counts — the cache-first client design is mandatory, not optional. Policies can change at any time; changes are announced in BGG's Geek Tools News forum.
|