Files

T

Eric Wagoner 74fc4fe847 RPGs become local library citizens: identified, enriched, never uploaded

When the board-game search (and truncation heads) runs dry, resolve
falls back to type=rpgitem — the geekdo database is shared, so the same
API, token, cache, and classification machinery apply. Matched rpgitems
flow through review and enrich normally but diff routes them to a
local_only bucket, structurally outside to_add/to_update: their
collections live on RPGGeek, beyond this pipeline's write scope. The
library page gains an All/Board games/RPGs filter and an "RPG · local
only" badge; the catalog tags them too. Fixture generators write blanket
empty rpgitem stubs for every known query (the fallback fires for every
unmatched title) with real synthetic entries for Alice Is Missing.

Data: both Alice rows re-resolved from unmatched to auto rpgitem
matches. First diff since the audit reworks also lands their real-data
consequences: Dungeon! gains its TSR edition update on a versionless
copy the old claim ordering missed, to_add rows carry unioned reshoot
provenance, and the Herbaceous typo row's survivor is now the
correctly-spelled title.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-08-02 19:01:09 -04:00

6.5 KiB

Raw Permalink Blame History

name, description

name	description
bgg-api	Reference for BoardGameGeek's XML API2 and website automation — endpoints, 202 queueing, rate limits, collection quirks, upload etiquette. Use when writing or debugging any code that talks to boardgamegeek.com (resolve, diff, or upload stages).

BoardGameGeek API & site automation reference

Authentication (required since 2025)

Every XML API request must carry Authorization: Bearer <token> or BGG returns 401 Unauthorized. Tokens come from a registered application: create one at https://boardgamegeek.com/applications (non-commercial license is free; approval can take a week or more), then generate a token under "Tokens". bggpipe reads it from the BGG_API_TOKEN env var — never put it in config.toml, code, or logs. Requests must go to boardgamegeek.com without a leading www or the token is ignored. Exception: downloading your own collection while logged in on the website needs no registration — relevant to the Playwright stages, not the API client. Usage is monitored per-application at /applications → "Usage". Open-source note: each user of this tool registers their OWN application and supplies their own token — never ship or share a token in the repo. Future frontend note: public-facing apps must display the "Powered by BGG" logo linking back to boardgamegeek.com.

Endpoints (XML API2 — the only sanctioned read API)

Search: https://boardgamegeek.com/xmlapi2/search?query=<title>&type=boardgame,boardgameexpansion
RPG search: same endpoint with type=rpgitem — the geekdo database is shared across BGG/RPGGeek, so the same API and token serve RPG products. bggpipe uses this as a fallback for titles absent from the board-game types; matched rpgitems are LOCAL-ONLY (enriched, browsable, never uploaded — RPG collections live on rpggeek.com, outside this pipeline's write scope).
Thing (details/stats): https://boardgamegeek.com/xmlapi2/thing?id=<id1,id2,...>&stats=1 — accepts comma-separated IDs; batch (~20) to reduce request count.
Thing versions: https://boardgamegeek.com/xmlapi2/thing?id=<id>&versions=1 — lists every published edition/version of a game, each with its own version id, name, publisher, year, and language. Used to match photo edition cues to a concrete version.
Collection: https://boardgamegeek.com/xmlapi2/collection?username=<user>&own=1 — add &version=1 to include version info on collection items.

All responses are XML. There is no JSON API and no write API — writes automate the website via Playwright with a logged-in session.

HTTP 202 queueing (collection endpoint)

The first /collection call typically returns HTTP 202 with a "please retry" message: BGG queues the export and serves it on a later request. Retry schedule: 2s, 5s, 10s, 30s, give up after ~5 tries with a clear error message. Treat 202 as normal flow, not an error.

Collection endpoint quirks

The default subtype excludes expansions. Make a second call with &subtype=boardgameexpansion and merge, or owned expansions will be invisible to diff and re-uploaded.
own=1 filters to owned items; other statuses (wishlist, previously owned) exist and must not be counted as owned.
Each collection item has a collid (unique per copy) alongside objectid (the game). Owning two editions of one game = two items, same objectid, different collids. Diff on (objectid, version) pairs, not bare objectid, when versions are known.
With &version=1, items that have a version set include it; items without one simply don't — version-less entries are legal and common.

Rate limiting

≤1 request every 2 seconds to any BGG endpoint (API or website). Jittered exponential backoff on 429/503.
Cache every API response on disk under data/bgg_cache/ keyed by query/ID; check cache before hitting the network so re-runs are free.
Upload stage is deliberately slower: 2–4 s randomized delay between games — this is a real account on a community site.

Search & matching heuristics

Exact normalized-name match → strong candidate.
Fuzzy match: rapidfuzz token_sort_ratio ≥ 90 → good candidate.
Ties: fetch /thing?stats=1 for top ~5 candidates; prefer higher owned-count / better BGG rank (well-known game beats obscure duplicate of the same name).

Search results include boardgameexpansion as a distinct type — keep expansions but tag them so review catches base/expansion confusion.
Classify every result: auto (single confident match) / ambiguous (store all candidates) / unmatched. When in doubt between editions or base-vs-expansion, choose ambiguous.

Website automation (upload stage)

Login with BGG_USERNAME / BGG_PASSWORD env vars; persist Playwright storage state locally (gitignored) so repeat runs skip login. Never write credentials to disk, logs, or error messages.
Per game: navigate to the game page → "Add to Collection" flow → status Owned → if a version_id is known, set it in the collection item's version picker → save. Never guess a version — omit it when unknown. The flow has been walked and documented: see docs/bgg-upload-flow.md for the dialog structure, version-picker behavior (paginated, no search — match by canonical version NAME from the XML API), and observed automation gotchas (stale elements, hidden-not-removed dialogs, hydration races).
A second copy of an owned game must be a NEW collection entry (new collid), not an edit of the existing item. Conversely, version UPGRADES from to_update.csv must edit the EXISTING item (same collid) — additive only: fill the empty version field, change nothing else, and skip any entry that already has a version.
Expect UI fragility: wrap each game in its own try/except, log the failure to upload_log.csv, and continue. --retry-failed re-attempts failures; --dry-run logs without touching the site.
Idempotency: skip IDs already logged added; --verify re-fetches the collection to confirm.

Policy note

BGG's 2025 policy (version 2025-07-02, boardgamegeek.com/using_the_xml_api) locked the API behind registered applications and Bearer tokens, breaking older community tools. Do not use undocumented endpoints, scraped JSON blobs, or third-party mirrors — only the authenticated XML API2 and the public website. Third-party services that proxy BGG data to other applications are explicitly prohibited. The policy asks for server-side requests, aggressive caching, and minimal request counts — the cache-first client design is mandatory, not optional. Policies can change at any time; changes are announced in BGG's Geek Tools News forum.

6.5 KiB Raw Permalink Blame History Unescape Escape