eric/bggpipe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Stage 5 upload: Playwright-driven adds and version updates

Browse Source 
Queue from to_add/to_update minus upload_log.csv (append-per-attempt,
so runs resume); per-game failure isolation with 2-4s pacing;
--dry-run/--verify/--retry-failed/--limit; stub-fixture marker blocks
real runs, dry-run warns. Headed browser by default: live recon showed
Cloudflare Turnstile hard-blocks headless, and BGG never reaches
networkidle. Login selectors verified anonymously; version-picker
pagination and the collection-row update flow remain unverified until
real data exists. Client collection fetches gain a refresh passthrough
so --verify sees the live collection, not cache.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Eric Wagoner
2026-08-01 19:03:25 -04:00
parent 196862e243
commit 3ca7e7f650
8 changed files with 968 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -3
View File
@@ -12,21 +12,21 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
| 2 | `bggpipe resolve` | match titles to BGG IDs and versions via XML API2 | working (stub data — see hard rules) |
| 3 | `bggpipe review` | human review of ambiguous/unmatched items; `--web` serves a FastAPI UI on port 8377 | working |
| 4 | `bggpipe diff` | diff approved matches against the existing BGG collection | working |
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | **not built** — CLI stub exits 1; Playwright not yet a dependency |
| 5 | `bggpipe upload` | add games via a logged-in Playwright session | built; browser flows unverified until real data exists (`--dry-run` works now) |
| 6 | `bggpipe enrich` | fetch full game/version metadata into `games.json` | working |
Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipeline semantics); the upload-stage walkthrough is in `docs/bgg-upload-flow.md`.
## Commands
- `uv sync` — install deps (Python 3.12+, managed by **uv**; use `uv add`, never pip).
- `uv sync` — install deps (Python 3.12+, managed by **uv**; use `uv add`, never pip). Playwright needs a one-time `uv run playwright install chromium`.
- `uv run bggpipe <stage>` — run a pipeline stage. Non-secret settings come from `config.toml` (username, dirs, vision model, rate limit); `--config` overrides the path.
- `uv run pytest` — 105 tests, all offline against fixtures. Tests marked `live` hit the real BGG API (read-only) and are skipped unless you pass `--run-live`.
- `uv run ruff check` / `uv run ruff format` — lint (rules E, F, I, UP, B, SIM) and format.
## Layout
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `enrich`), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`.
- `src/bggpipe/` — `cli.py` (typer app), one module per stage (`extract`, `resolve`, `review` + `webreview`, `diff`, `upload`, `enrich`), plus `bgg_client.py` (rate-limited XML API2 client that caches responses to `data/bgg_cache/`), `normalize.py` (title normalization), `models.py` (dataclasses), `config.py`.
- `scripts/` — `write_stub_fixtures.py` / `write_photo_fixtures.py` generate synthetic fixtures; `record_fixtures.py` re-records real API responses once a token exists.
- `tests/fixtures/bgg_cache/` — stub XML fixtures the offline tests run against.
- `data/` — pipeline state (CSV/JSON artifacts are committed; caches are not — see Git).