Count pending work: BGG's export lags, so raw queue rows lie

The card read "1 version updates" while upload said "skipping 1 already
done". Both were right. Recon on the live site shows the update DID
apply — the version cell reads "English first edition Year: 2012" and
its radio is checked — but BGG's XML collection export still reports
that collid with no version, even on a forced refresh. diff reads the
API, so it re-queued finished work; the log correctly refused it.

Nothing to fix in the flow: the pipeline card now counts PENDING jobs
(queue rows minus what the log completed) for both to_add and
to_update, reports outstanding failures rather than every failure ever
logged, and when everything queued is already applied it says so and
names the cause. Documented under "BGG's collection export lags the
site" so the next person doesn't chase it as a bug.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g

data/to_update.csv

@@ -1,37 +1,2 @@
 collid,bgg_id,bgg_name,version_id,version_name
-53429559,240,Britannia,24621,Avalon Hill second edition
-53429530,71,Civilization,24006,Avalon Hill first edition 1981
-53429542,177,Advanced Civilization,24972,English edition
-53429504,3090,Doctor Who: The Game of Time & Space,25782,English edition
-53429482,3585,Sorcerer: The Game of Magical Conflict,211154,SPI Designer's Edition
-53429367,2524,StarForce 'Alpha Centauri': Interstellar Conflict in the 25th Century,219137,SPI boxed designer's edition
-53429816,224,History of the World,20727,Avalon Hill/Hasbro edition
-53429636,9209,Ticket to Ride,294188,English edition 2012 with Spiel des Jahres logo
-53430595,13,Catan,347121,English edition 2016
-53430485,1219,Labyrinth,488621,English edition 2017
-53429649,137909,Bugs in the Kitchen,216474,English first edition
-53430142,27710,Catan Dice Game,606657,English edition 2018
-53429953,169784,Castle Panic: The Dark Titan,260454,English first edition
-53429666,147370,Robot Turtles,231664,English second edition
-53430559,43443,Castle Panic,103068,"English edition, fourth printing"
-53429632,24310,The Red Dragon Inn,374258,English fifth edition
-53429748,39856,Dixit,296529,Asmodee/Libellud English edition 2015
-53430257,35505,Walk the Plank!,781669,English MDG-4311 edition
-53430020,164,"Before I Kill You, Mister Bond",28465,"Cheapass black and white ""Better Edition"" edition"
-148198097,181304,Mysterium,536289,English edition 2018-2
-53429887,65244,Forbidden Island,31208,English edition 051210
-53429908,95386,Tempurra,290572,English edition
-53430200,258,Fluxx,20840,Looney Labs Version v2.1
-148198100,312786,Poetry for Neanderthals,514050,English edition 2020
-53430332,188614,Simon's Cat Card Game,779792,"English edition, second printing"
-53429850,4324,Risk: The Lord of the Rings,26859,English edition 2002
-53429933,1927,Munchkin,28144,"English edition 2009, 15th printing"
-53430007,1784,Dark Cults,110926,First edition
-53429986,176,Give Me the Brain!,27673,Czech edition
-53430040,257,Kill Doctor Lucky,520364,"Cheapass envelope edition, second printing"
-53612880,153999,"...and then, we held hands.",280845,English edition
-53612866,166976,Consentacle,410382,English Kickstarter edition
-53430652,231302,The Cat Game,361880,English/French edition
-53430517,40692,Small World,294178,English third edition
-53429718,172503,Mage Wars Academy,265107,English edition
 53429642,104710,Wiz-War (Eighth Edition),117685,English first edition

docs/bgg-upload-flow.md

@@ -167,3 +167,22 @@ cannot say which copy it edits. The collection table can:
 
 This is strictly additive: it sets one field on one collid and cannot
 create a duplicate entry.
+
+### BGG's collection export lags the site
+
+After a successful update the website shows the new version immediately
+(the row's version cell reads e.g. "English first edition Year: 2012",
+and reopening the editor shows that radio checked), but the XML API's
+`/collection` export can still report the entry with no version — even
+with a cache-busting re-request. `diff` reads the API, so it will
+re-queue work that has already landed.
+
+Consequences to keep in mind rather than "fix":
+
+- The upload log is the authority on what this tool did; the API is the
+  authority on what BGG has published. They disagree for a while.
+- `build_queue` skipping a job whose log says done is CORRECT here — the
+  work is applied, and re-running it would be a no-op at best.
+- Anything user-facing should count PENDING jobs (queue rows minus what
+  the log completed), never raw queue rows, or settled work reads as
+  outstanding until the next diff.

src/bggpipe/templates/pages/pipeline.html

@@ -42,8 +42,13 @@ function render() {
 
   const uploadFacts = `<b>${P.to_add}</b> to add · <b>${P.to_update}</b> version updates
     ${log.added || log.added_no_version ? `· <b>${(log.added ?? 0) + (log.added_no_version ?? 0)}</b> added` : ""}
-    ${log.failed ? `· <b>${log.failed}</b> failed` : ""}
+    ${P.upload_failed ? `· <b>${P.upload_failed}</b> failed` : ""}
-    · <a href="/queue">inspect the queue</a>`;
+    · <a href="/queue">inspect the queue</a>
+    ${!P.to_add && !P.to_update && P.queued_total
+      ? `<br><span class="meta">${P.queued_total} queued row(s) already applied —
+         the next <b>diff</b> clears them (BGG's collection export can lag the
+         site by a while)</span>`
+      : ""}`;
 
   document.getElementById("stages").innerHTML = [
     stageCard(1, "extract", `read titles off <b>${P.photos}</b> <a href="/photos">photo(s)</a> — <b>${P.titles}</b> so far`

src/bggpipe/webreview.py

@@ -786,6 +786,15 @@ def create_app(
         games = read_games()
         return sorted(games.values(), key=lambda g: (g.get("name") or "").casefold())
 
+    def _pending(path: Path, action: str, log_rows: list[dict]) -> int:
+        from bggpipe.upload import annotate_queue
+
+        if not path.exists():
+            return 0
+        with path.open(newline="") as f:
+            rows = list(csv.DictReader(f))
+        return sum(1 for r in annotate_queue(rows, action, log_rows) if not r["state"])
+
     def _csv_count(path: Path) -> int:
         if not path.exists():
             return 0
@@ -799,6 +808,7 @@ def create_app(
             match_counts = Counter(row["match_status"] for row in session.rows)
             log_counts: Counter[str] = Counter()
             failed_now = 0
+            log_rows: list[dict] = []
             log_path = cfg.upload_log_path
             if log_path.exists():
                 from bggpipe.upload import outstanding_failures
@@ -855,8 +865,14 @@ def create_app(
                 "matches": dict(match_counts),
                 "pending_review": len(session.pending_rows())
                 + len(session.version_rows()),
-                "to_add": _csv_count(cfg.to_add_path),
+                # PENDING work, not raw rows: the queue CSVs are diff-time
-                "to_update": _csv_count(cfg.to_update_path),
+                # snapshots that keep listing finished jobs (and BGG's
+                # collection export lags the site, so a re-run diff can
+                # re-queue work that already landed)
+                "to_add": _pending(cfg.to_add_path, "add", log_rows),
+                "to_update": _pending(cfg.to_update_path, "update", log_rows),
+                "queued_total": _csv_count(cfg.to_add_path)
+                + _csv_count(cfg.to_update_path),
                 "upload_log": dict(log_counts),
                 "upload_failed": failed_now,
                 "games": games,

tests/test_web_dashboard.py

@@ -649,3 +649,24 @@ def test_photo_detail_page_serves_with_photos_nav_active(tmp_path):
     (current,) = re.findall(r'<a href="([^"]+)" aria-current="page"', html)
     assert current == "/photos"
     assert 'src="/static/app.js"' in html
+
+
+def test_pipeline_counts_pending_work_not_queue_rows(tmp_path):
+    """A finished job stays in to_update.csv until the next diff, and BGG's
+    collection export can lag the site — so the card must count what is
+    actually left to do, or settled work reads as outstanding forever."""
+    cfg = _cfg(tmp_path)
+    cfg.to_update_path.write_text(
+        "action,bgg_id,bgg_name,collid,version_id,version_name\n"
+        "update,104710,Wiz-War,53429642,117685,English first\n"
+    )
+    p = _app(cfg).get("/api/pipeline").json()
+    assert p["to_update"] == 1 and p["queued_total"] == 1  # nothing done yet
+
+    cfg.upload_log_path.write_text(
+        "action,bgg_id,collid,name,version_id,second_copy,status,timestamp,error\n"
+        "update,104710,53429642,Wiz-War,117685,,updated,t,\n"
+    )
+    p = _app(cfg).get("/api/pipeline").json()
+    assert p["to_update"] == 0  # applied
+    assert p["queued_total"] == 1  # still listed until diff reruns