Extract reports unidentifiable boxes for human retakes

The vision prompt now returns {titles, unidentified}: boxes that look
like games but can't be confidently titled are reported (location
relative to identified neighbors, partial text, art notes) instead of
silently omitted. They land in data/unidentified.json keyed by photo,
and the end-of-run summary lists them — plus low-confidence reads —
with instructions to retake a closer photo and re-run. New --force flag
re-extracts everything; pre-feature raw caches (bare arrays) still
parse. Live run on IMG_4499 confirmed the flow and the low-confidence
list correctly flags the known "Hebarceos" misread.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -31,4 +31,4 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ## Git
 
 - Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social`), **not GitHub** — `gh` CLI does not work here.
-- Commit `data/matches.csv`, `data/to_add.csv`, `data/upload_log.csv`, `data/titles.json`, `data/games.json`. Never commit `data/bgg_cache/`, `photos/`, Playwright storage state, or `.env`.
+- Commit `data/matches.csv`, `data/to_add.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/games.json`. Never commit `data/bgg_cache/`, `photos/`, Playwright storage state, or `.env`.

bgg-shelf-pipeline-spec.md

@@ -45,6 +45,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
 - Output: `titles.json` — list of `{title_raw, title_normalized, confidence, publisher_hint, edition_hint, year_hint, language_hint, art_notes, source_photos[]}`.
 - Dedupe nuance: identical normalized titles from different photos collapse to one entry ONLY if their edition cues don't conflict; conflicting cues (different publisher/edition text) stay as separate entries.
 - Support `--only <photo>` to re-run a single photo (e.g., after retaking a blurry shot).
+- **Unidentified sightings**: boxes that appear to be games but can't be confidently titled (blurry, obscured, sharp angle, frame edge) are reported rather than silently omitted — location described relative to identified neighbors, plus any partial text and art notes → `unidentified.json`, keyed by photo. The end-of-run summary lists them (and low-confidence reads) so I can take a closer photo and re-run with `--only`.
 
 ### Stage 2 — `resolve`: Match titles to BGG IDs
 
@@ -104,6 +105,7 @@ Each stage reads the previous stage's artifact and writes its own. Re-running a
 All artifacts are flat files in a `data/` directory — human-readable, git-friendly, and reusable by the future frontend:
 
 - `titles.json` — extraction output (stage 1)
+- `unidentified.json` — game boxes seen but not identified (stage 1); retake prompts
 - `bgg_cache/` — cached XML API responses
 - `matches.csv` — the master matching table (stages 2–3)
 - `to_add.csv` — upload queue, new entries (stage 4)

data/titles.json

@@ -6,7 +6,7 @@
     "edition_hint": "",
     "year_hint": null,
     "language_hint": "English",
-    "art_notes": "Dark maroon/brown spine with ornate gold scroll lettering and a portrait of a woman in period dress on the lower spine",
+    "art_notes": "Dark maroon/brown spine with gold ornate lettering and a portrait of a woman in period costume",
     "source_photos": [
       "IMG_4499.jpeg"
     ],
@@ -15,11 +15,11 @@
   {
     "title_raw": "CAT CRIMES",
     "confidence": "high",
-    "publisher_hint": "ThinkFun",
+    "publisher_hint": "",
     "edition_hint": "",
     "year_hint": null,
     "language_hint": "English",
-    "art_notes": "Top-facing box with subtitle 'Who's to Blame Logic Game', illustration of an orange cat, 'AGES 8+' label",
+    "art_notes": "Face-out box, blue and gray with cartoon cat illustration, subtitle 'Who's to Blame Logic Game', shrink-wrapped",
     "source_photos": [
       "IMG_4499.jpeg"
     ],
@@ -32,7 +32,7 @@
     "edition_hint": "",
     "year_hint": null,
     "language_hint": "English",
-    "art_notes": "Black spine with colorful bubble-style logo text, 'the game that breaks its own rules' tagline, 'For 3 to 6 Players AGES 12 TO ADULT'",
+    "art_notes": "Black spine with colorful stylized 'COSMIC ENCOUNTER' logo text, tagline 'the game that breaks its own rules', player count '3 to 6 players, ages 12 to adult'",
     "source_photos": [
       "IMG_4499.jpeg"
     ],
@@ -40,12 +40,12 @@
   },
   {
     "title_raw": "SHERIFF OF NOTTINGHAM",
-    "confidence": "high",
+    "confidence": "medium",
     "publisher_hint": "",
     "edition_hint": "",
     "year_hint": null,
     "language_hint": "English",
-    "art_notes": "Orange/wood-toned spine with green stylized text, illustrated medieval character artwork at top",
+    "art_notes": "Orange/wood-textured spine with green stylized 'SHERIFF' text, top edge shows an illustrated sheriff character; part of title cut off at top of spine",
     "source_photos": [
       "IMG_4499.jpeg"
     ],
@@ -58,7 +58,7 @@
     "edition_hint": "",
     "year_hint": null,
     "language_hint": "English",
-    "art_notes": "Yellow/orange spine with dark fantasy creature artwork, credited 'Connor Reid' at top, green '5 MINUTE' text near bottom",
+    "art_notes": "Yellow/orange spine with fantasy creature artwork (wolf-like armored character), 'Connor Reid' credit at top, '5-MINUTE' in green text",
     "source_photos": [
       "IMG_4499.jpeg"
     ],

data/unidentified.json

@@ -0,0 +1 @@
+{}

src/bggpipe/cli.py

@@ -32,13 +32,16 @@ def extract(
     only: Annotated[
         str | None, typer.Option("--only", help="Re-run a single photo")
     ] = None,
+    force: Annotated[
+        bool, typer.Option("--force", help="Re-extract every photo (ignore cache)")
+    ] = False,
     config: ConfigOpt = None,
 ) -> None:
     """Stage 1: extract game titles + edition cues from shelf photos."""
     from bggpipe.extract import run_extract
 
     cfg = load_config(config)
-    run_extract(cfg, only=only)
+    run_extract(cfg, only=only, force=force)
 
 
 @app.command()

src/bggpipe/config.py

@@ -31,6 +31,10 @@ class Config:
     def titles_path(self) -> Path:
         return self.data_dir / "titles.json"
 
+    @property
+    def unidentified_path(self) -> Path:
+        return self.data_dir / "unidentified.json"
+
     @property
     def matches_path(self) -> Path:
         return self.data_dir / "matches.csv"

src/bggpipe/extract.py

@@ -40,22 +40,45 @@ and anything that is not a board/card game.
 If the SAME title appears more than once with visibly different boxes, report
 each as a separate entry — the owner has multiple editions of some games.
 
-Respond with ONLY a JSON array, no prose, where each element is:
+Respond with ONLY a JSON object, no prose:
 {
-  "title_raw": "title as printed on the box",
-  "confidence": "high" | "medium" | "low",
-  "publisher_hint": "publisher name or logo if legible, else null",
-  "edition_hint": "edition wording if visible ('2nd Edition', 'Deluxe',
-                   'Big Box', anniversary marks), else null",
-  "year_hint": copyright or publication year as an integer, ONLY if printed
-               as publishing info (copyright line, edition year). NEVER use
-               a year that is part of the game's title, theme, or subject
-               matter — a wargame about 1942 is not published in 1942.
-               When unsure whether a year is thematic, use null,
-  "language_hint": "language of the box text if determinable, else null",
-  "art_notes": "distinctive box-art notes (colorway, artwork style) that
-                could identify the edition, else null"
+  "titles": [
+    {
+      "title_raw": "title as printed on the box",
+      "confidence": "high" | "medium" | "low",
+      "publisher_hint": "publisher name or logo if legible, else null",
+      "edition_hint": "edition wording if visible ('2nd Edition', 'Deluxe',
+                       'Big Box', anniversary marks), else null",
+      "year_hint": copyright or publication year as an integer, ONLY if
+                   printed as publishing info (copyright line, edition
+                   year). NEVER use a year that is part of the game's
+                   title, theme, or subject matter — a wargame about 1942
+                   is not published in 1942. When unsure whether a year is
+                   thematic, use null,
+      "language_hint": "language of the box text if determinable, else null",
+      "art_notes": "distinctive box-art notes (colorway, artwork style)
+                    that could identify the edition, else null"
+    }
+  ],
+  "unidentified": [
+    {
+      "location": "where a person standing at this shelf would find the
+                   box: shelf row, position, and the identified games on
+                   either side of it",
+      "partial_text": "any letters or word fragments you can make out,
+                       else null",
+      "art_notes": "color, artwork, box size/shape — anything that would
+                    help identify it"
+    }
+  ]
 }
+
+"unidentified" is for boxes that appear to be games but whose title you
+CANNOT confidently transcribe — too blurry, obscured, at too sharp an
+angle, or cut off at the frame edge. It is ALWAYS better to report an
+unidentifiable box here than to silently omit it, and better here than
+guessing a title into "titles". Never list the same box in both arrays.
+Use an empty array when everything is identified.
 """
 
 
@@ -80,18 +103,37 @@ def prepare_image(path: Path) -> tuple[str, str]:
     return base64.standard_b64encode(buffer.getvalue()).decode(), "image/jpeg"
 
 
-def parse_vision_json(text: str) -> list[dict]:
-    """Parse the model's JSON defensively: strip code fences, find the array."""
+def parse_vision_response(text: str) -> tuple[list[dict], list[dict]]:
+    """Parse the model's JSON defensively: strip code fences, locate the
+    payload amid any prose. Returns (title entries, unidentified sightings).
+    A bare JSON array (the pre-unidentified response shape) still parses —
+    it's all titles."""
     cleaned = _CODE_FENCE.sub("", text).strip()
-    if not cleaned.startswith("["):
-        start, end = cleaned.find("["), cleaned.rfind("]")
-        if start == -1 or end == -1:
-            raise ValueError(f"no JSON array in vision response: {text[:200]!r}")
+    if cleaned[:1] not in ("[", "{"):
+        starts = [i for i in (cleaned.find("["), cleaned.find("{")) if i != -1]
+        if not starts:
+            raise ValueError(f"no JSON in vision response: {text[:200]!r}")
+        start = min(starts)
+        end = cleaned.rfind("]" if cleaned[start] == "[" else "}")
+        if end == -1:
+            raise ValueError(f"unterminated JSON in vision response: {text[:200]!r}")
         cleaned = cleaned[start : end + 1]
-    entries = json.loads(cleaned)
-    if not isinstance(entries, list):
-        raise ValueError("vision response is not a JSON array")
-    return [e for e in entries if isinstance(e, dict) and e.get("title_raw")]
+    data = json.loads(cleaned)
+    if isinstance(data, list):
+        titles_raw, unidentified_raw = data, []
+    elif isinstance(data, dict):
+        titles_raw = data.get("titles") or []
+        unidentified_raw = data.get("unidentified") or []
+    else:
+        raise ValueError("vision response is neither a JSON object nor array")
+    titles = [e for e in titles_raw if isinstance(e, dict) and e.get("title_raw")]
+    unidentified = [
+        u
+        for u in unidentified_raw
+        if isinstance(u, dict)
+        and (u.get("location") or u.get("partial_text") or u.get("art_notes"))
+    ]
+    return titles, unidentified
 
 
 def default_vision(model: str) -> VisionFn:
@@ -126,9 +168,19 @@ def default_vision(model: str) -> VisionFn:
     return vision
 
 
-def extract_photo(photo: Path, vision: VisionFn) -> list[dict]:
+def extract_photo(photo: Path, vision: VisionFn) -> dict:
+    """One photo -> {"titles": [...], "unidentified": [...]} (the raw-cache
+    file format)."""
     image_b64, media_type = prepare_image(photo)
-    raw_entries = parse_vision_json(vision(image_b64, media_type))
+    raw_entries, raw_unidentified = parse_vision_response(vision(image_b64, media_type))
+    unidentified = [
+        {
+            "location": str(u.get("location") or "").strip(),
+            "partial_text": str(u.get("partial_text") or "").strip(),
+            "art_notes": str(u.get("art_notes") or "").strip(),
+        }
+        for u in raw_unidentified
+    ]
     entries = []
     for raw in raw_entries:
         year = raw.get("year_hint")
@@ -146,7 +198,7 @@ def extract_photo(photo: Path, vision: VisionFn) -> list[dict]:
                 "source_photos": [photo.name],
             }
         )
-    return entries
+    return {"titles": entries, "unidentified": unidentified}
 
 
 def _cues_conflict(a: dict, b: dict) -> bool:
@@ -194,18 +246,38 @@ def dedupe_entries(entries: list[dict]) -> list[dict]:
     return result
 
 
-def rebuild_titles(raw_dir: Path, titles_path: Path) -> list[dict]:
+def rebuild_artifacts(
+    raw_dir: Path, titles_path: Path, unidentified_path: Path
+) -> tuple[list[dict], dict[str, list[dict]]]:
+    """Regenerate titles.json and unidentified.json from the per-photo raw
+    cache. Raw files written before the unidentified feature are bare
+    arrays — still readable."""
     entries: list[dict] = []
+    unidentified: dict[str, list[dict]] = {}
     for raw_file in sorted(raw_dir.glob("*.json")):
-        entries.extend(json.loads(raw_file.read_text()))
+        data = json.loads(raw_file.read_text())
+        if isinstance(data, list):  # legacy format
+            entries.extend(data)
+            continue
+        entries.extend(data.get("titles") or [])
+        photo = raw_file.name.removesuffix(".json")
+        if data.get("unidentified"):
+            unidentified[photo] = data["unidentified"]
     deduped = dedupe_entries(entries)
     titles_path.parent.mkdir(parents=True, exist_ok=True)
     titles_path.write_text(json.dumps(deduped, indent=2, ensure_ascii=False) + "\n")
-    return deduped
+    unidentified_path.write_text(
+        json.dumps(unidentified, indent=2, ensure_ascii=False) + "\n"
+    )
+    return deduped, unidentified
 
 
 def run_extract(
-    cfg: Config, *, only: str | None = None, vision: VisionFn | None = None
+    cfg: Config,
+    *,
+    only: str | None = None,
+    force: bool = False,
+    vision: VisionFn | None = None,
 ) -> list[dict]:
     photos = (
         sorted(
@@ -232,13 +304,49 @@ def run_extract(
 
     for photo in photos:
         raw_path = raw_dir / f"{photo.name}.json"
-        if raw_path.exists() and not only:
+        if raw_path.exists() and not only and not force:
             typer.echo(f"  {photo.name}: already extracted, skipping")
             continue
-        entries = extract_photo(photo, vision)
-        raw_path.write_text(json.dumps(entries, indent=2, ensure_ascii=False) + "\n")
-        typer.echo(f"  {photo.name}: {len(entries)} title(s)")
+        result = extract_photo(photo, vision)
+        raw_path.write_text(json.dumps(result, indent=2, ensure_ascii=False) + "\n")
+        note = (
+            f" ({len(result['unidentified'])} unidentified)"
+            if result["unidentified"]
+            else ""
+        )
+        typer.echo(f"  {photo.name}: {len(result['titles'])} title(s){note}")
 
-    deduped = rebuild_titles(raw_dir, cfg.titles_path)
+    deduped, unidentified = rebuild_artifacts(
+        raw_dir, cfg.titles_path, cfg.unidentified_path
+    )
     typer.echo(f"Wrote {len(deduped)} unique title(s) to {cfg.titles_path}.")
+
+    if unidentified:
+        typer.echo(
+            "\nSaw but couldn't identify — take a closer photo of each, drop "
+            "it in photos/, and run extract again:"
+        )
+        for photo_name, sightings in unidentified.items():
+            for s in sightings:
+                detail = "; ".join(
+                    part
+                    for part in (
+                        s["location"],
+                        f"text visible: {s['partial_text']!r}"
+                        if s["partial_text"]
+                        else "",
+                        s["art_notes"],
+                    )
+                    if part
+                )
+                typer.echo(f"  [{photo_name}] {detail}")
+
+    shaky = [e for e in deduped if e["confidence"] != "high"]
+    if shaky:
+        typer.echo("\nLow-confidence reads worth double-checking:")
+        for e in shaky:
+            typer.echo(
+                f"  {e['title_raw']!r} ({e['confidence']}) in "
+                f"{', '.join(e['source_photos'])}"
+            )
     return deduped

tests/test_extract.py

@@ -12,7 +12,7 @@ import typer
 from bggpipe.config import Config
 from bggpipe.extract import (
     dedupe_entries,
-    parse_vision_json,
+    parse_vision_response,
     prepare_image,
     run_extract,
 )
@@ -77,25 +77,43 @@ def test_prepare_image_converts_heic(tmp_path):
 # -- defensive JSON parsing ---------------------------------------------
 
 
-def test_parse_strips_code_fences():
+def test_parse_object_with_titles_and_unidentified():
+    text = """```json
+{"titles": [{"title_raw": "Catan", "confidence": "high"}],
+ "unidentified": [{"location": "top shelf, left of Catan",
+                   "partial_text": "WAR", "art_notes": "red spine"}]}
+```"""
+    titles, unidentified = parse_vision_response(text)
+    assert titles[0]["title_raw"] == "Catan"
+    assert unidentified[0]["location"] == "top shelf, left of Catan"
+
+
+def test_parse_legacy_bare_array_still_works():
     text = '```json\n[{"title_raw": "Catan", "confidence": "high"}]\n```'
-    assert parse_vision_json(text)[0]["title_raw"] == "Catan"
+    titles, unidentified = parse_vision_response(text)
+    assert titles[0]["title_raw"] == "Catan"
+    assert unidentified == []
 
 
-def test_parse_tolerates_prose_around_array():
+def test_parse_tolerates_prose_around_json():
     text = 'Here are the games:\n[{"title_raw": "Wingspan"}]\nLet me know!'
-    assert parse_vision_json(text)[0]["title_raw"] == "Wingspan"
+    titles, _ = parse_vision_response(text)
+    assert titles[0]["title_raw"] == "Wingspan"
 
 
 def test_parse_drops_malformed_entries():
-    text = '[{"title_raw": "Catan"}, {"no_title": true}, "just a string"]'
-    entries = parse_vision_json(text)
-    assert len(entries) == 1
+    text = (
+        '{"titles": [{"title_raw": "Catan"}, {"no_title": true}, "just a string"],'
+        ' "unidentified": [{}, "not a dict", {"location": "somewhere"}]}'
+    )
+    titles, unidentified = parse_vision_response(text)
+    assert len(titles) == 1
+    assert len(unidentified) == 1  # empty {} and the bare string are dropped
 
 
 def test_parse_raises_on_garbage():
     with pytest.raises(ValueError):
-        parse_vision_json("I couldn't see any games clearly.")
+        parse_vision_response("I couldn't see any games clearly.")
 
 
 # -- dedupe with edition cues -------------------------------------------
@@ -205,3 +223,97 @@ def test_run_extract_empty_photos_dir_exits(tmp_path):
     vision, _ = _vision_stub("[]")
     with pytest.raises(typer.Exit):
         run_extract(cfg, vision=vision)
+
+
+# -- unidentified sightings ---------------------------------------------
+
+import json  # noqa: E402
+
+OBJECT_PAYLOAD = json.dumps(
+    {
+        "titles": [{"title_raw": "Catan", "confidence": "high"}],
+        "unidentified": [
+            {
+                "location": "middle shelf, between Catan and the frame edge",
+                "partial_text": "EMP",
+                "art_notes": "tall black box with gold lettering",
+            }
+        ],
+    }
+)
+
+
+def test_unidentified_lands_in_artifact_and_summary(tmp_path, capsys):
+    cfg = _cfg(tmp_path)
+    cfg.photos_dir.mkdir()
+    _write_image(cfg.photos_dir / "shelf1.jpg")
+    vision, _ = _vision_stub(OBJECT_PAYLOAD)
+    run_extract(cfg, vision=vision)
+
+    saved = json.loads(cfg.unidentified_path.read_text())
+    assert saved == {
+        "shelf1.jpg": [
+            {
+                "location": "middle shelf, between Catan and the frame edge",
+                "partial_text": "EMP",
+                "art_notes": "tall black box with gold lettering",
+            }
+        ]
+    }
+    out = capsys.readouterr().out
+    assert "couldn't identify" in out
+    assert "between Catan and the frame edge" in out
+
+
+def test_clean_photo_writes_empty_unidentified(tmp_path):
+    cfg = _cfg(tmp_path)
+    cfg.photos_dir.mkdir()
+    _write_image(cfg.photos_dir / "shelf1.jpg")
+    vision, _ = _vision_stub('{"titles": [{"title_raw": "Azul"}], "unidentified": []}')
+    run_extract(cfg, vision=vision)
+    assert json.loads(cfg.unidentified_path.read_text()) == {}
+
+
+def test_legacy_array_raw_cache_still_rebuilds(tmp_path):
+    """Raw files written before the unidentified feature are bare arrays."""
+    cfg = _cfg(tmp_path)
+    cfg.photos_dir.mkdir()
+    _write_image(cfg.photos_dir / "old.jpg")
+    raw_dir = cfg.data_dir / "extract_raw"
+    raw_dir.mkdir(parents=True)
+    (raw_dir / "old.jpg.json").write_text(json.dumps([_entry("Catan", "old.jpg")]))
+    vision, calls = _vision_stub("[]")
+    result = run_extract(cfg, vision=vision)
+    assert calls == []  # cached photo untouched
+    assert result[0]["title_raw"] == "Catan"
+    assert json.loads(cfg.unidentified_path.read_text()) == {}
+
+
+def test_force_reextracts_cached_photos(tmp_path):
+    cfg = _cfg(tmp_path)
+    cfg.photos_dir.mkdir()
+    _write_image(cfg.photos_dir / "shelf1.jpg")
+    vision, calls = _vision_stub('[{"title_raw": "Azul"}]')
+    run_extract(cfg, vision=vision)
+    run_extract(cfg, vision=vision)
+    assert len(calls) == 1  # cache hit
+
+    vision2, calls2 = _vision_stub(OBJECT_PAYLOAD)
+    result = run_extract(cfg, force=True, vision=vision2)
+    assert len(calls2) == 1  # cache ignored
+    assert result[0]["title_raw"] == "Catan"
+
+
+def test_low_confidence_reads_are_surfaced(tmp_path, capsys):
+    cfg = _cfg(tmp_path)
+    cfg.photos_dir.mkdir()
+    _write_image(cfg.photos_dir / "shelf1.jpg")
+    vision, _ = _vision_stub(
+        '[{"title_raw": "Patchwork", "confidence": "medium"},'
+        ' {"title_raw": "Catan", "confidence": "high"}]'
+    )
+    run_extract(cfg, vision=vision)
+    out = capsys.readouterr().out
+    assert "Low-confidence reads" in out
+    assert "'Patchwork' (medium)" in out
+    assert "'Catan'" not in out.split("Low-confidence reads")[1]