Compare commits

...
2 Commits
Author SHA1 Message Date
Eric WagonerandClaude Fable 5 86d434a400 Audit round 5 (curation feature): 5 blind reviewers, 14 confirmed fixes
The standing post-feature audit over a7f0cfe. Correctness (data): splits
become photo-scoped store records so splitting one edition no longer
force-splits same-named editions, and renaming a split copy migrates its
protection to the corrected title instead of silently re-merging copies.
Correctness (web): edit scoping now counts siblings by NORMALIZED title
(matching how stored edits apply), same-title-same-photos edits are
refused rather than corrupting the sibling entry, split copies serve
their real per-photo cues to the edit form instead of blanks, and a
split whose row vanished underneath returns 409 instead of a false 200.
Silent failures: replay_titles refuses to rebuild from a PARTIAL raw
cache (fresh clone + one --only extract would have truncated the
committed titles.json); the edit endpoint writes in crash-safe order
(cull, record, replay); corrupt curation stores fail loud naming the
file; retried edits don't double-record. Review-decision durability:
drop_rows never drops dedupe_veto rows — a rename retitles them in
place — and writes through a no-reload path so a concurrent rewrite
can't silently discard the cull. Style: catalog action cells get their
own class (.rowactions' flex display broke table alignment), editor
inputs match the design system and stop overriding the global
focus-visible outline, EditBody's clear-semantics docstring scoped to
cue fields, "nothing to change" derived from the record itself.

Tests: 8 new (photo-scoped splits, veto preservation, photo-narrowed
drops, 409s on both curation endpoints under a running job, partial-raw
replay guard, rename-keeps-protection lifecycle, corrupt-store error,
cue-field editing) and the dead edition_hint key in the edit test now
exercises real cue fields. 259 passing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
2026-08-02 20:04:15 -04:00
Eric WagonerandClaude Fable 5 a7f0cfee05 Durable curation: persisted splits + pre-resolve title edits, catalog A→Z
Wiz-War had no split button: can_split required a matches row, but fresh
extractions leave multi-photo titles rowless until resolve runs. Splits
are now a title-level decision persisted in data/title_splits.json,
honored by extract's dedupe and resolve's dedupe on every rebuild, with
the button on any multi-photo line — resolved or not.

Same mechanism carries human corrections: data/title_edits.json stores
fixed misreads and known cues (publisher/edition/year/language), applied
before dedupe on every titles.json rebuild, editable from a new inline
form on every catalog line. An edit drops the title's stale matches rows
so resolve re-queries with the corrected data.

The catalog page now sorts alphabetically (case-insensitive; split
copies stay adjacent) instead of extraction order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
2026-08-02 19:47:08 -04:00
14 changed files with 1032 additions and 39 deletions
+2 -1
View File
@@ -47,10 +47,11 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
- Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan). - Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan).
- Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry. - Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry.
- Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance. - Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance.
- **Human curation is durable**: `data/title_splits.json` (photo-scoped split-into-copies decisions, honored by extract's dedupe AND resolve's dedupe) and `data/title_edits.json` (corrected reads/cues, applied before dedupe on every titles.json rebuild) persist forever. Row-level decisions persist via the `dedupe_veto` column — edits never drop veto'd rows (a rename retitles them in place).
- **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope). - **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope).
- Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift. - Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift.
## Git ## Git
- Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub**`gh` CLI does not work here. - Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub**`gh` CLI does not work here.
- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`. - Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/title_splits.json`, `data/title_edits.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`.
+3
View File
@@ -0,0 +1,3 @@
[
"Wiz-War"
]
+30 -6
View File
@@ -260,16 +260,14 @@
}, },
{ {
"title_raw": "Wiz-War", "title_raw": "Wiz-War",
"confidence": "high", "confidence": "medium",
"publisher_hint": "Fantasy Flight Games", "publisher_hint": "",
"edition_hint": "9th Edition", "edition_hint": "",
"year_hint": null, "year_hint": null,
"language_hint": "English", "language_hint": "English",
"art_notes": "black spine with purple/blue text, partial visible 'Wiz-Wa...' with subtitle 'and board game of ...ly battle and treasures!'", "art_notes": "black spine with purple/blue text, partial visible 'Wiz-Wa...' with subtitle 'and board game of ...ly battle and treasures!'",
"source_photos": [ "source_photos": [
"IMG_4502.jpeg", "IMG_4502.jpeg"
"IMG_4504.jpeg",
"IMG_4528.jpeg"
], ],
"title_normalized": "wiz war" "title_normalized": "wiz war"
}, },
@@ -351,6 +349,19 @@
], ],
"title_normalized": "dungeon" "title_normalized": "dungeon"
}, },
{
"title_raw": "WIZ-WAR",
"confidence": "high",
"publisher_hint": "Fantasy Flight Games",
"edition_hint": "",
"year_hint": null,
"language_hint": "English",
"art_notes": "Dark spine with wizard/warrior artwork, sepia tones",
"source_photos": [
"IMG_4504.jpeg"
],
"title_normalized": "wiz war"
},
{ {
"title_raw": "TICKET TO RIDE", "title_raw": "TICKET TO RIDE",
"confidence": "high", "confidence": "high",
@@ -1280,6 +1291,19 @@
], ],
"title_normalized": "red dragon inn smorgasbox" "title_normalized": "red dragon inn smorgasbox"
}, },
{
"title_raw": "WIZ-WAR",
"confidence": "high",
"publisher_hint": "",
"edition_hint": "9th Edition",
"year_hint": null,
"language_hint": "English",
"art_notes": "Dark purple/maroon box with pink outlined logo text, illustration of a turbaned wizard character casting fire spell in bottom right corner, tagline 'KILL THEM WITH FIRE!'",
"source_photos": [
"IMG_4528.jpeg"
],
"title_normalized": "wiz war"
},
{ {
"title_raw": "SLUGFEST GAMES", "title_raw": "SLUGFEST GAMES",
"confidence": "low", "confidence": "low",
+45
View File
@@ -1,45 +1,90 @@
[ [
"IMG_4501.jpeg|Bottom shelf, left portion, above 'Exploding Kittens' box|EXPANSION|Small box with partial text visible, colorful design, title cut off by frame edge",
"IMG_4501.jpeg|Bottom shelf, right side, near Exploding Kittens box|COSMIC|Colorful box spine, blue/green tones, title partially visible but not fully confirmable",
"IMG_4501.jpeg|Middle shelf, purple box beneath Joking Hazard, right side of stack|A Boardgame for 1 to 6 players, ages 8 and up.|Purple/violet box, text describing player count, title not legible from this angle",
"IMG_4501.jpeg|Right side, middle shelf, black spine sticking out between Santorini and purple box below|T|Black spine, single letter visible, rest obscured",
"IMG_4501.jpeg|Top shelf, far left edge, cut off by frame; near 'a Gentle Rain' box||Dark/black box, only a thin sliver visible",
"IMG_4502.jpeg|bottom-left corner of shelf, beneath the History of the World area, left of Dungeon!|A Boardgame for 1 to 6 players, ages 8 and up|purple/tan box with descriptive text visible but no title readable",
"IMG_4502.jpeg|far left side of shelf, stacked horizontally beneath the yellow-floral box, left of Santorini||small colorful box spines, red and white coloring, too angled/blurry to read title",
"IMG_4502.jpeg|left side of shelf, below the horizontal stack, left of Santorini and above Joking Hazard||blue box with white illustration, appears to be a game box but title not legible",
"IMG_4502.jpeg|top-left corner of shelf, above Santorini and to the left of Wiz-War|possibly 'Herbaceous' or similar, text partially cut off|cream/white box with floral yellow illustration, thin spine",
"IMG_4505.jpeg|Bottom area, left side near Mantis Falls, small figure/box partially visible||Small gold/tan colored object, possibly a game piece or small box, mostly obscured", "IMG_4505.jpeg|Bottom area, left side near Mantis Falls, small figure/box partially visible||Small gold/tan colored object, possibly a game piece or small box, mostly obscured",
"IMG_4505.jpeg|Top shelf, far left edge behind Cat Crimes, cut off by frame edge||Dark brown/maroon spine with faint gold decorative border, partially visible vertical box",
"IMG_4505.jpeg|Top shelf, far left edge, partially cut off by frame, to the left of Cat Crimes||Dark brown/maroon box spine with gold decorative swirl design, portion visible at left edge of image", "IMG_4505.jpeg|Top shelf, far left edge, partially cut off by frame, to the left of Cat Crimes||Dark brown/maroon box spine with gold decorative swirl design, portion visible at left edge of image",
"IMG_4506.jpeg|Middle area, below and left of Santorini, dark red/maroon box spine with small square icons||Dark reddish-brown spine with small colored square icons in a row, text not legible", "IMG_4506.jpeg|Middle area, below and left of Santorini, dark red/maroon box spine with small square icons||Dark reddish-brown spine with small colored square icons in a row, text not legible",
"IMG_4506.jpeg|Right side of image, behind Santorini box, dark box with small visible logo||Dark/black box with a small circular or square colored logo element, mostly obscured by foreground Santorini box",
"IMG_4506.jpeg|Right side, background behind Santorini box, partially visible dark box with small game piece icon||Dark background box with a small blue/white icon, mostly obscured by Santorini box in foreground", "IMG_4506.jpeg|Right side, background behind Santorini box, partially visible dark box with small game piece icon||Dark background box with a small blue/white icon, mostly obscured by Santorini box in foreground",
"IMG_4506.jpeg|Top shelf, above Santorini spine, partially visible box with patterned tiles/squares in yellow, blue, and cream tones||Patchwork/quilt-like pattern of colored squares (yellow, tan, blue) along the top edge of the box, style suggests a tile-based board game", "IMG_4506.jpeg|Top shelf, above Santorini spine, partially visible box with patterned tiles/squares in yellow, blue, and cream tones||Patchwork/quilt-like pattern of colored squares (yellow, tan, blue) along the top edge of the box, style suggests a tile-based board game",
"IMG_4506.jpeg|Top shelf, right side near blue corner box, small red 'X' symbols visible on light blue background|X X|Light blue box edge with red X pattern, possibly a tally or scoring game box", "IMG_4506.jpeg|Top shelf, right side near blue corner box, small red 'X' symbols visible on light blue background|X X|Light blue box edge with red X pattern, possibly a tally or scoring game box",
"IMG_4506.jpeg|Top shelf, spine visible above Santorini box, partially cut off at top of frame||Multicolored patchwork/tile pattern spine in yellow, tan, and blue tones with geometric designs; appears to be a board game box edge but no title text is legible",
"IMG_4506.jpeg|Upper right corner of image, above and right of Santorini spine, next to unidentified patterned box||Small red and blue box corner visible, mostly cut off by frame edge, no discernible text",
"IMG_4507.jpeg|Below 'LABYRINTH', left side of shelf, partially cut off at frame edge|DRAGO...|Green/dark box, only top edge visible, text cut off",
"IMG_4507.jpeg|Middle shelf area, partially hidden behind Labyrinth box and below Utter Nonsense, left side of the stack|Dragon...|Green/dark colored box spine, text cut off by adjacent boxes, only 'Dragon' prefix visible", "IMG_4507.jpeg|Middle shelf area, partially hidden behind Labyrinth box and below Utter Nonsense, left side of the stack|Dragon...|Green/dark colored box spine, text cut off by adjacent boxes, only 'Dragon' prefix visible",
"IMG_4507.jpeg|Top of stack, right of center, between 'a Gentle Rain' and 'SANToRINI'|PATCH / WORK (possibly)|Colorful patchwork quilt-style pattern on spine, small box",
"IMG_4507.jpeg|Upper right area near Santorini and a Gentle Rain, appears to be a narrow spine box|PATCH WORK or similar|Colorful patchwork/quilt-like pattern visible on spine, tan and multicolor squares design", "IMG_4507.jpeg|Upper right area near Santorini and a Gentle Rain, appears to be a narrow spine box|PATCH WORK or similar|Colorful patchwork/quilt-like pattern visible on spine, tan and multicolor squares design",
"IMG_4508.jpeg|Top shelf, center, small yellow tin sitting next to a black box near the stack of papers/magazines, above the Labyrinth box||Small yellow tin container, no visible text, could be a card game tin",
"IMG_4508.jpeg|Top shelf, center, yellow tin box sitting next to the Super Mario Checkers box||Small yellow tin, no visible title text, could be a card game tin", "IMG_4508.jpeg|Top shelf, center, yellow tin box sitting next to the Super Mario Checkers box||Small yellow tin, no visible title text, could be a card game tin",
"IMG_4508.jpeg|Top shelf, far left, behind the stack of papers/notebooks, next to the framed picture||Small black box with unreadable logo, mostly obscured by papers", "IMG_4508.jpeg|Top shelf, far left, behind the stack of papers/notebooks, next to the framed picture||Small black box with unreadable logo, mostly obscured by papers",
"IMG_4508.jpeg|Top shelf, far left, partially cut off by frame edge, next to a green-framed picture and above the Super Mario Checkers box||Teal/green colored box edge, mostly obscured, shape suggests a game box but no readable text",
"IMG_4508.jpeg|Top shelf, right of center, wooden box with metal clasp between the checkers box and the papers stack||Wooden box with brass hardware, no visible title, appears decorative or a game storage box", "IMG_4508.jpeg|Top shelf, right of center, wooden box with metal clasp between the checkers box and the papers stack||Wooden box with brass hardware, no visible title, appears decorative or a game storage box",
"IMG_4508.jpeg|Top shelf, right side, partially visible box behind the wooden box, between the stack of papers and the edge of the frame|SCRABBLE (partial letters visible)|Red and teal colored box edge, appears to be a Scrabble-branded box but mostly obscured", "IMG_4508.jpeg|Top shelf, right side, partially visible box behind the wooden box, between the stack of papers and the edge of the frame|SCRABBLE (partial letters visible)|Red and teal colored box edge, appears to be a Scrabble-branded box but mostly obscured",
"IMG_4511.jpeg|Middle shelf, behind Superfight boxes, green/black box with skeletal figure artwork - between Superfight stack and the Rook box|...CTHULHU|Dark green box, illustrated skeletal/robed figure, likely a Cthulhu-themed game (e.g., Cthulhu Fluxx or Cthulhu Gloom) but exact title obscured",
"IMG_4511.jpeg|Middle shelf, black box stacked below SUPERFIGHT, above ROOK; left side of shelf|...Go... Yourself|Black spine box, small white/colored text, similar size to Superfight box", "IMG_4511.jpeg|Middle shelf, black box stacked below SUPERFIGHT, above ROOK; left side of shelf|...Go... Yourself|Black spine box, small white/colored text, similar size to Superfight box",
"IMG_4511.jpeg|Middle shelf, black spine box stacked above Rook, to the left of the dice tray|...Yourself|Black box with orange/white text, only partial word 'Yourself' legible, title obscured by angle",
"IMG_4511.jpeg|Right edge of shelf, partially cut off by frame, next to Cthulhu box|HEX|Blue box with yellow/tan geometric pattern, only edge visible", "IMG_4511.jpeg|Right edge of shelf, partially cut off by frame, next to Cthulhu box|HEX|Blue box with yellow/tan geometric pattern, only edge visible",
"IMG_4511.jpeg|Top right corner of shelf, box mostly out of frame, above the Hex box|Drag...|Orange and blue box, only a fragment of text visible, likely a dragon-themed game title",
"IMG_4517.jpeg|Middle of shelf, between Pie Face! (left) and the blue box with bird artwork (right)|...Grenade...|Tall narrow box, dark red/brown color scheme, appears to have a hand grenade or bomb-like illustration", "IMG_4517.jpeg|Middle of shelf, between Pie Face! (left) and the blue box with bird artwork (right)|...Grenade...|Tall narrow box, dark red/brown color scheme, appears to have a hand grenade or bomb-like illustration",
"IMG_4517.jpeg|Middle of shelf, right of the grenade-themed box and left of Walk the Plank!|Bird ... Bird ...|Blue box with tall bird-like cartoon character illustration, small/narrow box shape", "IMG_4517.jpeg|Middle of shelf, right of the grenade-themed box and left of Walk the Plank!|Bird ... Bird ...|Blue box with tall bird-like cartoon character illustration, small/narrow box shape",
"IMG_4517.jpeg|Middle shelf area, second box from left, between 'Pie Face!' and 'Bird on Your Bread?!'|...renade... Nintendo?|Narrow tall box, red and tan coloring, partially obscured by shadow and angle, appears to have small figure illustration",
"IMG_4518.jpeg|Between Mage Wars Academy Core Set and Saberu Merry Men||Thin black spine, largely obscured, possibly a second small game box", "IMG_4518.jpeg|Between Mage Wars Academy Core Set and Saberu Merry Men||Thin black spine, largely obscured, possibly a second small game box",
"IMG_4519.jpeg|Far right of shelf, white spine box next to the two red 'G' logo boxes|Before I Kill You, Mister Bond... / Kill Dr. No Lucky|White worn spine with black text, possibly James Bond themed game, 'CHEAPEST GAMES' text also visible on spine which may be a store sticker rather than title", "IMG_4519.jpeg|Far right of shelf, white spine box next to the two red 'G' logo boxes|Before I Kill You, Mister Bond... / Kill Dr. No Lucky|White worn spine with black text, possibly James Bond themed game, 'CHEAPEST GAMES' text also visible on spine which may be a store sticker rather than title",
"IMG_4519.jpeg|Left shelf, red spine with black text between the cream D&D box and the red-orange boxes|THE BEST OF RA... (possibly 'RAMPHA' or similar)|Dark red spine with illustrated figure artwork, book-like thickness",
"IMG_4519.jpeg|Middle shelf area, two matching red boxes with orange circular 'G' logo, positioned between the white D&D boxes on the left and the white spined books on the right|G (stylized orange circular logo)|Two identical red boxes, narrow spine width, orange/yellow circular logo with letter G, taped/worn edges",
"IMG_4519.jpeg|Right-center of shelf, two matching red boxes standing between the D&D group on the left and the white 'Cheapest Games' spine on the right|Orange 'G' logo visible on both spines|Two identical red boxes with worn/taped edges, orange circular 'G' emblem, appears to be same game duplicated", "IMG_4519.jpeg|Right-center of shelf, two matching red boxes standing between the D&D group on the left and the white 'Cheapest Games' spine on the right|Orange 'G' logo visible on both spines|Two identical red boxes with worn/taped edges, orange circular 'G' emblem, appears to be same game duplicated",
"IMG_4520.jpeg|Middle shelf, between Carcassonne and Evolution: The Beginning||Plain wooden-colored vertical box, no visible text, appears to be a wood grain finish game box or insert", "IMG_4520.jpeg|Middle shelf, between Carcassonne and Evolution: The Beginning||Plain wooden-colored vertical box, no visible text, appears to be a wood grain finish game box or insert",
"IMG_4521.jpeg|Top edge of frame, above the Sleeping Gods box, partially cut off||Dark spines/boxes barely visible at top, too obscured by shadow and cropping to identify", "IMG_4521.jpeg|Top edge of frame, above the Sleeping Gods box, partially cut off||Dark spines/boxes barely visible at top, too obscured by shadow and cropping to identify",
"IMG_4521.jpeg|Top of shelf, above the Sleeping Gods box, partially cut off by frame edge||Dark spines of several boxes, indistinct due to shadow and angle, colors appear dark blue/black and tan",
"IMG_4527.jpeg|Left edge of shelf, partially visible spines to the left of the main game box, on the top shelf|01, T...|Appears to be book-like spines, red and dark covers, stacked vertically, likely not games but partially obscured", "IMG_4527.jpeg|Left edge of shelf, partially visible spines to the left of the main game box, on the top shelf|01, T...|Appears to be book-like spines, red and dark covers, stacked vertically, likely not games but partially obscured",
"IMG_4527.jpeg|Right edge of shelf, behind/beside the Red Dragon Inn box, partially cut off by frame|4, 3|Dark colored spine/box edge, numbers visible but title obscured",
"IMG_4528.jpeg|Top shelf, above Wiz-War box, partially obscured by rolled poster/tube on left side|SLUGFEST GAMES|Red and orange colored box with illustrated fantasy/comic style artwork, only top edge visible",
"IMG_4528.jpeg|Top shelf, back corner above the Wiz-War box; only the publisher logo area is visible, partially obscured by other items|SLUGFEST GAMES|Colorful red/orange box art, only top publisher banner visible, rest of box obscured", "IMG_4528.jpeg|Top shelf, back corner above the Wiz-War box; only the publisher logo area is visible, partially obscured by other items|SLUGFEST GAMES|Colorful red/orange box art, only top publisher banner visible, rest of box obscured",
"IMG_4529.jpeg|Below the 1000-piece puzzle box, spine showing armored warrior artwork, above the face-out Risk Lord of the Rings box|The Middle-earth|Dark battle scene with armored figures, likely same LOTR-themed game box viewed from spine angle", "IMG_4529.jpeg|Below the 1000-piece puzzle box, spine showing armored warrior artwork, above the face-out Risk Lord of the Rings box|The Middle-earth|Dark battle scene with armored figures, likely same LOTR-themed game box viewed from spine angle",
"IMG_4529.jpeg|Middle of stack between the puzzle box and Risk Lord of the Rings, tan/cream colored spine with Asian characters and blue castle icon|unreadable Asian characters, blue icon|Tan spine with small blue building/castle graphic and Asian script text", "IMG_4529.jpeg|Middle of stack between the puzzle box and Risk Lord of the Rings, tan/cream colored spine with Asian characters and blue castle icon|unreadable Asian characters, blue icon|Tan spine with small blue building/castle graphic and Asian script text",
"IMG_4529.jpeg|Top of stack, above the 1000-piece puzzle box, partially cut off at top of frame|V, 3|Dark spines with minimal visible text, one shows a circular 'V' logo, another shows number '3' on purple/dark background; too obscured to identify title",
"IMG_4529.jpeg|Top of stack, second partial box edge, above the puzzle box|0|White/light colored box edge with a circular icon, only corner visible", "IMG_4529.jpeg|Top of stack, second partial box edge, above the puzzle box|0|White/light colored box edge with a circular icon, only corner visible",
"IMG_4529.jpeg|Very top of stack, box behind/above the numbered fragments, purple/dark spine||Dark purple or navy spine, mostly obscured, edge of frame", "IMG_4529.jpeg|Very top of stack, box behind/above the numbered fragments, purple/dark spine||Dark purple or navy spine, mostly obscured, edge of frame",
"IMG_4529.jpeg|Very top of stack, partial box edge visible above the puzzle box|3|Dark colored spine/edge, only a sliver visible, number '3' printed in white", "IMG_4529.jpeg|Very top of stack, partial box edge visible above the puzzle box|3|Dark colored spine/edge, only a sliver visible, number '3' printed in white",
"IMG_4532.jpeg|Top right corner of frame, above Gloomhaven box, partially cut off|GLO...|Fiery orange/red background with dark textured lettering, appears to be another large box possibly related to Gloomhaven expansion",
"IMG_4532.jpeg|Top right corner of image, partially cut off, above and behind the Gloomhaven box|GLO... (possibly part of another title starting with GLO)|Dark box with fiery orange/red background and large white curved text, only edge visible", "IMG_4532.jpeg|Top right corner of image, partially cut off, above and behind the Gloomhaven box|GLO... (possibly part of another title starting with GLO)|Dark box with fiery orange/red background and large white curved text, only edge visible",
"IMG_4535.jpeg|Top shelf, far right edge of frame, next to the last visible Etherfields big box||Black box spine, partially cut off by frame edge, appears similar in size to adjacent Etherfields big boxes",
"IMG_4542.jpeg|Bottom right shelf area, dark spine next to unidentified train-themed box||Dark spine, illegible text, appears to be a board game box based on shape",
"IMG_4542.jpeg|Right side shelf, below CATAN, dark spine with red/white text|T...T / Ride (possible)|Dark red/maroon spine with train-like graphic element, narrow box spine",
"IMG_4542.jpeg|Right side shelf, below Catan box, next to unidentified black-spined game|T...C...|Teal/orange box with train-like graphic, partially obscured, could be a train-themed game", "IMG_4542.jpeg|Right side shelf, below Catan box, next to unidentified black-spined game|T...C...|Teal/orange box with train-like graphic, partially obscured, could be a train-themed game",
"IMG_4542.jpeg|Right side shelf, lower row near 'Catan' box||Dark spine with red accent, text too small/blurry to read", "IMG_4542.jpeg|Right side shelf, lower row near 'Catan' box||Dark spine with red accent, text too small/blurry to read",
"IMG_4542.jpeg|Top area, small red square box with white 'M' logo, near Dragon Land|M|Small red square box with large white letter M, possibly a classic board game logo",
"IMG_4542.jpeg|Top left corner of shelf, partially cut off by frame edge, above the Skip-Bo box|...O / From the Makers of...|Red box edge with white lettering, likely a card game box, mostly obscured",
"IMG_4542.jpeg|Top right shelf, partially hidden behind Dragon Land box|From the Makers of, M|Red box with a large stylized 'M' logo and small icons, likely a Mattel-published card game (possibly UNO), edge cut off by frame", "IMG_4542.jpeg|Top right shelf, partially hidden behind Dragon Land box|From the Makers of, M|Red box with a large stylized 'M' logo and small icons, likely a Mattel-published card game (possibly UNO), edge cut off by frame",
"IMG_4543.jpeg|Background, lower right area behind the hand holding Scrabble Slam, blurred and out of focus||Indistinct colorful box shapes, cannot make out title or artwork details due to heavy blur",
"IMG_4543.jpeg|Right edge of frame, middle row, next to the Scrabble Slam box|CAT...|Dark colored box spine, partially cut off at right edge of photo, red/dark background with light text",
"IMG_4543.jpeg|Top right corner of frame, behind and above the Scrabble Slam box, partially cut off by frame edge|Dogs Kitchen|Small box or book with illustrated cartoon-style cover, colorful, appears to feature a dog character; too small/blurry to confirm if it's a game",
"IMG_4543.jpeg|right edge of image, behind Scrabble Slam box|CAT|Yellow/orange box edge visible, partial text 'CAT' in bold letters, rest obscured by foreground box", "IMG_4543.jpeg|right edge of image, behind Scrabble Slam box|CAT|Yellow/orange box edge visible, partial text 'CAT' in bold letters, rest obscured by foreground box",
"IMG_4543.jpeg|top right corner of image, above and behind the main Scrabble Slam box|Dogs Kitchen|Small blue/purple box partially cut off at top edge, illustrated cartoon-style artwork", "IMG_4543.jpeg|top right corner of image, above and behind the main Scrabble Slam box|Dogs Kitchen|Small blue/purple box partially cut off at top edge, illustrated cartoon-style artwork",
"IMG_4544.jpeg|Background shelf behind the Cat Stax box, blurred and out of focus, appears to be a row of game/book boxes||Multiple colorful spines/boxes including orange, white, and dark tones, too blurry to make out any text", "IMG_4544.jpeg|Background shelf behind the Cat Stax box, blurred and out of focus, appears to be a row of game/book boxes||Multiple colorful spines/boxes including orange, white, and dark tones, too blurry to make out any text",
"IMG_4544.jpeg|Background shelf behind the Cat Stax box, top of frame, multiple stacked boxes with no clearly identifiable neighbors||Blurry stack of colorful boxes (orange, white, dark tones) in background, out of focus, too indistinct to read any text",
"IMG_4545.jpeg|Bottom right corner of image, yellowish box edge visible||Yellow/tan colored box spine, text not legible due to angle and cropping",
"IMG_4545.jpeg|Lower left shelf area, dark colored box partially visible behind hand holding Pocket Farkel||Dark/black box edge visible, no legible text",
"IMG_4545.jpeg|Top right shelf, box with 'Ravensburger' text visible, title obscured|Ravensburger|Colorful box, brand name visible but game title not legible",
"IMG_4545.jpeg|Top shelf, far right edge, partially cut off by frame border|Ravens...|Yellow/orange box with small logo resembling a bird, likely Ravensburger branding but game title not legible", "IMG_4545.jpeg|Top shelf, far right edge, partially cut off by frame border|Ravens...|Yellow/orange box with small logo resembling a bird, likely Ravensburger branding but game title not legible",
"IMG_4545.jpeg|Top shelf, right side, above the 'Ravensburger' labeled box, partial title visible at top edge of frame|DRAGON ...and|Orange/red box with dragon and fantasy artwork, cut off at top of image",
"IMG_4545.jpeg|Top shelf, upper right corner, above and behind the Pocket Farkel tin, to the right of 'Bugs in the Kitchen'|DRAGON, land|Colorful fantasy-style box art, orange/red hues, dragon imagery visible, title cut off at top edge of frame", "IMG_4545.jpeg|Top shelf, upper right corner, above and behind the Pocket Farkel tin, to the right of 'Bugs in the Kitchen'|DRAGON, land|Colorful fantasy-style box art, orange/red hues, dragon imagery visible, title cut off at top edge of frame",
"IMG_4547.jpeg|Top area, right of the maze-like box, small orange/red box||Small red/orange box, text illegible, largely obscured", "IMG_4547.jpeg|Top area, right of the maze-like box, small orange/red box||Small red/orange box, text illegible, largely obscured",
"IMG_4547.jpeg|Top center-right, small red/orange box near top edge||Small reddish box with indistinct artwork, mostly cut off at top frame", "IMG_4547.jpeg|Top center-right, small red/orange box near top edge||Small reddish box with indistinct artwork, mostly cut off at top frame",
"IMG_4547.jpeg|Top left area, next to the '...AZE' box, second spine from left|UTTLES or similar|Bright blue box with light green accents, small toy-like image visible on top",
"IMG_4547.jpeg|Top left corner of image, partially visible behind main box|AZE|Blue and green box, appears to be a puzzle or maze-themed game, mostly cut off by frame edge", "IMG_4547.jpeg|Top left corner of image, partially visible behind main box|AZE|Blue and green box, appears to be a puzzle or maze-themed game, mostly cut off by frame edge",
"IMG_4547.jpeg|Top left corner of shelf, above the main held box, leftmost visible spine|...AZE|Blue and green box, partial text visible, cut off at frame edge",
"IMG_4547.jpeg|Top right area, red-orange box between the dark box and top edge|Roll Pla...|Red/orange box with illustrated artwork, text partially cut off",
"IMG_4547.jpeg|Top right corner of image|Dark T...|Dark blue/black box with white text, appears to be 'Dark T' something, possibly 'Dark Tower' or similar, cut off at frame edge", "IMG_4547.jpeg|Top right corner of image|Dark T...|Dark blue/black box with white text, appears to be 'Dark T' something, possibly 'Dark Tower' or similar, cut off at frame edge",
"IMG_4547.jpeg|Top right corner of shelf, above and to the right of the main held box|Dark Terr... or Dark Territory|Dark navy/black box with red and white text, partially obscured by other boxes",
"IMG_4547.jpeg|Top right, near 'Dark T' box||Colorful box with red/orange tones, illustration unclear, partially obscured by other boxes", "IMG_4547.jpeg|Top right, near 'Dark T' box||Colorful box with red/orange tones, illustration unclear, partially obscured by other boxes",
"IMG_4552.jpeg|Left edge of frame, vertical spine, sharply cropped|Giv... / Ci...|Dark spine with white lettering, partially visible artwork, cropped by frame edge", "IMG_4552.jpeg|Left edge of frame, vertical spine, sharply cropped|Giv... / Ci...|Dark spine with white lettering, partially visible artwork, cropped by frame edge",
"IMG_4552.jpeg|Top left corner of frame, behind and above the Dark Cults package, no clearly identified games adjacent due to cropping|'Giv...'|Dark colored box with white/red text fragment, likely a card or board game box mostly out of frame", "IMG_4552.jpeg|Top left corner of frame, behind and above the Dark Cults package, no clearly identified games adjacent due to cropping|'Giv...'|Dark colored box with white/red text fragment, likely a card or board game box mostly out of frame",
+12
View File
@@ -66,6 +66,18 @@ class Config:
def games_path(self) -> Path: def games_path(self) -> Path:
return self.data_dir / "games.json" return self.data_dir / "games.json"
@property
def title_edits_path(self) -> Path:
# human corrections to extracted reads (misspellings, known cues) —
# replayed on every titles.json rebuild
return self.data_dir / "title_edits.json"
@property
def title_splits_path(self) -> Path:
# titles the human declared to be MULTIPLE physical copies: extract
# and resolve dedupe must never cross-photo-merge them again
return self.data_dir / "title_splits.json"
@property @property
def dismissed_path(self) -> Path: def dismissed_path(self) -> Path:
return self.data_dir / "unidentified_dismissed.json" return self.data_dir / "unidentified_dismissed.json"
+184 -8
View File
@@ -240,15 +240,142 @@ def _merge(a: dict, b: dict) -> dict:
return merged return merged
def dedupe_entries(entries: list[dict]) -> list[dict]: # the fields a human correction may override on an extracted entry
"""Collapse same-normalized-title sightings unless their cues conflict.""" EDIT_FIELDS = (
"title_raw",
"publisher_hint",
"edition_hint",
"year_hint",
"language_hint",
"art_notes",
)
def _load_store(path: Path) -> list:
"""Curation stores hold irreplaceable human decisions and are committed
(merge conflicts are a realistic corruption vector) — so a broken file
must stop the pipeline loudly, never quietly reset it."""
if not path.exists():
return []
try:
return json.loads(path.read_text())
except json.JSONDecodeError as err:
raise ValueError(
f"{path} is corrupt ({err}) — fix or delete it; it holds human "
"review decisions, so check git history before deleting"
) from err
def load_title_edits(path: Path) -> list[dict]:
"""Human corrections to raw reads (fixed misspellings, cues the owner
knows offhand). Each record: {"match": <title as displayed when the fix
was made>, "photos": [...] to target one copy (optional), <EDIT_FIELDS
to override>}. Applied on every rebuild, before dedupe."""
return _load_store(path)
def record_title_edit(path: Path, record: dict) -> None:
existing = load_title_edits(path)
if existing and existing[-1] == record:
return # a retried request must not double-record
atomic_write_text(
path, json.dumps([*existing, record], indent=2, ensure_ascii=False) + "\n"
)
def apply_title_edits(entries: list[dict], edits: list[dict]) -> list[dict]:
"""Apply stored corrections in order. Matching is by the title an entry
CURRENTLY carries, so a later edit made against an earlier edit's result
chains naturally; each record applies at most once per entry."""
if not edits:
return entries
for entry in entries:
applied: set[int] = set()
while True:
norm = normalize_title(entry["title_raw"])
photos = set(entry.get("source_photos") or [])
hit = next(
(
i
for i, e in enumerate(edits)
if i not in applied
and normalize_title(e["match"]) == norm
and (not e.get("photos") or set(e["photos"]) & photos)
),
None,
)
if hit is None:
break
applied.add(hit)
for key in EDIT_FIELDS:
if key in edits[hit]:
entry[key] = edits[hit][key]
return entries
def load_title_splits(path: Path) -> list[dict]:
"""The human's split-into-copies decisions — durable: they must survive
extract rebuilds and resolve --force. Each stored record is
{"title": ..., "photos": [...]} scoping the split to the sightings that
were on the split line (a bare string is legacy: unscoped). Returns
[{"norm": <normalized title>, "photos": set | None}]."""
records = []
for item in _load_store(path):
if isinstance(item, str):
records.append({"norm": normalize_title(item), "photos": None})
else:
photos = item.get("photos")
records.append(
{
"norm": normalize_title(item["title"]),
"photos": set(photos) if photos else None,
}
)
return records
def is_split(norm: str, photos, splits: list[dict]) -> bool:
"""Does a split decision cover this (normalized title, photo set)?
Photo-scoped records only bind sightings from the photos that were on
the split line — a same-named different edition keeps deduping."""
photo_set = set(photos or [])
return any(
r["norm"] == norm and (r["photos"] is None or r["photos"] & photo_set)
for r in splits
)
def record_title_split(path: Path, title: str, photos: list[str] | None = None) -> None:
if is_split(normalize_title(title), photos or [], load_title_splits(path)):
return
existing = _load_store(path)
record = {"title": title, "photos": sorted(photos) if photos else None}
atomic_write_text(
path, json.dumps([*existing, record], indent=2, ensure_ascii=False) + "\n"
)
def dedupe_entries(entries: list[dict], splits: list[dict] | None = None) -> list[dict]:
"""Collapse same-normalized-title sightings unless their cues conflict —
or unless the human declared them split (several physical copies):
split sightings stay one entry per photo and never absorb new ones."""
splits = splits or []
def covered(e: dict) -> bool:
return is_split(e["title_normalized"], e.get("source_photos"), splits)
result: list[dict] = [] result: list[dict] = []
for entry in entries: for entry in entries:
entry = {**entry, "title_normalized": normalize_title(entry["title_raw"])} entry = {**entry, "title_normalized": normalize_title(entry["title_raw"])}
if covered(entry):
result.append(entry)
continue
for existing in result: for existing in result:
if existing["title_normalized"] == entry[ if (
"title_normalized" existing["title_normalized"] == entry["title_normalized"]
] and not cues_conflict(existing, entry): and not covered(existing)
and not cues_conflict(existing, entry)
):
existing.update(_merge(existing, entry)) existing.update(_merge(existing, entry))
break break
else: else:
@@ -257,7 +384,11 @@ def dedupe_entries(entries: list[dict]) -> list[dict]:
def rebuild_artifacts( def rebuild_artifacts(
raw_dir: Path, titles_path: Path, unidentified_path: Path raw_dir: Path,
titles_path: Path,
unidentified_path: Path,
splits: list[dict] | None = None,
edits: list[dict] | None = None,
) -> tuple[list[dict], dict[str, list[dict]]]: ) -> tuple[list[dict], dict[str, list[dict]]]:
"""Regenerate titles.json and unidentified.json from the per-photo raw """Regenerate titles.json and unidentified.json from the per-photo raw
cache. A raw file is either an object with titles/unidentified or a bare cache. A raw file is either an object with titles/unidentified or a bare
@@ -279,7 +410,7 @@ def rebuild_artifacts(
photo = raw_file.name.removesuffix(".json") photo = raw_file.name.removesuffix(".json")
if data.get("unidentified"): if data.get("unidentified"):
unidentified[photo] = data["unidentified"] unidentified[photo] = data["unidentified"]
deduped = dedupe_entries(entries) deduped = dedupe_entries(apply_title_edits(entries, edits or []), splits)
titles_path.parent.mkdir(parents=True, exist_ok=True) titles_path.parent.mkdir(parents=True, exist_ok=True)
atomic_write_text( atomic_write_text(
titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n" titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n"
@@ -290,6 +421,47 @@ def rebuild_artifacts(
return deduped, unidentified return deduped, unidentified
def replay_titles(cfg: Config) -> None:
"""Re-derive titles.json after a stored split or edit changed the rules.
Prefers the raw caches (per-photo cue fidelity); without them (trimmed
or hand-built data) it replays the current titles.json entries through
the same edit + dedupe path, exploding multi-photo entries first so
photo-level splits and targeted edits can take hold."""
splits = load_title_splits(cfg.title_splits_path)
edits = load_title_edits(cfg.title_edits_path)
raw_dir = cfg.extract_raw_dir
raw_photos = (
{f.name.removesuffix(".json") for f in raw_dir.glob("*.json")}
if raw_dir.is_dir()
else set()
)
known_photos: set[str] = set()
if cfg.titles_path.exists():
for entry in json.loads(cfg.titles_path.read_text()):
known_photos.update(entry.get("source_photos") or [])
# raw caches are gitignored: a fresh clone (or a killed/partial extract)
# can have FEWER raw files than titles.json has photos — rebuilding from
# them would silently truncate the committed catalog
if raw_photos and raw_photos >= known_photos:
rebuild_artifacts(
raw_dir, cfg.titles_path, cfg.unidentified_path, splits, edits
)
return
if not cfg.titles_path.exists():
return
exploded: list[dict] = []
for entry in json.loads(cfg.titles_path.read_text()):
photos = entry.get("source_photos") or []
if len(photos) > 1:
exploded.extend({**entry, "source_photos": [p]} for p in photos)
else:
exploded.append(dict(entry))
deduped = dedupe_entries(apply_title_edits(exploded, edits), splits)
atomic_write_text(
cfg.titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n"
)
def run_extract( def run_extract(
cfg: Config, cfg: Config,
*, *,
@@ -370,7 +542,11 @@ def run_extract(
typer.echo(f" {photo.name}: {len(result['titles'])} title(s){note}") typer.echo(f" {photo.name}: {len(result['titles'])} title(s){note}")
deduped, unidentified = rebuild_artifacts( deduped, unidentified = rebuild_artifacts(
raw_dir, cfg.titles_path, cfg.unidentified_path raw_dir,
cfg.titles_path,
cfg.unidentified_path,
load_title_splits(cfg.title_splits_path),
load_title_edits(cfg.title_edits_path),
) )
typer.echo(f"Wrote {len(deduped)} unique title(s) to {cfg.titles_path}.") typer.echo(f"Wrote {len(deduped)} unique title(s) to {cfg.titles_path}.")
if failed: if failed:
+15 -9
View File
@@ -22,7 +22,7 @@ from rapidfuzz import fuzz
from bggpipe.bgg_client import BGGAuthError, BGGClient, BGGQueueTimeout, client_for from bggpipe.bgg_client import BGGAuthError, BGGClient, BGGQueueTimeout, client_for
from bggpipe.config import Config from bggpipe.config import Config
from bggpipe.extract import cues_conflict from bggpipe.extract import cues_conflict, is_split, load_title_splits
from bggpipe.fsio import atomic_write_csv from bggpipe.fsio import atomic_write_csv
from bggpipe.models import ( from bggpipe.models import (
RECOGNIZED_MATCH_STATUSES, RECOGNIZED_MATCH_STATUSES,
@@ -406,9 +406,7 @@ def resolve_entry(client: BGGClient, entry: TitleEntry) -> MatchRow:
# type=rpgitem (same API, same token). A hit becomes a LOCAL # type=rpgitem (same API, same token). A hit becomes a LOCAL
# library citizen: identified and enriched, never uploaded (diff # library citizen: identified and enriched, never uploaded (diff
# routes rpgitem rows to local_only). # routes rpgitem rows to local_only).
cands = _plausible_candidates( cands = _plausible_candidates(client, entry, entry.title_raw, types="rpgitem")
client, entry, entry.title_raw, types="rpgitem"
)
if not cands: if not cands:
for head in _truncation_heads(entry.title_raw): for head in _truncation_heads(entry.title_raw):
cands = _plausible_candidates( cands = _plausible_candidates(
@@ -430,7 +428,11 @@ class MergeEvent:
bgg_id: str bgg_id: str
def dedupe_matches(rows: list[dict], titles: list[TitleEntry]) -> list[MergeEvent]: def dedupe_matches(
rows: list[dict],
titles: list[TitleEntry],
splits: list[dict] | None = None,
) -> list[MergeEvent]:
"""Post-resolve dedupe: rows resolving to the same (bgg_id, version_id — """Post-resolve dedupe: rows resolving to the same (bgg_id, version_id —
or both version-unknown) are the same physical game seen twice (a typo or both version-unknown) are the same physical game seen twice (a typo
read, a partial spine) UNLESS their extraction cues conflict, which read, a partial spine) UNLESS their extraction cues conflict, which
@@ -464,6 +466,12 @@ def dedupe_matches(rows: list[dict], titles: list[TitleEntry]) -> list[MergeEven
# re-running resolve must never overturn that (spec: re-runs # re-running resolve must never overturn that (spec: re-runs
# lose no work, least of all review decisions) # lose no work, least of all review decisions)
continue continue
if is_split(
normalize_title(row["title_raw"]),
row["source_photos"].split(";"),
splits or [],
):
continue # human-split copies: per-photo rows stay separate
key = ( key = (
row["bgg_id"], row["bgg_id"],
row["version_id"] if is_confident_version(row) else "", row["version_id"] if is_confident_version(row) else "",
@@ -603,9 +611,7 @@ def run_resolve(
row_dict = paired_by_id.get(id(entry)) row_dict = paired_by_id.get(id(entry))
if row_dict is not None: if row_dict is not None:
photos = ";".join(entry.source_photos) photos = ";".join(entry.source_photos)
if row_dict["source_photos"] != photos and not row_dict.get( if row_dict["source_photos"] != photos and not row_dict.get("dedupe_veto"):
"dedupe_veto"
):
# provenance follows the entry — except on split/vetoed rows, # provenance follows the entry — except on split/vetoed rows,
# whose per-copy photo sets are human-authored # whose per-copy photo sets are human-authored
row_dict["source_photos"] = photos row_dict["source_photos"] = photos
@@ -639,7 +645,7 @@ def run_resolve(
typer.echo(f" {entry.title_raw!r} -> {row.match_status}: {detail}{version}") typer.echo(f" {entry.title_raw!r} -> {row.match_status}: {detail}{version}")
all_rows = existing_rows + [row.to_csv() for row in new_rows] all_rows = existing_rows + [row.to_csv() for row in new_rows]
merges = dedupe_matches(all_rows, entries) merges = dedupe_matches(all_rows, entries, load_title_splits(cfg.title_splits_path))
if new_rows or photos_updated or merges: if new_rows or photos_updated or merges:
write_matches(cfg.matches_path, all_rows) # atomic full rewrite write_matches(cfg.matches_path, all_rows) # atomic full rewrite
if merges: if merges:
+42
View File
@@ -28,6 +28,7 @@ from bggpipe.models import (
UNDECIDED_MATCH_STATUSES, UNDECIDED_MATCH_STATUSES,
BGGResponseError, BGGResponseError,
) )
from bggpipe.normalize import normalize_title
from bggpipe.resolve import ( from bggpipe.resolve import (
MatchRow, MatchRow,
TitleEntry, TitleEntry,
@@ -158,6 +159,12 @@ class ReviewSession:
"you decided — decision NOT saved" "you decided — decision NOT saved"
) )
return return
self._write_rows()
def _write_rows(self) -> None:
"""Atomic write of self.rows exactly as they stand — no reload, so
a caller that just derived self.rows from a fresh reload can't have
its result silently replaced before the write."""
try: try:
own_mtime = write_matches(self.cfg.matches_path, self.rows) own_mtime = write_matches(self.cfg.matches_path, self.rows)
except OSError: except OSError:
@@ -258,6 +265,41 @@ class ReviewSession:
self._save(copies[0]) self._save(copies[0])
return copies return copies
def drop_rows(
self,
title_raw: str,
photos: list[str] | None = None,
new_title: str | None = None,
) -> int:
"""An edit invalidated these rows — the BGG match was made against
the uncorrected read. Remove them so resolve re-queries with the
fix; `photos` narrows the cull to one copy of a split title.
Rows carrying a human veto (dedupe_veto) are never dropped — the
match itself was human-vetted, and dropping would erase the veto;
a rename updates their title in place so they follow the entry."""
self.reload_if_changed()
norm = normalize_title(title_raw)
keep: list[dict] = []
dropped = renamed = 0
for row in self.rows:
targeted = normalize_title(row["title_raw"]) == norm and (
photos is None
or bool({p for p in row["source_photos"].split(";") if p} & set(photos))
)
if not targeted:
keep.append(row)
elif row.get("dedupe_veto"):
if new_title and row["title_raw"] != new_title:
row["title_raw"] = new_title
renamed += 1
keep.append(row)
else:
dropped += 1
if dropped or renamed:
self.rows = keep
self._write_rows()
return dropped
def veto_merge(self, row: dict) -> None: def veto_merge(self, row: dict) -> None:
"""The human says these are NOT the same physical game: restore the """The human says these are NOT the same physical game: restore the
row as a distinct, human-confirmed match.""" row as a distinct, human-confirmed match."""
+21
View File
@@ -310,6 +310,27 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
text-underline-offset: 2px; text-underline-offset: 2px;
} }
.catalog a:hover { color: var(--accent-ink); } .catalog a:hover { color: var(--accent-ink); }
.catalog td.actions { text-align: right; white-space: nowrap; }
.catalog td.actions button {
font: inherit; font-size: .78rem; border: var(--line); background: #fff;
border-radius: var(--radius); padding: .2rem .6rem; cursor: pointer;
box-shadow: none;
}
.catalog td.actions button:hover { background: var(--board); }
.catalog tr.editrow td { background: #fff; border-top: none; padding: .2rem .5rem .7rem; }
.editform { display: flex; gap: .7rem; align-items: end; flex-wrap: wrap; }
.editform label {
display: flex; flex-direction: column; gap: .15rem;
font-size: .72rem; text-transform: uppercase; letter-spacing: .06em;
color: var(--ink-soft);
}
.editform input {
font: inherit; font-size: .85rem; color: var(--ink);
border: 2px solid var(--board-edge); border-radius: var(--radius);
padding: .25rem .45rem; background: #fff;
}
.editactions { display: flex; gap: .5rem; }
.edithint { font-size: .78rem; color: var(--ink-soft); align-self: center; }
.chip { .chip {
font-size: .68rem; text-transform: uppercase; letter-spacing: .06em; font-size: .68rem; text-transform: uppercase; letter-spacing: .06em;
border-radius: 999px; padding: .1rem .55rem; white-space: nowrap; border-radius: 999px; padding: .1rem .55rem; white-space: nowrap;
+64 -7
View File
@@ -7,12 +7,37 @@
<script> <script>
"use strict"; "use strict";
let CATALOG = []; let CATALOG = [];
let EDITING = null; // lineKey of the row whose editor is open
function lineKey(c) { return c.title_raw + "|" + c.photos.join(";"); }
function editorRow(c) {
const cue = c.cues || {};
return `
<tr class="editrow"><td colspan="5">
<form class="editform" data-title="${esc(c.title_raw)}"
data-photos="${esc(c.photos.join(";"))}">
<label>Title <input name="title" value="${esc(c.title_raw)}" required></label>
<label>Publisher <input name="publisher" value="${esc(cue.publisher || "")}"></label>
<label>Edition <input name="edition" value="${esc(cue.edition || "")}"></label>
<label>Year <input name="year" value="${esc(cue.year ?? "")}" inputmode="numeric" size="6"></label>
<label>Language <input name="language" value="${esc(cue.language || "")}"></label>
<span class="editactions">
<button type="submit" class="primary">save</button>
<button type="button" class="canceledit">cancel</button>
</span>
<span class="edithint">saving re-queues this title for resolve with the corrected data</span>
</form>
</td></tr>`;
}
function render() { function render() {
const q = document.getElementById("catsearch").value.trim().toLowerCase(); const q = document.getElementById("catsearch").value.trim().toLowerCase();
const sorted = [...CATALOG].sort((a, b) =>
a.title_raw.localeCompare(b.title_raw, undefined, { sensitivity: "base" }));
const rows = q const rows = q
? CATALOG.filter(c => (c.title_raw + " " + c.bgg_name).toLowerCase().includes(q)) ? sorted.filter(c => (c.title_raw + " " + c.bgg_name).toLowerCase().includes(q))
: CATALOG; : sorted;
document.getElementById("catcount").innerHTML = document.getElementById("catcount").innerHTML =
`<b>${rows.length}</b> of <b>${CATALOG.length}</b> title(s)`; `<b>${rows.length}</b> of <b>${CATALOG.length}</b> title(s)`;
document.getElementById("catbody").innerHTML = rows.length document.getElementById("catbody").innerHTML = rows.length
@@ -27,13 +52,17 @@ function render() {
<td class="meta">${c.photos.map(p => <td class="meta">${c.photos.map(p =>
`<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>` `<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>`
).join(", ")}</td> ).join(", ")}</td>
<td>${c.can_split <td class="actions">${c.can_split
? `<button class="split" data-title="${esc(c.title_raw)}" ? `<button class="split" data-title="${esc(c.title_raw)}"
data-photos="${esc(c.photos.join(";"))}" data-rowix="${c.row_ix}" data-photos="${esc(c.photos.join(";"))}"
data-rowix="${c.row_ix ?? ""}"
title="one line, several boxes? make each photo its own copy"> title="one line, several boxes? make each photo its own copy">
split into copies</button>` split into copies</button>`
: ""}</td> : ""}
</tr>`).join("") + `</table></div>` <button class="edit" data-key="${esc(lineKey(c))}"
title="fix a misread title or add cues you already know">edit</button>
</td>
</tr>` + (EDITING === lineKey(c) ? editorRow(c) : "")).join("") + `</table></div>`
: `<p class="empty">${CATALOG.length : `<p class="empty">${CATALOG.length
? "No titles match that filter." ? "No titles match that filter."
: `Nothing extracted yet — start on the <a href="/photos">photos page</a>.`}</p>`; : `Nothing extracted yet — start on the <a href="/photos">photos page</a>.`}</p>`;
@@ -42,6 +71,7 @@ function render() {
let LAST = null; let LAST = null;
async function refresh() { async function refresh() {
const state = await fetchJSON("/api/state"); const state = await fetchJSON("/api/state");
if (EDITING) return; // never repaint under an open editor
const payload = JSON.stringify(state.catalog); const payload = JSON.stringify(state.catalog);
if (payload === LAST) return; if (payload === LAST) return;
LAST = payload; LAST = payload;
@@ -50,6 +80,14 @@ async function refresh() {
} }
document.getElementById("catbody").addEventListener("click", async e => { document.getElementById("catbody").addEventListener("click", async e => {
const cancel = e.target.closest("button.canceledit");
if (cancel) { EDITING = null; LAST = null; render(); refresh().catch(() => {}); return; }
const edit = e.target.closest("button.edit");
if (edit) {
EDITING = EDITING === edit.dataset.key ? null : edit.dataset.key;
render();
return;
}
const b = e.target.closest("button.split"); const b = e.target.closest("button.split");
if (!b) return; if (!b) return;
const n = b.dataset.photos.split(";").length; const n = b.dataset.photos.split(";").length;
@@ -58,10 +96,29 @@ document.getElementById("catbody").addEventListener("click", async e => {
const res = await apiPost("/api/split", { const res = await apiPost("/api/split", {
title_raw: b.dataset.title, title_raw: b.dataset.title,
source_photos: b.dataset.photos, source_photos: b.dataset.photos,
row_ix: Number(b.dataset.rowix), row_ix: b.dataset.rowix === "" ? null : Number(b.dataset.rowix),
}); });
if (res) refresh().catch(() => {}); if (res) refresh().catch(() => {});
}); });
document.getElementById("catbody").addEventListener("submit", async e => {
const f = e.target.closest("form.editform");
if (!f) return;
e.preventDefault();
const orig = CATALOG.find(c => lineKey(c) === EDITING) || {};
const cue = orig.cues || {};
const v = name => f.elements[name].value;
const body = { title_raw: f.dataset.title, source_photos: f.dataset.photos };
if (v("title").trim() !== f.dataset.title) body.title_new = v("title").trim();
if (v("publisher") !== (cue.publisher || "")) body.publisher = v("publisher");
if (v("edition") !== (cue.edition || "")) body.edition = v("edition");
if (v("year") !== String(cue.year ?? "")) body.year = v("year");
if (v("language") !== (cue.language || "")) body.language = v("language");
if (Object.keys(body).length <= 2) { EDITING = null; render(); return; }
const res = await apiPost("/api/edit-title", body);
if (res) { EDITING = null; LAST = null; refresh().catch(() => {}); }
});
document.getElementById("catsearch").addEventListener("input", render); document.getElementById("catsearch").addEventListener("input", render);
refresh().catch(err => errorBanner(err.message || err)); refresh().catch(err => errorBanner(err.message || err));
pollLoop(refresh, 5000, () => showBanner("")); pollLoop(refresh, 5000, () => showBanner(""));
+175 -8
View File
@@ -38,12 +38,21 @@ from rich.console import Console
from bggpipe.bgg_client import BGGClient, cached_paths from bggpipe.bgg_client import BGGClient, cached_paths
from bggpipe.config import DEFAULT_REVIEW_PORT, Config from bggpipe.config import DEFAULT_REVIEW_PORT, Config
from bggpipe.extract import (
is_split,
load_title_splits,
record_title_edit,
record_title_split,
replay_titles,
)
from bggpipe.fsio import atomic_write_bytes, atomic_write_text from bggpipe.fsio import atomic_write_bytes, atomic_write_text
from bggpipe.jobs import JobRunner from bggpipe.jobs import JobRunner
from bggpipe.models import ( from bggpipe.models import (
CONFIDENT_VERSION_STATUSES, CONFIDENT_VERSION_STATUSES,
RECOGNIZED_MATCH_STATUSES, RECOGNIZED_MATCH_STATUSES,
) )
from bggpipe.normalize import normalize_title
from bggpipe.resolve import TitleEntry
from bggpipe.review import ReviewSession from bggpipe.review import ReviewSession
@@ -148,6 +157,20 @@ class SplitBody(BaseModel):
row_ix: int | None = None row_ix: int | None = None
class EditBody(BaseModel):
"""A human correction to an extracted read. None = leave that field
alone; for the cue fields, empty string = clear it (a corrected title
may not be empty)."""
title_raw: str
source_photos: str = ""
title_new: str | None = None
publisher: str | None = None
edition: str | None = None
year: str | None = None
language: str | None = None
class RunBody(BaseModel): class RunBody(BaseModel):
dry_run: bool = True # upload only; the safe direction is the default dry_run: bool = True # upload only; the safe direction is the default
limit: int | None = None limit: int | None = None
@@ -304,6 +327,18 @@ def create_app(
app_warnings.append(note) app_warnings.append(note)
return {} return {}
def _find_entry(title_raw: str, source_photos: str) -> TitleEntry | None:
photos = [p for p in source_photos.split(";") if p]
return next(
(
e
for e in session.titles
if e.title_raw == title_raw
and (not photos or list(e.source_photos) == photos)
),
None,
)
def find_row(title_raw: str, source_photos: str, row_ix: int | None = None) -> dict: def find_row(title_raw: str, source_photos: str, row_ix: int | None = None) -> dict:
freshen() freshen()
row = session.find_row(title_raw, source_photos, row_ix) row = session.find_row(title_raw, source_photos, row_ix)
@@ -362,6 +397,9 @@ def create_app(
catalog = [] catalog = []
def catalog_line(entry, row) -> dict: def catalog_line(entry, row) -> dict:
cue_entry = entry or (
_find_entry(row["title_raw"], row["source_photos"]) if row else None
)
return { return {
"title_raw": entry.title_raw if entry else row["title_raw"], "title_raw": entry.title_raw if entry else row["title_raw"],
"confidence": entry.confidence if entry else "", "confidence": entry.confidence if entry else "",
@@ -377,12 +415,21 @@ def create_app(
"version_name": row["version_name"] if row else "", "version_name": row["version_name"] if row else "",
"merged_into": row.get("merged_into", "") if row else "", "merged_into": row.get("merged_into", "") if row else "",
"row_ix": _ix_of(session.rows, row) if row else None, "row_ix": _ix_of(session.rows, row) if row else None,
"cues": {
"publisher": cue_entry.publisher_hint if cue_entry else "",
"edition": cue_entry.edition_hint if cue_entry else "",
"year": cue_entry.year_hint if cue_entry else None,
"language": cue_entry.language_hint if cue_entry else "",
},
"can_split": bool( "can_split": bool(
row row
and row["match_status"] in RECOGNIZED_MATCH_STATUSES and row["match_status"] in RECOGNIZED_MATCH_STATUSES
and not row.get("dedupe_veto") and not row.get("dedupe_veto")
and len(row["source_photos"].split(";")) > 1 and len(row["source_photos"].split(";")) > 1
), )
# no matches row yet (title still awaiting resolve): the
# split is a titles.json decision, no BGG data at stake
or bool(not row and entry and len(entry.source_photos) > 1),
"split_copy": bool(row and row.get("dedupe_veto")), "split_copy": bool(row and row.get("dedupe_veto")),
} }
@@ -390,8 +437,10 @@ def create_app(
same_title = rows_by_title.get(entry.title_raw, []) same_title = rows_by_title.get(entry.title_raw, [])
ix = title_seen[entry.title_raw] ix = title_seen[entry.title_raw]
title_seen[entry.title_raw] += 1 title_seen[entry.title_raw] += 1
# positional pairing, same rule as run_resolve: the ix-th entry # positional pairing: the ix-th entry of a title reports the
# of a title reports the ix-th row of that title # ix-th row of that title (run_resolve pairs photo-overlap
# first, but split entries and split rows both keep sorted
# photo order, so the ordinals line up)
row = same_title[ix] if ix < len(same_title) else None row = same_title[ix] if ix < len(same_title) else None
# a split row's photo set is narrower than its entry's — show # a split row's photo set is narrower than its entry's — show
# the row's own photos for split copies # the row's own photos for split copies
@@ -700,11 +749,129 @@ def create_app(
with lock: with lock:
revision["n"] += 1 revision["n"] += 1
_refuse_if_rewriting() _refuse_if_rewriting()
row = find_row(body.title_raw, body.source_photos, body.row_ix) freshen()
try: row = session.find_row(body.title_raw, body.source_photos, body.row_ix)
session.split_row(row) if row is not None:
except ValueError as err: try:
raise HTTPException(400, str(err)) from err copies = session.split_row(row)
except ValueError as err:
raise HTTPException(400, str(err)) from err
if not copies:
# split_row adopted nothing: matches.csv was rewritten
# underneath — recording the store now would report a
# success that didn't happen
raise HTTPException(
409,
"matches.csv changed while you split — nothing "
"changed; retry from the refreshed page",
)
else:
# no matches row yet — the title is still awaiting resolve;
# splitting is purely a titles.json (extraction) decision
entry = _find_entry(body.title_raw, body.source_photos)
if entry is None:
raise HTTPException(
404, "title not found — titles.json changed underneath?"
)
if len(entry.source_photos) < 2:
raise HTTPException(
400, "only a multi-photo title can be split into copies"
)
# persist the decision so every future extract rebuild and
# resolve dedupe keeps the copies apart, then re-derive
# titles.json so each copy carries its own photo's cues
record_title_split(
cfg.title_splits_path,
body.title_raw,
[p for p in body.source_photos.split(";") if p],
)
replay_titles(cfg)
return state()
@app.post("/api/edit-title")
def api_edit_title(body: EditBody) -> dict:
with lock:
revision["n"] += 1
_refuse_if_rewriting()
freshen()
entry = _find_entry(body.title_raw, body.source_photos)
if entry is None:
raise HTTPException(
404, "title not found — titles.json changed underneath?"
)
record: dict = {"match": body.title_raw}
photos = [p for p in body.source_photos.split(";") if p]
norm = normalize_title(body.title_raw)
# scope by NORMALIZED title: stored edits apply by normalized
# match, so raw-equality counting here would let an edit bleed
# onto a differently-cased sighting of another edition
same_norm = [
e for e in session.titles if normalize_title(e.title_raw) == norm
]
if photos and len(same_norm) > 1:
# several copies/editions share this title: the fix targets
# only the copy the human was looking at
record["photos"] = photos
if (
sum(
1
for e in same_norm
if list(e.source_photos) == list(entry.source_photos)
)
> 1
):
# two same-named editions seen in the same photo(s): the
# store can only target (title, photos), so an edit would
# hit both — refuse rather than corrupt the sibling
raise HTTPException(
409,
"two entries share this exact title and photo set — an "
"edit cannot target just one; resolve or reject one of "
"them first",
)
if body.title_new is not None:
corrected = body.title_new.strip()
if not corrected:
raise HTTPException(400, "corrected title cannot be empty")
if corrected != body.title_raw:
record["title_raw"] = corrected
for key, value in (
("publisher_hint", body.publisher),
("edition_hint", body.edition),
("language_hint", body.language),
):
if value is not None:
record[key] = value.strip()
if body.year is not None:
year = body.year.strip()
if year and not year.isdigit():
raise HTTPException(400, "year must be a number")
record["year_hint"] = int(year) if year else None
if not (record.keys() - {"match", "photos"}):
raise HTTPException(400, "nothing to change")
# write order is crash-safety: drop the stale rows first (worst
# case on a crash: resolve recreates them from the uncorrected
# entry), then the durable record (replayed by every future
# rebuild), then the titles.json replay
session.drop_rows(
body.title_raw,
record.get("photos"),
new_title=record.get("title_raw"),
)
if "title_raw" in record and is_split(
norm,
entry.source_photos,
load_title_splits(cfg.title_splits_path),
):
# a renamed split copy must stay under split protection —
# the old record keys the OLD normalized title
record_title_split(
cfg.title_splits_path,
record["title_raw"],
list(entry.source_photos),
)
record_title_edit(cfg.title_edits_path, record)
replay_titles(cfg)
return state() return state()
@app.post("/api/veto-merge") @app.post("/api/veto-merge")
+136
View File
@@ -12,9 +12,16 @@ import typer
from bggpipe.config import Config from bggpipe.config import Config
from bggpipe.extract import ( from bggpipe.extract import (
apply_title_edits,
dedupe_entries, dedupe_entries,
load_title_edits,
load_title_splits,
parse_vision_response, parse_vision_response,
prepare_image, prepare_image,
rebuild_artifacts,
record_title_edit,
record_title_split,
replay_titles,
run_extract, run_extract,
) )
@@ -167,6 +174,135 @@ def test_dedupe_conflicting_years_stay_separate():
assert len(deduped) == 2 assert len(deduped) == 2
def test_dedupe_split_titles_never_merge():
deduped = dedupe_entries(
[_entry("Wiz-War", "a.jpg"), _entry("Wiz-War", "b.jpg")],
splits=[{"norm": "wiz war", "photos": None}],
)
assert len(deduped) == 2 # the human said: separate physical copies
def test_photo_scoped_split_spares_other_editions():
# splitting the copies seen in a/b must not force-split a same-named
# different edition (c/d, kept separate by its conflicting cue)
entries = [
_entry("Carcassonne", "a.jpg", publisher_hint="Rio Grande"),
_entry("Carcassonne", "b.jpg", publisher_hint="Rio Grande"),
_entry("Carcassonne", "c.jpg", publisher_hint="Z-Man"),
_entry("Carcassonne", "d.jpg", publisher_hint="Z-Man"),
]
splits = [{"norm": "carcassonne", "photos": {"a.jpg", "b.jpg"}}]
deduped = dedupe_entries(entries, splits)
photo_sets = [e["source_photos"] for e in deduped]
assert ["a.jpg"] in photo_sets and ["b.jpg"] in photo_sets # split copies
assert ["c.jpg", "d.jpg"] in photo_sets # other edition still dedupes
def test_corrupt_store_fails_loud_with_filename(tmp_path):
path = tmp_path / "title_splits.json"
path.write_text("<<<<<<< merge conflict")
with pytest.raises(ValueError, match="title_splits.json"):
load_title_splits(path)
def test_edits_fix_misreads_before_dedupe():
# a corrected misspelling merges with the correctly-read sighting
edits = [{"match": "Hebarceos", "title_raw": "Herbaceous"}]
deduped = dedupe_entries(
apply_title_edits(
[_entry("Hebarceos", "a.jpg"), _entry("Herbaceous", "b.jpg")], edits
)
)
assert len(deduped) == 1
assert deduped[0]["title_raw"] == "Herbaceous"
assert deduped[0]["source_photos"] == ["a.jpg", "b.jpg"]
def test_edits_chain_and_target_photos():
edits = [
{"match": "Wiz-War", "photos": ["a.jpg"], "edition_hint": "7th Edition"},
{"match": "Wiz-War", "title_raw": "Wiz-War!", "photos": ["a.jpg"]},
# made later, against the renamed title — must chain onto the result
{"match": "Wiz-War!", "photos": ["a.jpg"], "year_hint": 1997},
]
entries = apply_title_edits(
[_entry("Wiz-War", "a.jpg"), _entry("Wiz-War", "b.jpg")], edits
)
assert entries[0]["title_raw"] == "Wiz-War!"
assert entries[0]["edition_hint"] == "7th Edition"
assert entries[0]["year_hint"] == 1997
assert entries[1] == _entry("Wiz-War", "b.jpg") # untargeted copy untouched
def test_stores_roundtrip_and_replay_from_raw(tmp_path):
cfg = Config(data_dir=tmp_path / "data", photos_dir=tmp_path / "photos")
raw = cfg.extract_raw_dir
raw.mkdir(parents=True)
for photo in ("a.jpg", "b.jpg"):
(raw / f"{photo}.json").write_text(
json.dumps({"titles": [_entry("Wiz-War", photo)], "unidentified": []})
)
record_title_split(cfg.title_splits_path, "Wiz-War", ["a.jpg", "b.jpg"])
record_title_split(cfg.title_splits_path, "wiz war", ["a.jpg"]) # covered: no dupe
record_title_edit(
cfg.title_edits_path,
{"match": "Wiz-War", "photos": ["a.jpg"], "edition_hint": "7th Edition"},
)
record_title_edit( # identical retry must not double-record
cfg.title_edits_path,
{"match": "Wiz-War", "photos": ["a.jpg"], "edition_hint": "7th Edition"},
)
replay_titles(cfg)
titles = json.loads(cfg.titles_path.read_text())
assert [e["source_photos"] for e in titles] == [["a.jpg"], ["b.jpg"]]
assert titles[0]["edition_hint"] == "7th Edition"
assert len(load_title_splits(cfg.title_splits_path)) == 1
assert len(load_title_edits(cfg.title_edits_path)) == 1
# a later full rebuild (a real extract run) honors the same stores
rebuild_artifacts(
raw,
cfg.titles_path,
cfg.unidentified_path,
load_title_splits(cfg.title_splits_path),
load_title_edits(cfg.title_edits_path),
)
assert len(json.loads(cfg.titles_path.read_text())) == 2
def test_replay_ignores_partial_raw_caches(tmp_path):
# fresh-clone shape: committed titles.json spans two photos, but only
# one raw cache file exists (raw is gitignored) — replay must not
# rebuild from the partial raws and truncate the catalog
cfg = Config(data_dir=tmp_path / "data", photos_dir=tmp_path / "photos")
raw = cfg.extract_raw_dir
raw.mkdir(parents=True)
(raw / "a.jpg.json").write_text(
json.dumps({"titles": [_entry("Catan", "a.jpg")], "unidentified": []})
)
cfg.titles_path.write_text(
json.dumps([_entry("Catan", "a.jpg"), _entry("Wingspan", "b.jpg")])
)
replay_titles(cfg)
titles = {e["title_raw"] for e in json.loads(cfg.titles_path.read_text())}
assert titles == {"Catan", "Wingspan"}
def test_replay_without_raw_caches_explodes_merged_entries(tmp_path):
cfg = Config(data_dir=tmp_path / "data", photos_dir=tmp_path / "photos")
cfg.data_dir.mkdir(parents=True)
merged = _entry("Wiz-War", "a.jpg")
merged["source_photos"] = ["a.jpg", "b.jpg", "c.jpg"]
cfg.titles_path.write_text(json.dumps([merged, _entry("Catan", "a.jpg")]))
record_title_split(cfg.title_splits_path, "Wiz-War", ["a.jpg", "b.jpg", "c.jpg"])
replay_titles(cfg)
titles = json.loads(cfg.titles_path.read_text())
by_title = {}
for e in titles:
by_title.setdefault(e["title_raw"], []).append(e["source_photos"])
assert by_title["Wiz-War"] == [["a.jpg"], ["b.jpg"], ["c.jpg"]]
assert by_title["Catan"] == [["a.jpg"]] # non-split entries survive intact
def test_dedupe_upgrades_confidence(): def test_dedupe_upgrades_confidence():
deduped = dedupe_entries( deduped = dedupe_entries(
[ [
+9
View File
@@ -439,6 +439,15 @@ def test_dedupe_versions_must_agree():
assert dedupe_matches(rows3, []) == [] assert dedupe_matches(rows3, []) == []
def test_dedupe_matches_skips_split_titles():
rows = [
_mrow("Wiz-War", "94", "a.jpg"),
_mrow("Wiz-War", "94", "b.jpg"),
]
assert dedupe_matches(rows, [], splits=[{"norm": "wiz war", "photos": None}]) == []
assert all(not r["merged_into"] for r in rows)
def test_dedupe_is_idempotent_and_skips_merged(): def test_dedupe_is_idempotent_and_skips_merged():
rows = [ rows = [
_mrow("Jokin Ha...", "193621", "a.jpg"), _mrow("Jokin Ha...", "193621", "a.jpg"),
+294
View File
@@ -432,3 +432,297 @@ def test_duplicate_rows_are_individually_decidable_via_row_ix(tmp_path):
) )
rows = read_m(cfg.matches_path) rows = read_m(cfg.matches_path)
assert [r["match_status"] for r in rows] == ["unmatched", "rejected"] assert [r["match_status"] for r in rows] == ["unmatched", "rejected"]
# -- catalog curation: rowless splits and title edits ---------------------
def _rowless_wizwar(cfg) -> None:
"""Add a multi-photo extracted title with NO matches row (still awaiting
resolve — the Wiz-War shape)."""
titles = json.loads(cfg.titles_path.read_text())
titles.append(
{
"title_raw": "Wiz-War",
"confidence": "high",
"publisher_hint": "Fantasy Flight Games",
"edition_hint": "",
"source_photos": ["shelf.jpg", "shelf2.jpg"],
}
)
cfg.titles_path.write_text(json.dumps(titles))
def test_rowless_multiphoto_title_is_splittable(tmp_path):
cfg = make_cfg(tmp_path)
_rowless_wizwar(cfg)
web = TestClient(create_app(cfg, client=unauthorized_client(tmp_path)))
line = next(
c
for c in web.get("/api/state").json()["catalog"]
if c["title_raw"] == "Wiz-War"
)
assert line["status"] == "awaiting_resolve"
assert line["can_split"] is True
res = web.post(
"/api/split",
json={
"title_raw": "Wiz-War",
"source_photos": "shelf.jpg;shelf2.jpg",
"row_ix": None,
},
)
assert res.status_code == 200
copies = [c for c in res.json()["catalog"] if c["title_raw"] == "Wiz-War"]
assert [c["photos"] for c in copies] == [["shelf.jpg"], ["shelf2.jpg"]]
assert all(c["can_split"] is False for c in copies)
# the decision persisted: the store holds it (photo-scoped) and
# titles.json is split
(record,) = json.loads(cfg.title_splits_path.read_text())
assert record == {"title": "Wiz-War", "photos": ["shelf.jpg", "shelf2.jpg"]}
per_photo = [
e["source_photos"]
for e in json.loads(cfg.titles_path.read_text())
if e["title_raw"] == "Wiz-War"
]
assert per_photo == [["shelf.jpg"], ["shelf2.jpg"]]
def test_split_still_requires_multiple_photos(tmp_path):
web, _ = make_client(tmp_path)
res = web.post(
"/api/split",
json={
"title_raw": "Fresh Off The Shelf",
"source_photos": "shelf.jpg",
"row_ix": None,
},
)
assert res.status_code == 400
def test_edit_title_corrects_read_and_requeues_row(tmp_path):
web, cfg = make_client(tmp_path)
# Citadels has an ambiguous matches row; correcting its read must drop
# the stale row (it was searched with the old data) and store the fix
res = web.post(
"/api/edit-title",
json={
"title_raw": "Citadels",
"source_photos": "shelf.jpg",
"title_new": "Citadels: Dark City",
"edition": "2nd Edition",
"publisher": "",
"year": "2004",
},
)
assert res.status_code == 200
state = res.json()
titles = [c["title_raw"] for c in state["catalog"]]
assert "Citadels: Dark City" in titles and "Citadels" not in titles
corrected = next(
e
for e in json.loads(cfg.titles_path.read_text())
if e["title_raw"] == "Citadels: Dark City"
)
assert corrected["year_hint"] == 2004
assert corrected["edition_hint"] == "2nd Edition"
assert corrected["publisher_hint"] == "" # empty string cleared the cue
assert not corrected.get("language_hint") # untouched (was unset)
assert not any(r["title_raw"] == "Citadels" for r in read_matches(cfg.matches_path))
(record,) = json.loads(cfg.title_edits_path.read_text())
assert record["match"] == "Citadels"
assert record["publisher_hint"] == ""
def test_edit_title_rejects_empty_and_noop(tmp_path):
web, _ = make_client(tmp_path)
assert (
web.post(
"/api/edit-title",
json={
"title_raw": "Citadels",
"source_photos": "shelf.jpg",
"title_new": " ",
},
).status_code
== 400
)
assert (
web.post(
"/api/edit-title",
json={"title_raw": "Citadels", "source_photos": "shelf.jpg"},
).status_code
== 400
)
assert (
web.post(
"/api/edit-title",
json={
"title_raw": "No Such Game",
"source_photos": "shelf.jpg",
"title_new": "X",
},
).status_code
== 404
)
def test_edit_preserves_vetoed_rows_and_renames_them(tmp_path):
cfg = make_cfg(tmp_path)
rows = read_matches(cfg.matches_path)
rows.append(
_row(
title_raw="Citadels",
match_status="approved",
bgg_id="478",
source_photos="shelf2.jpg",
dedupe_veto="1",
)
)
write_matches(cfg.matches_path, rows)
web = TestClient(create_app(cfg, client=unauthorized_client(tmp_path)))
res = web.post(
"/api/edit-title",
json={
"title_raw": "Citadels",
"source_photos": "shelf.jpg",
"title_new": "Citadels (2016)",
},
)
assert res.status_code == 200
after = read_matches(cfg.matches_path)
# the human-vetoed row survives the cull, renamed to follow the entry;
# the unvetoed ambiguous row is requeued (dropped)
veto = [r for r in after if r.get("dedupe_veto")]
assert len(veto) == 1
assert veto[0]["title_raw"] == "Citadels (2016)"
assert veto[0]["match_status"] == "approved"
assert not any(
r["title_raw"] == "Citadels" and not r.get("dedupe_veto") for r in after
)
def test_drop_rows_photo_narrowing_spares_other_copy(tmp_path):
import io as _io
from rich.console import Console
from bggpipe.review import ReviewSession
cfg = make_cfg(tmp_path)
write_matches(
cfg.matches_path,
[
_row(title_raw="Wiz-War", match_status="auto", source_photos="a.jpg"),
_row(title_raw="Wiz-War", match_status="auto", source_photos="b.jpg"),
],
)
session = ReviewSession(
cfg,
console=Console(file=_io.StringIO()),
input_fn=lambda prompt: "",
client=unauthorized_client(tmp_path),
)
assert session.drop_rows("Wiz-War", ["a.jpg"]) == 1
(survivor,) = read_matches(cfg.matches_path)
assert survivor["source_photos"] == "b.jpg"
def test_split_and_edit_refuse_while_pipeline_rewrites(tmp_path):
import threading
from bggpipe.jobs import JobRunner
cfg = make_cfg(tmp_path)
_rowless_wizwar(cfg)
release = threading.Event()
started = threading.Event()
def blocking_extract():
started.set()
release.wait(timeout=5)
jobs = JobRunner()
web = TestClient(
create_app(
cfg,
client=unauthorized_client(tmp_path),
stages={"extract": blocking_extract},
jobs=jobs,
)
)
assert web.post("/api/run/extract").status_code == 200
assert started.wait(timeout=5)
try:
split = web.post(
"/api/split",
json={
"title_raw": "Wiz-War",
"source_photos": "shelf.jpg;shelf2.jpg",
"row_ix": None,
},
)
edit = web.post(
"/api/edit-title",
json={
"title_raw": "Wiz-War",
"source_photos": "shelf.jpg;shelf2.jpg",
"title_new": "Wiz-War!",
},
)
finally:
release.set()
assert split.status_code == 409
assert edit.status_code == 409
# neither curation store was written under the in-flight rewrite
assert not cfg.title_splits_path.exists()
assert not cfg.title_edits_path.exists()
def test_renaming_a_split_copy_keeps_split_protection(tmp_path):
cfg = make_cfg(tmp_path)
_rowless_wizwar(cfg)
web = TestClient(create_app(cfg, client=unauthorized_client(tmp_path)))
assert (
web.post(
"/api/split",
json={
"title_raw": "Wiz-War",
"source_photos": "shelf.jpg;shelf2.jpg",
"row_ix": None,
},
).status_code
== 200
)
# rename BOTH copies to the same corrected title, one at a time — the
# store must keep protecting them or the rebuild re-merges the copies
for photo in ("shelf.jpg", "shelf2.jpg"):
assert (
web.post(
"/api/edit-title",
json={
"title_raw": "Wiz-War",
"source_photos": photo,
"title_new": "Wiz War 2000",
},
).status_code
== 200
)
per_photo = [
e["source_photos"]
for e in json.loads(cfg.titles_path.read_text())
if e["title_raw"] == "Wiz War 2000"
]
assert per_photo == [["shelf.jpg"], ["shelf2.jpg"]] # still two copies
def test_single_photo_rowless_entry_is_not_splittable(tmp_path):
web, _ = make_client(tmp_path)
line = next(
c
for c in web.get("/api/state").json()["catalog"]
if c["title_raw"] == "Fresh Off The Shelf"
)
assert line["can_split"] is False