Durable curation: persisted splits + pre-resolve title edits, catalog A→Z

Wiz-War had no split button: can_split required a matches row, but fresh
extractions leave multi-photo titles rowless until resolve runs. Splits
are now a title-level decision persisted in data/title_splits.json,
honored by extract's dedupe and resolve's dedupe on every rebuild, with
the button on any multi-photo line — resolved or not.

Same mechanism carries human corrections: data/title_edits.json stores
fixed misreads and known cues (publisher/edition/year/language), applied
before dedupe on every titles.json rebuild, editable from a new inline
form on every catalog line. An edit drops the title's stale matches rows
so resolve re-queries with the corrected data.

The catalog page now sorts alphabetically (case-insensitive; split
copies stay adjacent) instead of extraction order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jXZFSTZQKzAC8fqpWSz9g
This commit is contained in:
Eric Wagoner
2026-08-02 19:47:08 -04:00
parent 24a7bcb9e0
commit a7f0cfee05
12 changed files with 641 additions and 28 deletions
+2 -1
View File
@@ -47,10 +47,11 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel
- Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan). - Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan).
- Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry. - Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry.
- Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance. - Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance.
- **Human curation is durable**: `data/title_splits.json` (titles split into per-photo copies) and `data/title_edits.json` (corrected reads/cues) are replayed on every titles.json rebuild, and both extract's and resolve's dedupe honor them forever. Row-level split decisions also persist via the `dedupe_veto` column.
- **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope). - **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope).
- Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift. - Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift.
## Git ## Git
- Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub**`gh` CLI does not work here. - Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub**`gh` CLI does not work here.
- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`. - Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/title_splits.json`, `data/title_edits.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`.
+45
View File
@@ -1,45 +1,90 @@
[ [
"IMG_4501.jpeg|Bottom shelf, left portion, above 'Exploding Kittens' box|EXPANSION|Small box with partial text visible, colorful design, title cut off by frame edge",
"IMG_4501.jpeg|Bottom shelf, right side, near Exploding Kittens box|COSMIC|Colorful box spine, blue/green tones, title partially visible but not fully confirmable",
"IMG_4501.jpeg|Middle shelf, purple box beneath Joking Hazard, right side of stack|A Boardgame for 1 to 6 players, ages 8 and up.|Purple/violet box, text describing player count, title not legible from this angle",
"IMG_4501.jpeg|Right side, middle shelf, black spine sticking out between Santorini and purple box below|T|Black spine, single letter visible, rest obscured",
"IMG_4501.jpeg|Top shelf, far left edge, cut off by frame; near 'a Gentle Rain' box||Dark/black box, only a thin sliver visible",
"IMG_4502.jpeg|bottom-left corner of shelf, beneath the History of the World area, left of Dungeon!|A Boardgame for 1 to 6 players, ages 8 and up|purple/tan box with descriptive text visible but no title readable",
"IMG_4502.jpeg|far left side of shelf, stacked horizontally beneath the yellow-floral box, left of Santorini||small colorful box spines, red and white coloring, too angled/blurry to read title",
"IMG_4502.jpeg|left side of shelf, below the horizontal stack, left of Santorini and above Joking Hazard||blue box with white illustration, appears to be a game box but title not legible",
"IMG_4502.jpeg|top-left corner of shelf, above Santorini and to the left of Wiz-War|possibly 'Herbaceous' or similar, text partially cut off|cream/white box with floral yellow illustration, thin spine",
"IMG_4505.jpeg|Bottom area, left side near Mantis Falls, small figure/box partially visible||Small gold/tan colored object, possibly a game piece or small box, mostly obscured", "IMG_4505.jpeg|Bottom area, left side near Mantis Falls, small figure/box partially visible||Small gold/tan colored object, possibly a game piece or small box, mostly obscured",
"IMG_4505.jpeg|Top shelf, far left edge behind Cat Crimes, cut off by frame edge||Dark brown/maroon spine with faint gold decorative border, partially visible vertical box",
"IMG_4505.jpeg|Top shelf, far left edge, partially cut off by frame, to the left of Cat Crimes||Dark brown/maroon box spine with gold decorative swirl design, portion visible at left edge of image", "IMG_4505.jpeg|Top shelf, far left edge, partially cut off by frame, to the left of Cat Crimes||Dark brown/maroon box spine with gold decorative swirl design, portion visible at left edge of image",
"IMG_4506.jpeg|Middle area, below and left of Santorini, dark red/maroon box spine with small square icons||Dark reddish-brown spine with small colored square icons in a row, text not legible", "IMG_4506.jpeg|Middle area, below and left of Santorini, dark red/maroon box spine with small square icons||Dark reddish-brown spine with small colored square icons in a row, text not legible",
"IMG_4506.jpeg|Right side of image, behind Santorini box, dark box with small visible logo||Dark/black box with a small circular or square colored logo element, mostly obscured by foreground Santorini box",
"IMG_4506.jpeg|Right side, background behind Santorini box, partially visible dark box with small game piece icon||Dark background box with a small blue/white icon, mostly obscured by Santorini box in foreground", "IMG_4506.jpeg|Right side, background behind Santorini box, partially visible dark box with small game piece icon||Dark background box with a small blue/white icon, mostly obscured by Santorini box in foreground",
"IMG_4506.jpeg|Top shelf, above Santorini spine, partially visible box with patterned tiles/squares in yellow, blue, and cream tones||Patchwork/quilt-like pattern of colored squares (yellow, tan, blue) along the top edge of the box, style suggests a tile-based board game", "IMG_4506.jpeg|Top shelf, above Santorini spine, partially visible box with patterned tiles/squares in yellow, blue, and cream tones||Patchwork/quilt-like pattern of colored squares (yellow, tan, blue) along the top edge of the box, style suggests a tile-based board game",
"IMG_4506.jpeg|Top shelf, right side near blue corner box, small red 'X' symbols visible on light blue background|X X|Light blue box edge with red X pattern, possibly a tally or scoring game box", "IMG_4506.jpeg|Top shelf, right side near blue corner box, small red 'X' symbols visible on light blue background|X X|Light blue box edge with red X pattern, possibly a tally or scoring game box",
"IMG_4506.jpeg|Top shelf, spine visible above Santorini box, partially cut off at top of frame||Multicolored patchwork/tile pattern spine in yellow, tan, and blue tones with geometric designs; appears to be a board game box edge but no title text is legible",
"IMG_4506.jpeg|Upper right corner of image, above and right of Santorini spine, next to unidentified patterned box||Small red and blue box corner visible, mostly cut off by frame edge, no discernible text",
"IMG_4507.jpeg|Below 'LABYRINTH', left side of shelf, partially cut off at frame edge|DRAGO...|Green/dark box, only top edge visible, text cut off",
"IMG_4507.jpeg|Middle shelf area, partially hidden behind Labyrinth box and below Utter Nonsense, left side of the stack|Dragon...|Green/dark colored box spine, text cut off by adjacent boxes, only 'Dragon' prefix visible", "IMG_4507.jpeg|Middle shelf area, partially hidden behind Labyrinth box and below Utter Nonsense, left side of the stack|Dragon...|Green/dark colored box spine, text cut off by adjacent boxes, only 'Dragon' prefix visible",
"IMG_4507.jpeg|Top of stack, right of center, between 'a Gentle Rain' and 'SANToRINI'|PATCH / WORK (possibly)|Colorful patchwork quilt-style pattern on spine, small box",
"IMG_4507.jpeg|Upper right area near Santorini and a Gentle Rain, appears to be a narrow spine box|PATCH WORK or similar|Colorful patchwork/quilt-like pattern visible on spine, tan and multicolor squares design", "IMG_4507.jpeg|Upper right area near Santorini and a Gentle Rain, appears to be a narrow spine box|PATCH WORK or similar|Colorful patchwork/quilt-like pattern visible on spine, tan and multicolor squares design",
"IMG_4508.jpeg|Top shelf, center, small yellow tin sitting next to a black box near the stack of papers/magazines, above the Labyrinth box||Small yellow tin container, no visible text, could be a card game tin",
"IMG_4508.jpeg|Top shelf, center, yellow tin box sitting next to the Super Mario Checkers box||Small yellow tin, no visible title text, could be a card game tin", "IMG_4508.jpeg|Top shelf, center, yellow tin box sitting next to the Super Mario Checkers box||Small yellow tin, no visible title text, could be a card game tin",
"IMG_4508.jpeg|Top shelf, far left, behind the stack of papers/notebooks, next to the framed picture||Small black box with unreadable logo, mostly obscured by papers", "IMG_4508.jpeg|Top shelf, far left, behind the stack of papers/notebooks, next to the framed picture||Small black box with unreadable logo, mostly obscured by papers",
"IMG_4508.jpeg|Top shelf, far left, partially cut off by frame edge, next to a green-framed picture and above the Super Mario Checkers box||Teal/green colored box edge, mostly obscured, shape suggests a game box but no readable text",
"IMG_4508.jpeg|Top shelf, right of center, wooden box with metal clasp between the checkers box and the papers stack||Wooden box with brass hardware, no visible title, appears decorative or a game storage box", "IMG_4508.jpeg|Top shelf, right of center, wooden box with metal clasp between the checkers box and the papers stack||Wooden box with brass hardware, no visible title, appears decorative or a game storage box",
"IMG_4508.jpeg|Top shelf, right side, partially visible box behind the wooden box, between the stack of papers and the edge of the frame|SCRABBLE (partial letters visible)|Red and teal colored box edge, appears to be a Scrabble-branded box but mostly obscured", "IMG_4508.jpeg|Top shelf, right side, partially visible box behind the wooden box, between the stack of papers and the edge of the frame|SCRABBLE (partial letters visible)|Red and teal colored box edge, appears to be a Scrabble-branded box but mostly obscured",
"IMG_4511.jpeg|Middle shelf, behind Superfight boxes, green/black box with skeletal figure artwork - between Superfight stack and the Rook box|...CTHULHU|Dark green box, illustrated skeletal/robed figure, likely a Cthulhu-themed game (e.g., Cthulhu Fluxx or Cthulhu Gloom) but exact title obscured",
"IMG_4511.jpeg|Middle shelf, black box stacked below SUPERFIGHT, above ROOK; left side of shelf|...Go... Yourself|Black spine box, small white/colored text, similar size to Superfight box", "IMG_4511.jpeg|Middle shelf, black box stacked below SUPERFIGHT, above ROOK; left side of shelf|...Go... Yourself|Black spine box, small white/colored text, similar size to Superfight box",
"IMG_4511.jpeg|Middle shelf, black spine box stacked above Rook, to the left of the dice tray|...Yourself|Black box with orange/white text, only partial word 'Yourself' legible, title obscured by angle",
"IMG_4511.jpeg|Right edge of shelf, partially cut off by frame, next to Cthulhu box|HEX|Blue box with yellow/tan geometric pattern, only edge visible", "IMG_4511.jpeg|Right edge of shelf, partially cut off by frame, next to Cthulhu box|HEX|Blue box with yellow/tan geometric pattern, only edge visible",
"IMG_4511.jpeg|Top right corner of shelf, box mostly out of frame, above the Hex box|Drag...|Orange and blue box, only a fragment of text visible, likely a dragon-themed game title",
"IMG_4517.jpeg|Middle of shelf, between Pie Face! (left) and the blue box with bird artwork (right)|...Grenade...|Tall narrow box, dark red/brown color scheme, appears to have a hand grenade or bomb-like illustration", "IMG_4517.jpeg|Middle of shelf, between Pie Face! (left) and the blue box with bird artwork (right)|...Grenade...|Tall narrow box, dark red/brown color scheme, appears to have a hand grenade or bomb-like illustration",
"IMG_4517.jpeg|Middle of shelf, right of the grenade-themed box and left of Walk the Plank!|Bird ... Bird ...|Blue box with tall bird-like cartoon character illustration, small/narrow box shape", "IMG_4517.jpeg|Middle of shelf, right of the grenade-themed box and left of Walk the Plank!|Bird ... Bird ...|Blue box with tall bird-like cartoon character illustration, small/narrow box shape",
"IMG_4517.jpeg|Middle shelf area, second box from left, between 'Pie Face!' and 'Bird on Your Bread?!'|...renade... Nintendo?|Narrow tall box, red and tan coloring, partially obscured by shadow and angle, appears to have small figure illustration",
"IMG_4518.jpeg|Between Mage Wars Academy Core Set and Saberu Merry Men||Thin black spine, largely obscured, possibly a second small game box", "IMG_4518.jpeg|Between Mage Wars Academy Core Set and Saberu Merry Men||Thin black spine, largely obscured, possibly a second small game box",
"IMG_4519.jpeg|Far right of shelf, white spine box next to the two red 'G' logo boxes|Before I Kill You, Mister Bond... / Kill Dr. No Lucky|White worn spine with black text, possibly James Bond themed game, 'CHEAPEST GAMES' text also visible on spine which may be a store sticker rather than title", "IMG_4519.jpeg|Far right of shelf, white spine box next to the two red 'G' logo boxes|Before I Kill You, Mister Bond... / Kill Dr. No Lucky|White worn spine with black text, possibly James Bond themed game, 'CHEAPEST GAMES' text also visible on spine which may be a store sticker rather than title",
"IMG_4519.jpeg|Left shelf, red spine with black text between the cream D&D box and the red-orange boxes|THE BEST OF RA... (possibly 'RAMPHA' or similar)|Dark red spine with illustrated figure artwork, book-like thickness",
"IMG_4519.jpeg|Middle shelf area, two matching red boxes with orange circular 'G' logo, positioned between the white D&D boxes on the left and the white spined books on the right|G (stylized orange circular logo)|Two identical red boxes, narrow spine width, orange/yellow circular logo with letter G, taped/worn edges",
"IMG_4519.jpeg|Right-center of shelf, two matching red boxes standing between the D&D group on the left and the white 'Cheapest Games' spine on the right|Orange 'G' logo visible on both spines|Two identical red boxes with worn/taped edges, orange circular 'G' emblem, appears to be same game duplicated", "IMG_4519.jpeg|Right-center of shelf, two matching red boxes standing between the D&D group on the left and the white 'Cheapest Games' spine on the right|Orange 'G' logo visible on both spines|Two identical red boxes with worn/taped edges, orange circular 'G' emblem, appears to be same game duplicated",
"IMG_4520.jpeg|Middle shelf, between Carcassonne and Evolution: The Beginning||Plain wooden-colored vertical box, no visible text, appears to be a wood grain finish game box or insert", "IMG_4520.jpeg|Middle shelf, between Carcassonne and Evolution: The Beginning||Plain wooden-colored vertical box, no visible text, appears to be a wood grain finish game box or insert",
"IMG_4521.jpeg|Top edge of frame, above the Sleeping Gods box, partially cut off||Dark spines/boxes barely visible at top, too obscured by shadow and cropping to identify", "IMG_4521.jpeg|Top edge of frame, above the Sleeping Gods box, partially cut off||Dark spines/boxes barely visible at top, too obscured by shadow and cropping to identify",
"IMG_4521.jpeg|Top of shelf, above the Sleeping Gods box, partially cut off by frame edge||Dark spines of several boxes, indistinct due to shadow and angle, colors appear dark blue/black and tan",
"IMG_4527.jpeg|Left edge of shelf, partially visible spines to the left of the main game box, on the top shelf|01, T...|Appears to be book-like spines, red and dark covers, stacked vertically, likely not games but partially obscured", "IMG_4527.jpeg|Left edge of shelf, partially visible spines to the left of the main game box, on the top shelf|01, T...|Appears to be book-like spines, red and dark covers, stacked vertically, likely not games but partially obscured",
"IMG_4527.jpeg|Right edge of shelf, behind/beside the Red Dragon Inn box, partially cut off by frame|4, 3|Dark colored spine/box edge, numbers visible but title obscured",
"IMG_4528.jpeg|Top shelf, above Wiz-War box, partially obscured by rolled poster/tube on left side|SLUGFEST GAMES|Red and orange colored box with illustrated fantasy/comic style artwork, only top edge visible",
"IMG_4528.jpeg|Top shelf, back corner above the Wiz-War box; only the publisher logo area is visible, partially obscured by other items|SLUGFEST GAMES|Colorful red/orange box art, only top publisher banner visible, rest of box obscured", "IMG_4528.jpeg|Top shelf, back corner above the Wiz-War box; only the publisher logo area is visible, partially obscured by other items|SLUGFEST GAMES|Colorful red/orange box art, only top publisher banner visible, rest of box obscured",
"IMG_4529.jpeg|Below the 1000-piece puzzle box, spine showing armored warrior artwork, above the face-out Risk Lord of the Rings box|The Middle-earth|Dark battle scene with armored figures, likely same LOTR-themed game box viewed from spine angle", "IMG_4529.jpeg|Below the 1000-piece puzzle box, spine showing armored warrior artwork, above the face-out Risk Lord of the Rings box|The Middle-earth|Dark battle scene with armored figures, likely same LOTR-themed game box viewed from spine angle",
"IMG_4529.jpeg|Middle of stack between the puzzle box and Risk Lord of the Rings, tan/cream colored spine with Asian characters and blue castle icon|unreadable Asian characters, blue icon|Tan spine with small blue building/castle graphic and Asian script text", "IMG_4529.jpeg|Middle of stack between the puzzle box and Risk Lord of the Rings, tan/cream colored spine with Asian characters and blue castle icon|unreadable Asian characters, blue icon|Tan spine with small blue building/castle graphic and Asian script text",
"IMG_4529.jpeg|Top of stack, above the 1000-piece puzzle box, partially cut off at top of frame|V, 3|Dark spines with minimal visible text, one shows a circular 'V' logo, another shows number '3' on purple/dark background; too obscured to identify title",
"IMG_4529.jpeg|Top of stack, second partial box edge, above the puzzle box|0|White/light colored box edge with a circular icon, only corner visible", "IMG_4529.jpeg|Top of stack, second partial box edge, above the puzzle box|0|White/light colored box edge with a circular icon, only corner visible",
"IMG_4529.jpeg|Very top of stack, box behind/above the numbered fragments, purple/dark spine||Dark purple or navy spine, mostly obscured, edge of frame", "IMG_4529.jpeg|Very top of stack, box behind/above the numbered fragments, purple/dark spine||Dark purple or navy spine, mostly obscured, edge of frame",
"IMG_4529.jpeg|Very top of stack, partial box edge visible above the puzzle box|3|Dark colored spine/edge, only a sliver visible, number '3' printed in white", "IMG_4529.jpeg|Very top of stack, partial box edge visible above the puzzle box|3|Dark colored spine/edge, only a sliver visible, number '3' printed in white",
"IMG_4532.jpeg|Top right corner of frame, above Gloomhaven box, partially cut off|GLO...|Fiery orange/red background with dark textured lettering, appears to be another large box possibly related to Gloomhaven expansion",
"IMG_4532.jpeg|Top right corner of image, partially cut off, above and behind the Gloomhaven box|GLO... (possibly part of another title starting with GLO)|Dark box with fiery orange/red background and large white curved text, only edge visible", "IMG_4532.jpeg|Top right corner of image, partially cut off, above and behind the Gloomhaven box|GLO... (possibly part of another title starting with GLO)|Dark box with fiery orange/red background and large white curved text, only edge visible",
"IMG_4535.jpeg|Top shelf, far right edge of frame, next to the last visible Etherfields big box||Black box spine, partially cut off by frame edge, appears similar in size to adjacent Etherfields big boxes",
"IMG_4542.jpeg|Bottom right shelf area, dark spine next to unidentified train-themed box||Dark spine, illegible text, appears to be a board game box based on shape",
"IMG_4542.jpeg|Right side shelf, below CATAN, dark spine with red/white text|T...T / Ride (possible)|Dark red/maroon spine with train-like graphic element, narrow box spine",
"IMG_4542.jpeg|Right side shelf, below Catan box, next to unidentified black-spined game|T...C...|Teal/orange box with train-like graphic, partially obscured, could be a train-themed game", "IMG_4542.jpeg|Right side shelf, below Catan box, next to unidentified black-spined game|T...C...|Teal/orange box with train-like graphic, partially obscured, could be a train-themed game",
"IMG_4542.jpeg|Right side shelf, lower row near 'Catan' box||Dark spine with red accent, text too small/blurry to read", "IMG_4542.jpeg|Right side shelf, lower row near 'Catan' box||Dark spine with red accent, text too small/blurry to read",
"IMG_4542.jpeg|Top area, small red square box with white 'M' logo, near Dragon Land|M|Small red square box with large white letter M, possibly a classic board game logo",
"IMG_4542.jpeg|Top left corner of shelf, partially cut off by frame edge, above the Skip-Bo box|...O / From the Makers of...|Red box edge with white lettering, likely a card game box, mostly obscured",
"IMG_4542.jpeg|Top right shelf, partially hidden behind Dragon Land box|From the Makers of, M|Red box with a large stylized 'M' logo and small icons, likely a Mattel-published card game (possibly UNO), edge cut off by frame", "IMG_4542.jpeg|Top right shelf, partially hidden behind Dragon Land box|From the Makers of, M|Red box with a large stylized 'M' logo and small icons, likely a Mattel-published card game (possibly UNO), edge cut off by frame",
"IMG_4543.jpeg|Background, lower right area behind the hand holding Scrabble Slam, blurred and out of focus||Indistinct colorful box shapes, cannot make out title or artwork details due to heavy blur",
"IMG_4543.jpeg|Right edge of frame, middle row, next to the Scrabble Slam box|CAT...|Dark colored box spine, partially cut off at right edge of photo, red/dark background with light text",
"IMG_4543.jpeg|Top right corner of frame, behind and above the Scrabble Slam box, partially cut off by frame edge|Dogs Kitchen|Small box or book with illustrated cartoon-style cover, colorful, appears to feature a dog character; too small/blurry to confirm if it's a game",
"IMG_4543.jpeg|right edge of image, behind Scrabble Slam box|CAT|Yellow/orange box edge visible, partial text 'CAT' in bold letters, rest obscured by foreground box", "IMG_4543.jpeg|right edge of image, behind Scrabble Slam box|CAT|Yellow/orange box edge visible, partial text 'CAT' in bold letters, rest obscured by foreground box",
"IMG_4543.jpeg|top right corner of image, above and behind the main Scrabble Slam box|Dogs Kitchen|Small blue/purple box partially cut off at top edge, illustrated cartoon-style artwork", "IMG_4543.jpeg|top right corner of image, above and behind the main Scrabble Slam box|Dogs Kitchen|Small blue/purple box partially cut off at top edge, illustrated cartoon-style artwork",
"IMG_4544.jpeg|Background shelf behind the Cat Stax box, blurred and out of focus, appears to be a row of game/book boxes||Multiple colorful spines/boxes including orange, white, and dark tones, too blurry to make out any text", "IMG_4544.jpeg|Background shelf behind the Cat Stax box, blurred and out of focus, appears to be a row of game/book boxes||Multiple colorful spines/boxes including orange, white, and dark tones, too blurry to make out any text",
"IMG_4544.jpeg|Background shelf behind the Cat Stax box, top of frame, multiple stacked boxes with no clearly identifiable neighbors||Blurry stack of colorful boxes (orange, white, dark tones) in background, out of focus, too indistinct to read any text",
"IMG_4545.jpeg|Bottom right corner of image, yellowish box edge visible||Yellow/tan colored box spine, text not legible due to angle and cropping",
"IMG_4545.jpeg|Lower left shelf area, dark colored box partially visible behind hand holding Pocket Farkel||Dark/black box edge visible, no legible text",
"IMG_4545.jpeg|Top right shelf, box with 'Ravensburger' text visible, title obscured|Ravensburger|Colorful box, brand name visible but game title not legible",
"IMG_4545.jpeg|Top shelf, far right edge, partially cut off by frame border|Ravens...|Yellow/orange box with small logo resembling a bird, likely Ravensburger branding but game title not legible", "IMG_4545.jpeg|Top shelf, far right edge, partially cut off by frame border|Ravens...|Yellow/orange box with small logo resembling a bird, likely Ravensburger branding but game title not legible",
"IMG_4545.jpeg|Top shelf, right side, above the 'Ravensburger' labeled box, partial title visible at top edge of frame|DRAGON ...and|Orange/red box with dragon and fantasy artwork, cut off at top of image",
"IMG_4545.jpeg|Top shelf, upper right corner, above and behind the Pocket Farkel tin, to the right of 'Bugs in the Kitchen'|DRAGON, land|Colorful fantasy-style box art, orange/red hues, dragon imagery visible, title cut off at top edge of frame", "IMG_4545.jpeg|Top shelf, upper right corner, above and behind the Pocket Farkel tin, to the right of 'Bugs in the Kitchen'|DRAGON, land|Colorful fantasy-style box art, orange/red hues, dragon imagery visible, title cut off at top edge of frame",
"IMG_4547.jpeg|Top area, right of the maze-like box, small orange/red box||Small red/orange box, text illegible, largely obscured", "IMG_4547.jpeg|Top area, right of the maze-like box, small orange/red box||Small red/orange box, text illegible, largely obscured",
"IMG_4547.jpeg|Top center-right, small red/orange box near top edge||Small reddish box with indistinct artwork, mostly cut off at top frame", "IMG_4547.jpeg|Top center-right, small red/orange box near top edge||Small reddish box with indistinct artwork, mostly cut off at top frame",
"IMG_4547.jpeg|Top left area, next to the '...AZE' box, second spine from left|UTTLES or similar|Bright blue box with light green accents, small toy-like image visible on top",
"IMG_4547.jpeg|Top left corner of image, partially visible behind main box|AZE|Blue and green box, appears to be a puzzle or maze-themed game, mostly cut off by frame edge", "IMG_4547.jpeg|Top left corner of image, partially visible behind main box|AZE|Blue and green box, appears to be a puzzle or maze-themed game, mostly cut off by frame edge",
"IMG_4547.jpeg|Top left corner of shelf, above the main held box, leftmost visible spine|...AZE|Blue and green box, partial text visible, cut off at frame edge",
"IMG_4547.jpeg|Top right area, red-orange box between the dark box and top edge|Roll Pla...|Red/orange box with illustrated artwork, text partially cut off",
"IMG_4547.jpeg|Top right corner of image|Dark T...|Dark blue/black box with white text, appears to be 'Dark T' something, possibly 'Dark Tower' or similar, cut off at frame edge", "IMG_4547.jpeg|Top right corner of image|Dark T...|Dark blue/black box with white text, appears to be 'Dark T' something, possibly 'Dark Tower' or similar, cut off at frame edge",
"IMG_4547.jpeg|Top right corner of shelf, above and to the right of the main held box|Dark Terr... or Dark Territory|Dark navy/black box with red and white text, partially obscured by other boxes",
"IMG_4547.jpeg|Top right, near 'Dark T' box||Colorful box with red/orange tones, illustration unclear, partially obscured by other boxes", "IMG_4547.jpeg|Top right, near 'Dark T' box||Colorful box with red/orange tones, illustration unclear, partially obscured by other boxes",
"IMG_4552.jpeg|Left edge of frame, vertical spine, sharply cropped|Giv... / Ci...|Dark spine with white lettering, partially visible artwork, cropped by frame edge", "IMG_4552.jpeg|Left edge of frame, vertical spine, sharply cropped|Giv... / Ci...|Dark spine with white lettering, partially visible artwork, cropped by frame edge",
"IMG_4552.jpeg|Top left corner of frame, behind and above the Dark Cults package, no clearly identified games adjacent due to cropping|'Giv...'|Dark colored box with white/red text fragment, likely a card or board game box mostly out of frame", "IMG_4552.jpeg|Top left corner of frame, behind and above the Dark Cults package, no clearly identified games adjacent due to cropping|'Giv...'|Dark colored box with white/red text fragment, likely a card or board game box mostly out of frame",
+12
View File
@@ -66,6 +66,18 @@ class Config:
def games_path(self) -> Path: def games_path(self) -> Path:
return self.data_dir / "games.json" return self.data_dir / "games.json"
@property
def title_edits_path(self) -> Path:
# human corrections to extracted reads (misspellings, known cues) —
# replayed on every titles.json rebuild
return self.data_dir / "title_edits.json"
@property
def title_splits_path(self) -> Path:
# titles the human declared to be MULTIPLE physical copies: extract
# and resolve dedupe must never cross-photo-merge them again
return self.data_dir / "title_splits.json"
@property @property
def dismissed_path(self) -> Path: def dismissed_path(self) -> Path:
return self.data_dir / "unidentified_dismissed.json" return self.data_dir / "unidentified_dismissed.json"
+124 -5
View File
@@ -240,11 +240,93 @@ def _merge(a: dict, b: dict) -> dict:
return merged return merged
def dedupe_entries(entries: list[dict]) -> list[dict]: # the fields a human correction may override on an extracted entry
"""Collapse same-normalized-title sightings unless their cues conflict.""" EDIT_FIELDS = (
"title_raw",
"publisher_hint",
"edition_hint",
"year_hint",
"language_hint",
"art_notes",
)
def load_title_edits(path: Path) -> list[dict]:
"""Human corrections to raw reads (fixed misspellings, cues the owner
knows offhand). Each record: {"match": <title as displayed when the fix
was made>, "photos": [...] to target one copy (optional), <EDIT_FIELDS
to override>}. Applied on every rebuild, before dedupe."""
if not path.exists():
return []
return json.loads(path.read_text())
def record_title_edit(path: Path, record: dict) -> None:
existing = load_title_edits(path)
atomic_write_text(
path, json.dumps([*existing, record], indent=2, ensure_ascii=False) + "\n"
)
def apply_title_edits(entries: list[dict], edits: list[dict]) -> list[dict]:
"""Apply stored corrections in order. Matching is by the title an entry
CURRENTLY carries, so a later edit made against an earlier edit's result
chains naturally; each record applies at most once per entry."""
if not edits:
return entries
for entry in entries:
applied: set[int] = set()
while True:
norm = normalize_title(entry["title_raw"])
photos = set(entry.get("source_photos") or [])
hit = next(
(
i
for i, e in enumerate(edits)
if i not in applied
and normalize_title(e["match"]) == norm
and (not e.get("photos") or set(e["photos"]) & photos)
),
None,
)
if hit is None:
break
applied.add(hit)
for key in EDIT_FIELDS:
if key in edits[hit]:
entry[key] = edits[hit][key]
return entries
def load_title_splits(path: Path) -> set[str]:
"""Normalized titles the human split into per-photo copies — a durable
review decision that must survive extract rebuilds and resolve --force."""
if not path.exists():
return set()
return {normalize_title(t) for t in json.loads(path.read_text())}
def record_title_split(path: Path, title: str) -> None:
existing = json.loads(path.read_text()) if path.exists() else []
if normalize_title(title) in {normalize_title(t) for t in existing}:
return
atomic_write_text(
path, json.dumps([*existing, title], indent=2, ensure_ascii=False) + "\n"
)
def dedupe_entries(
entries: list[dict], split_titles: set[str] = frozenset()
) -> list[dict]:
"""Collapse same-normalized-title sightings unless their cues conflict —
or unless the human declared the title split (several physical copies):
those stay one entry per photo."""
result: list[dict] = [] result: list[dict] = []
for entry in entries: for entry in entries:
entry = {**entry, "title_normalized": normalize_title(entry["title_raw"])} entry = {**entry, "title_normalized": normalize_title(entry["title_raw"])}
if entry["title_normalized"] in split_titles:
result.append(entry)
continue
for existing in result: for existing in result:
if existing["title_normalized"] == entry[ if existing["title_normalized"] == entry[
"title_normalized" "title_normalized"
@@ -257,7 +339,11 @@ def dedupe_entries(entries: list[dict]) -> list[dict]:
def rebuild_artifacts( def rebuild_artifacts(
raw_dir: Path, titles_path: Path, unidentified_path: Path raw_dir: Path,
titles_path: Path,
unidentified_path: Path,
split_titles: set[str] = frozenset(),
edits: list[dict] | None = None,
) -> tuple[list[dict], dict[str, list[dict]]]: ) -> tuple[list[dict], dict[str, list[dict]]]:
"""Regenerate titles.json and unidentified.json from the per-photo raw """Regenerate titles.json and unidentified.json from the per-photo raw
cache. A raw file is either an object with titles/unidentified or a bare cache. A raw file is either an object with titles/unidentified or a bare
@@ -279,7 +365,7 @@ def rebuild_artifacts(
photo = raw_file.name.removesuffix(".json") photo = raw_file.name.removesuffix(".json")
if data.get("unidentified"): if data.get("unidentified"):
unidentified[photo] = data["unidentified"] unidentified[photo] = data["unidentified"]
deduped = dedupe_entries(entries) deduped = dedupe_entries(apply_title_edits(entries, edits or []), split_titles)
titles_path.parent.mkdir(parents=True, exist_ok=True) titles_path.parent.mkdir(parents=True, exist_ok=True)
atomic_write_text( atomic_write_text(
titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n" titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n"
@@ -290,6 +376,35 @@ def rebuild_artifacts(
return deduped, unidentified return deduped, unidentified
def replay_titles(cfg: Config) -> None:
"""Re-derive titles.json after a stored split or edit changed the rules.
Prefers the raw caches (per-photo cue fidelity); without them (trimmed
or hand-built data) it replays the current titles.json entries through
the same edit + dedupe path, exploding multi-photo entries first so
photo-level splits and targeted edits can take hold."""
splits = load_title_splits(cfg.title_splits_path)
edits = load_title_edits(cfg.title_edits_path)
raw_dir = cfg.extract_raw_dir
if raw_dir.is_dir() and any(raw_dir.glob("*.json")):
rebuild_artifacts(
raw_dir, cfg.titles_path, cfg.unidentified_path, splits, edits
)
return
if not cfg.titles_path.exists():
return
exploded: list[dict] = []
for entry in json.loads(cfg.titles_path.read_text()):
photos = entry.get("source_photos") or []
if len(photos) > 1:
exploded.extend({**entry, "source_photos": [p]} for p in photos)
else:
exploded.append(dict(entry))
deduped = dedupe_entries(apply_title_edits(exploded, edits), splits)
atomic_write_text(
cfg.titles_path, json.dumps(deduped, indent=2, ensure_ascii=False) + "\n"
)
def run_extract( def run_extract(
cfg: Config, cfg: Config,
*, *,
@@ -370,7 +485,11 @@ def run_extract(
typer.echo(f" {photo.name}: {len(result['titles'])} title(s){note}") typer.echo(f" {photo.name}: {len(result['titles'])} title(s){note}")
deduped, unidentified = rebuild_artifacts( deduped, unidentified = rebuild_artifacts(
raw_dir, cfg.titles_path, cfg.unidentified_path raw_dir,
cfg.titles_path,
cfg.unidentified_path,
load_title_splits(cfg.title_splits_path),
load_title_edits(cfg.title_edits_path),
) )
typer.echo(f"Wrote {len(deduped)} unique title(s) to {cfg.titles_path}.") typer.echo(f"Wrote {len(deduped)} unique title(s) to {cfg.titles_path}.")
if failed: if failed:
+11 -9
View File
@@ -22,7 +22,7 @@ from rapidfuzz import fuzz
from bggpipe.bgg_client import BGGAuthError, BGGClient, BGGQueueTimeout, client_for from bggpipe.bgg_client import BGGAuthError, BGGClient, BGGQueueTimeout, client_for
from bggpipe.config import Config from bggpipe.config import Config
from bggpipe.extract import cues_conflict from bggpipe.extract import cues_conflict, load_title_splits
from bggpipe.fsio import atomic_write_csv from bggpipe.fsio import atomic_write_csv
from bggpipe.models import ( from bggpipe.models import (
RECOGNIZED_MATCH_STATUSES, RECOGNIZED_MATCH_STATUSES,
@@ -406,9 +406,7 @@ def resolve_entry(client: BGGClient, entry: TitleEntry) -> MatchRow:
# type=rpgitem (same API, same token). A hit becomes a LOCAL # type=rpgitem (same API, same token). A hit becomes a LOCAL
# library citizen: identified and enriched, never uploaded (diff # library citizen: identified and enriched, never uploaded (diff
# routes rpgitem rows to local_only). # routes rpgitem rows to local_only).
cands = _plausible_candidates( cands = _plausible_candidates(client, entry, entry.title_raw, types="rpgitem")
client, entry, entry.title_raw, types="rpgitem"
)
if not cands: if not cands:
for head in _truncation_heads(entry.title_raw): for head in _truncation_heads(entry.title_raw):
cands = _plausible_candidates( cands = _plausible_candidates(
@@ -430,7 +428,11 @@ class MergeEvent:
bgg_id: str bgg_id: str
def dedupe_matches(rows: list[dict], titles: list[TitleEntry]) -> list[MergeEvent]: def dedupe_matches(
rows: list[dict],
titles: list[TitleEntry],
split_titles: set[str] = frozenset(),
) -> list[MergeEvent]:
"""Post-resolve dedupe: rows resolving to the same (bgg_id, version_id — """Post-resolve dedupe: rows resolving to the same (bgg_id, version_id —
or both version-unknown) are the same physical game seen twice (a typo or both version-unknown) are the same physical game seen twice (a typo
read, a partial spine) UNLESS their extraction cues conflict, which read, a partial spine) UNLESS their extraction cues conflict, which
@@ -464,6 +466,8 @@ def dedupe_matches(rows: list[dict], titles: list[TitleEntry]) -> list[MergeEven
# re-running resolve must never overturn that (spec: re-runs # re-running resolve must never overturn that (spec: re-runs
# lose no work, least of all review decisions) # lose no work, least of all review decisions)
continue continue
if normalize_title(row["title_raw"]) in split_titles:
continue # human-split title: per-photo rows stay separate
key = ( key = (
row["bgg_id"], row["bgg_id"],
row["version_id"] if is_confident_version(row) else "", row["version_id"] if is_confident_version(row) else "",
@@ -603,9 +607,7 @@ def run_resolve(
row_dict = paired_by_id.get(id(entry)) row_dict = paired_by_id.get(id(entry))
if row_dict is not None: if row_dict is not None:
photos = ";".join(entry.source_photos) photos = ";".join(entry.source_photos)
if row_dict["source_photos"] != photos and not row_dict.get( if row_dict["source_photos"] != photos and not row_dict.get("dedupe_veto"):
"dedupe_veto"
):
# provenance follows the entry — except on split/vetoed rows, # provenance follows the entry — except on split/vetoed rows,
# whose per-copy photo sets are human-authored # whose per-copy photo sets are human-authored
row_dict["source_photos"] = photos row_dict["source_photos"] = photos
@@ -639,7 +641,7 @@ def run_resolve(
typer.echo(f" {entry.title_raw!r} -> {row.match_status}: {detail}{version}") typer.echo(f" {entry.title_raw!r} -> {row.match_status}: {detail}{version}")
all_rows = existing_rows + [row.to_csv() for row in new_rows] all_rows = existing_rows + [row.to_csv() for row in new_rows]
merges = dedupe_matches(all_rows, entries) merges = dedupe_matches(all_rows, entries, load_title_splits(cfg.title_splits_path))
if new_rows or photos_updated or merges: if new_rows or photos_updated or merges:
write_matches(cfg.matches_path, all_rows) # atomic full rewrite write_matches(cfg.matches_path, all_rows) # atomic full rewrite
if merges: if merges:
+21
View File
@@ -258,6 +258,27 @@ class ReviewSession:
self._save(copies[0]) self._save(copies[0])
return copies return copies
def drop_rows(self, title_raw: str, photos: list[str] | None = None) -> int:
"""An edit invalidated these rows — the BGG match was made against
the uncorrected read. Remove them so resolve re-queries with the
fix; `photos` narrows the cull to one copy of a split title."""
self.reload_if_changed()
def stale(row: dict) -> bool:
if row["title_raw"] != title_raw:
return False
if photos is None:
return True
row_photos = {p for p in row["source_photos"].split(";") if p}
return bool(row_photos & set(photos))
keep = [r for r in self.rows if not stale(r)]
dropped = len(self.rows) - len(keep)
if dropped:
self.rows = keep
self._save()
return dropped
def veto_merge(self, row: dict) -> None: def veto_merge(self, row: dict) -> None:
"""The human says these are NOT the same physical game: restore the """The human says these are NOT the same physical game: restore the
row as a distinct, human-confirmed match.""" row as a distinct, human-confirmed match."""
+15
View File
@@ -310,6 +310,21 @@ button.danger { color: var(--stop-ink); border-color: var(--stop); background: #
text-underline-offset: 2px; text-underline-offset: 2px;
} }
.catalog a:hover { color: var(--accent-ink); } .catalog a:hover { color: var(--accent-ink); }
.catalog tr.editrow td { background: #fff; border-top: none; padding: .2rem .5rem .7rem; }
.editform { display: flex; gap: .7rem; align-items: end; flex-wrap: wrap; }
.editform label {
display: flex; flex-direction: column; gap: .15rem;
font-size: .72rem; text-transform: uppercase; letter-spacing: .06em;
color: var(--ink-soft);
}
.editform input {
font: inherit; font-size: .85rem; color: var(--ink);
border: 1px solid var(--board-edge); border-radius: var(--radius);
padding: .25rem .45rem; background: var(--board);
}
.editform input:focus { outline: 2px solid var(--accent); outline-offset: 1px; }
.editactions { display: flex; gap: .5rem; }
.edithint { font-size: .78rem; color: var(--ink-soft); align-self: center; }
.chip { .chip {
font-size: .68rem; text-transform: uppercase; letter-spacing: .06em; font-size: .68rem; text-transform: uppercase; letter-spacing: .06em;
border-radius: 999px; padding: .1rem .55rem; white-space: nowrap; border-radius: 999px; padding: .1rem .55rem; white-space: nowrap;
+64 -7
View File
@@ -7,12 +7,37 @@
<script> <script>
"use strict"; "use strict";
let CATALOG = []; let CATALOG = [];
let EDITING = null; // lineKey of the row whose editor is open
function lineKey(c) { return c.title_raw + "|" + c.photos.join(";"); }
function editorRow(c) {
const cue = c.cues || {};
return `
<tr class="editrow"><td colspan="5">
<form class="editform" data-title="${esc(c.title_raw)}"
data-photos="${esc(c.photos.join(";"))}">
<label>Title <input name="title" value="${esc(c.title_raw)}" required></label>
<label>Publisher <input name="publisher" value="${esc(cue.publisher || "")}"></label>
<label>Edition <input name="edition" value="${esc(cue.edition || "")}"></label>
<label>Year <input name="year" value="${esc(cue.year ?? "")}" inputmode="numeric" size="6"></label>
<label>Language <input name="language" value="${esc(cue.language || "")}"></label>
<span class="editactions">
<button type="submit" class="primary">save</button>
<button type="button" class="canceledit">cancel</button>
</span>
<span class="edithint">saving re-queues this title for resolve with the corrected data</span>
</form>
</td></tr>`;
}
function render() { function render() {
const q = document.getElementById("catsearch").value.trim().toLowerCase(); const q = document.getElementById("catsearch").value.trim().toLowerCase();
const sorted = [...CATALOG].sort((a, b) =>
a.title_raw.localeCompare(b.title_raw, undefined, { sensitivity: "base" }));
const rows = q const rows = q
? CATALOG.filter(c => (c.title_raw + " " + c.bgg_name).toLowerCase().includes(q)) ? sorted.filter(c => (c.title_raw + " " + c.bgg_name).toLowerCase().includes(q))
: CATALOG; : sorted;
document.getElementById("catcount").innerHTML = document.getElementById("catcount").innerHTML =
`<b>${rows.length}</b> of <b>${CATALOG.length}</b> title(s)`; `<b>${rows.length}</b> of <b>${CATALOG.length}</b> title(s)`;
document.getElementById("catbody").innerHTML = rows.length document.getElementById("catbody").innerHTML = rows.length
@@ -27,13 +52,17 @@ function render() {
<td class="meta">${c.photos.map(p => <td class="meta">${c.photos.map(p =>
`<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>` `<a href="/photos/view/${encodeURIComponent(p)}">${esc(p)}</a>`
).join(", ")}</td> ).join(", ")}</td>
<td>${c.can_split <td class="rowactions">${c.can_split
? `<button class="split" data-title="${esc(c.title_raw)}" ? `<button class="split" data-title="${esc(c.title_raw)}"
data-photos="${esc(c.photos.join(";"))}" data-rowix="${c.row_ix}" data-photos="${esc(c.photos.join(";"))}"
data-rowix="${c.row_ix ?? ""}"
title="one line, several boxes? make each photo its own copy"> title="one line, several boxes? make each photo its own copy">
split into copies</button>` split into copies</button>`
: ""}</td> : ""}
</tr>`).join("") + `</table></div>` <button class="edit" data-key="${esc(lineKey(c))}"
title="fix a misread title or add cues you already know">edit</button>
</td>
</tr>` + (EDITING === lineKey(c) ? editorRow(c) : "")).join("") + `</table></div>`
: `<p class="empty">${CATALOG.length : `<p class="empty">${CATALOG.length
? "No titles match that filter." ? "No titles match that filter."
: `Nothing extracted yet — start on the <a href="/photos">photos page</a>.`}</p>`; : `Nothing extracted yet — start on the <a href="/photos">photos page</a>.`}</p>`;
@@ -42,6 +71,7 @@ function render() {
let LAST = null; let LAST = null;
async function refresh() { async function refresh() {
const state = await fetchJSON("/api/state"); const state = await fetchJSON("/api/state");
if (EDITING) return; // never repaint under an open editor
const payload = JSON.stringify(state.catalog); const payload = JSON.stringify(state.catalog);
if (payload === LAST) return; if (payload === LAST) return;
LAST = payload; LAST = payload;
@@ -50,6 +80,14 @@ async function refresh() {
} }
document.getElementById("catbody").addEventListener("click", async e => { document.getElementById("catbody").addEventListener("click", async e => {
const cancel = e.target.closest("button.canceledit");
if (cancel) { EDITING = null; LAST = null; render(); refresh().catch(() => {}); return; }
const edit = e.target.closest("button.edit");
if (edit) {
EDITING = EDITING === edit.dataset.key ? null : edit.dataset.key;
render();
return;
}
const b = e.target.closest("button.split"); const b = e.target.closest("button.split");
if (!b) return; if (!b) return;
const n = b.dataset.photos.split(";").length; const n = b.dataset.photos.split(";").length;
@@ -58,10 +96,29 @@ document.getElementById("catbody").addEventListener("click", async e => {
const res = await apiPost("/api/split", { const res = await apiPost("/api/split", {
title_raw: b.dataset.title, title_raw: b.dataset.title,
source_photos: b.dataset.photos, source_photos: b.dataset.photos,
row_ix: Number(b.dataset.rowix), row_ix: b.dataset.rowix === "" ? null : Number(b.dataset.rowix),
}); });
if (res) refresh().catch(() => {}); if (res) refresh().catch(() => {});
}); });
document.getElementById("catbody").addEventListener("submit", async e => {
const f = e.target.closest("form.editform");
if (!f) return;
e.preventDefault();
const orig = CATALOG.find(c => lineKey(c) === EDITING) || {};
const cue = orig.cues || {};
const v = name => f.elements[name].value;
const body = { title_raw: f.dataset.title, source_photos: f.dataset.photos };
if (v("title").trim() !== f.dataset.title) body.title_new = v("title").trim();
if (v("publisher") !== (cue.publisher || "")) body.publisher = v("publisher");
if (v("edition") !== (cue.edition || "")) body.edition = v("edition");
if (v("year") !== String(cue.year ?? "")) body.year = v("year");
if (v("language") !== (cue.language || "")) body.language = v("language");
if (Object.keys(body).length <= 2) { EDITING = null; render(); return; }
const res = await apiPost("/api/edit-title", body);
if (res) { EDITING = null; LAST = null; refresh().catch(() => {}); }
});
document.getElementById("catsearch").addEventListener("input", render); document.getElementById("catsearch").addEventListener("input", render);
refresh().catch(err => errorBanner(err.message || err)); refresh().catch(err => errorBanner(err.message || err));
pollLoop(refresh, 5000, () => showBanner("")); pollLoop(refresh, 5000, () => showBanner(""));
+114 -2
View File
@@ -38,6 +38,11 @@ from rich.console import Console
from bggpipe.bgg_client import BGGClient, cached_paths from bggpipe.bgg_client import BGGClient, cached_paths
from bggpipe.config import DEFAULT_REVIEW_PORT, Config from bggpipe.config import DEFAULT_REVIEW_PORT, Config
from bggpipe.extract import (
record_title_edit,
record_title_split,
replay_titles,
)
from bggpipe.fsio import atomic_write_bytes, atomic_write_text from bggpipe.fsio import atomic_write_bytes, atomic_write_text
from bggpipe.jobs import JobRunner from bggpipe.jobs import JobRunner
from bggpipe.models import ( from bggpipe.models import (
@@ -148,6 +153,19 @@ class SplitBody(BaseModel):
row_ix: int | None = None row_ix: int | None = None
class EditBody(BaseModel):
"""A human correction to an extracted read. None = leave that field
alone; empty string = clear it."""
title_raw: str
source_photos: str = ""
title_new: str | None = None
publisher: str | None = None
edition: str | None = None
year: str | None = None
language: str | None = None
class RunBody(BaseModel): class RunBody(BaseModel):
dry_run: bool = True # upload only; the safe direction is the default dry_run: bool = True # upload only; the safe direction is the default
limit: int | None = None limit: int | None = None
@@ -377,12 +395,21 @@ def create_app(
"version_name": row["version_name"] if row else "", "version_name": row["version_name"] if row else "",
"merged_into": row.get("merged_into", "") if row else "", "merged_into": row.get("merged_into", "") if row else "",
"row_ix": _ix_of(session.rows, row) if row else None, "row_ix": _ix_of(session.rows, row) if row else None,
"cues": {
"publisher": entry.publisher_hint if entry else "",
"edition": entry.edition_hint if entry else "",
"year": entry.year_hint if entry else None,
"language": entry.language_hint if entry else "",
},
"can_split": bool( "can_split": bool(
row row
and row["match_status"] in RECOGNIZED_MATCH_STATUSES and row["match_status"] in RECOGNIZED_MATCH_STATUSES
and not row.get("dedupe_veto") and not row.get("dedupe_veto")
and len(row["source_photos"].split(";")) > 1 and len(row["source_photos"].split(";")) > 1
), )
# no matches row yet (title still awaiting resolve): the
# split is a titles.json decision, no BGG data at stake
or bool(not row and entry and len(entry.source_photos) > 1),
"split_copy": bool(row and row.get("dedupe_veto")), "split_copy": bool(row and row.get("dedupe_veto")),
} }
@@ -695,16 +722,101 @@ def create_app(
raise HTTPException(400, f"unknown action {body.action!r}") raise HTTPException(400, f"unknown action {body.action!r}")
return state() return state()
def _find_entry(title_raw: str, source_photos: str):
photos = [p for p in source_photos.split(";") if p]
return next(
(
e
for e in session.titles
if e.title_raw == title_raw
and (not photos or list(e.source_photos) == photos)
),
None,
)
@app.post("/api/split") @app.post("/api/split")
def api_split(body: SplitBody) -> dict: def api_split(body: SplitBody) -> dict:
with lock: with lock:
revision["n"] += 1 revision["n"] += 1
_refuse_if_rewriting() _refuse_if_rewriting()
row = find_row(body.title_raw, body.source_photos, body.row_ix) freshen()
row = session.find_row(body.title_raw, body.source_photos, body.row_ix)
if row is not None:
try: try:
session.split_row(row) session.split_row(row)
except ValueError as err: except ValueError as err:
raise HTTPException(400, str(err)) from err raise HTTPException(400, str(err)) from err
else:
# no matches row yet — the title is still awaiting resolve;
# splitting is purely a titles.json (extraction) decision
entry = _find_entry(body.title_raw, body.source_photos)
if entry is None:
raise HTTPException(
404, "title not found — titles.json changed underneath?"
)
if len(entry.source_photos) < 2:
raise HTTPException(
400, "only a multi-photo title can be split into copies"
)
# persist the decision so every future extract rebuild and
# resolve dedupe keeps the copies apart, then re-derive
# titles.json so each copy carries its own photo's cues
record_title_split(cfg.title_splits_path, body.title_raw)
replay_titles(cfg)
return state()
@app.post("/api/edit-title")
def api_edit_title(body: EditBody) -> dict:
with lock:
revision["n"] += 1
_refuse_if_rewriting()
freshen()
entry = _find_entry(body.title_raw, body.source_photos)
if entry is None:
raise HTTPException(
404, "title not found — titles.json changed underneath?"
)
record: dict = {"match": body.title_raw}
photos = [p for p in body.source_photos.split(";") if p]
same_title = [e for e in session.titles if e.title_raw == body.title_raw]
if photos and len(same_title) > 1:
# several copies/editions share this title: the fix targets
# only the copy the human was looking at
record["photos"] = photos
if body.title_new is not None:
corrected = body.title_new.strip()
if not corrected:
raise HTTPException(400, "corrected title cannot be empty")
if corrected != body.title_raw:
record["title_raw"] = corrected
for key, value in (
("publisher_hint", body.publisher),
("edition_hint", body.edition),
("language_hint", body.language),
):
if value is not None:
record[key] = value.strip()
if body.year is not None:
year = body.year.strip()
if year and not year.isdigit():
raise HTTPException(400, "year must be a number")
record["year_hint"] = int(year) if year else None
if not any(
key in record
for key in (
"title_raw",
"publisher_hint",
"edition_hint",
"language_hint",
"year_hint",
)
):
raise HTTPException(400, "nothing to change")
record_title_edit(cfg.title_edits_path, record)
replay_titles(cfg)
# rows matched against the uncorrected read are stale: drop
# them so the next resolve re-queries with the fix
session.drop_rows(body.title_raw, photos if "photos" in record else None)
return state() return state()
@app.post("/api/veto-merge") @app.post("/api/veto-merge")
+91
View File
@@ -12,9 +12,16 @@ import typer
from bggpipe.config import Config from bggpipe.config import Config
from bggpipe.extract import ( from bggpipe.extract import (
apply_title_edits,
dedupe_entries, dedupe_entries,
load_title_edits,
load_title_splits,
parse_vision_response, parse_vision_response,
prepare_image, prepare_image,
rebuild_artifacts,
record_title_edit,
record_title_split,
replay_titles,
run_extract, run_extract,
) )
@@ -167,6 +174,90 @@ def test_dedupe_conflicting_years_stay_separate():
assert len(deduped) == 2 assert len(deduped) == 2
def test_dedupe_split_titles_never_merge():
deduped = dedupe_entries(
[_entry("Wiz-War", "a.jpg"), _entry("Wiz-War", "b.jpg")],
split_titles={"wiz war"},
)
assert len(deduped) == 2 # the human said: separate physical copies
def test_edits_fix_misreads_before_dedupe():
# a corrected misspelling merges with the correctly-read sighting
edits = [{"match": "Hebarceos", "title_raw": "Herbaceous"}]
deduped = dedupe_entries(
apply_title_edits(
[_entry("Hebarceos", "a.jpg"), _entry("Herbaceous", "b.jpg")], edits
)
)
assert len(deduped) == 1
assert deduped[0]["title_raw"] == "Herbaceous"
assert deduped[0]["source_photos"] == ["a.jpg", "b.jpg"]
def test_edits_chain_and_target_photos():
edits = [
{"match": "Wiz-War", "photos": ["a.jpg"], "edition_hint": "7th Edition"},
{"match": "Wiz-War", "title_raw": "Wiz-War!", "photos": ["a.jpg"]},
# made later, against the renamed title — must chain onto the result
{"match": "Wiz-War!", "photos": ["a.jpg"], "year_hint": 1997},
]
entries = apply_title_edits(
[_entry("Wiz-War", "a.jpg"), _entry("Wiz-War", "b.jpg")], edits
)
assert entries[0]["title_raw"] == "Wiz-War!"
assert entries[0]["edition_hint"] == "7th Edition"
assert entries[0]["year_hint"] == 1997
assert entries[1] == _entry("Wiz-War", "b.jpg") # untargeted copy untouched
def test_stores_roundtrip_and_replay_from_raw(tmp_path):
cfg = Config(data_dir=tmp_path / "data", photos_dir=tmp_path / "photos")
raw = cfg.extract_raw_dir
raw.mkdir(parents=True)
for photo in ("a.jpg", "b.jpg"):
(raw / f"{photo}.json").write_text(
json.dumps({"titles": [_entry("Wiz-War", photo)], "unidentified": []})
)
record_title_split(cfg.title_splits_path, "Wiz-War")
record_title_split(cfg.title_splits_path, "wiz war") # dupe, normalized away
record_title_edit(
cfg.title_edits_path,
{"match": "Wiz-War", "photos": ["a.jpg"], "edition_hint": "7th Edition"},
)
replay_titles(cfg)
titles = json.loads(cfg.titles_path.read_text())
assert [e["source_photos"] for e in titles] == [["a.jpg"], ["b.jpg"]]
assert titles[0]["edition_hint"] == "7th Edition"
assert len(load_title_splits(cfg.title_splits_path)) == 1
assert len(load_title_edits(cfg.title_edits_path)) == 1
# a later full rebuild (a real extract run) honors the same stores
rebuild_artifacts(
raw,
cfg.titles_path,
cfg.unidentified_path,
load_title_splits(cfg.title_splits_path),
load_title_edits(cfg.title_edits_path),
)
assert len(json.loads(cfg.titles_path.read_text())) == 2
def test_replay_without_raw_caches_explodes_merged_entries(tmp_path):
cfg = Config(data_dir=tmp_path / "data", photos_dir=tmp_path / "photos")
cfg.data_dir.mkdir(parents=True)
merged = _entry("Wiz-War", "a.jpg")
merged["source_photos"] = ["a.jpg", "b.jpg", "c.jpg"]
cfg.titles_path.write_text(json.dumps([merged, _entry("Catan", "a.jpg")]))
record_title_split(cfg.title_splits_path, "Wiz-War")
replay_titles(cfg)
titles = json.loads(cfg.titles_path.read_text())
by_title = {}
for e in titles:
by_title.setdefault(e["title_raw"], []).append(e["source_photos"])
assert by_title["Wiz-War"] == [["a.jpg"], ["b.jpg"], ["c.jpg"]]
assert by_title["Catan"] == [["a.jpg"]] # non-split entries survive intact
def test_dedupe_upgrades_confidence(): def test_dedupe_upgrades_confidence():
deduped = dedupe_entries( deduped = dedupe_entries(
[ [
+9
View File
@@ -439,6 +439,15 @@ def test_dedupe_versions_must_agree():
assert dedupe_matches(rows3, []) == [] assert dedupe_matches(rows3, []) == []
def test_dedupe_matches_skips_split_titles():
rows = [
_mrow("Wiz-War", "94", "a.jpg"),
_mrow("Wiz-War", "94", "b.jpg"),
]
assert dedupe_matches(rows, [], split_titles={"wiz war"}) == []
assert all(not r["merged_into"] for r in rows)
def test_dedupe_is_idempotent_and_skips_merged(): def test_dedupe_is_idempotent_and_skips_merged():
rows = [ rows = [
_mrow("Jokin Ha...", "193621", "a.jpg"), _mrow("Jokin Ha...", "193621", "a.jpg"),
+129
View File
@@ -432,3 +432,132 @@ def test_duplicate_rows_are_individually_decidable_via_row_ix(tmp_path):
) )
rows = read_m(cfg.matches_path) rows = read_m(cfg.matches_path)
assert [r["match_status"] for r in rows] == ["unmatched", "rejected"] assert [r["match_status"] for r in rows] == ["unmatched", "rejected"]
# -- catalog curation: rowless splits and title edits ---------------------
def _rowless_wizwar(cfg) -> None:
"""Add a multi-photo extracted title with NO matches row (still awaiting
resolve — the Wiz-War shape)."""
titles = json.loads(cfg.titles_path.read_text())
titles.append(
{
"title_raw": "Wiz-War",
"confidence": "high",
"publisher_hint": "Fantasy Flight Games",
"edition_hint": "",
"source_photos": ["shelf.jpg", "shelf2.jpg"],
}
)
cfg.titles_path.write_text(json.dumps(titles))
def test_rowless_multiphoto_title_is_splittable(tmp_path):
cfg = make_cfg(tmp_path)
_rowless_wizwar(cfg)
web = TestClient(create_app(cfg, client=unauthorized_client(tmp_path)))
line = next(
c
for c in web.get("/api/state").json()["catalog"]
if c["title_raw"] == "Wiz-War"
)
assert line["status"] == "awaiting_resolve"
assert line["can_split"] is True
res = web.post(
"/api/split",
json={
"title_raw": "Wiz-War",
"source_photos": "shelf.jpg;shelf2.jpg",
"row_ix": None,
},
)
assert res.status_code == 200
copies = [c for c in res.json()["catalog"] if c["title_raw"] == "Wiz-War"]
assert [c["photos"] for c in copies] == [["shelf.jpg"], ["shelf2.jpg"]]
assert all(c["can_split"] is False for c in copies)
# the decision persisted: the store holds it and titles.json is split
assert "Wiz-War" in json.loads(cfg.title_splits_path.read_text())
per_photo = [
e["source_photos"]
for e in json.loads(cfg.titles_path.read_text())
if e["title_raw"] == "Wiz-War"
]
assert per_photo == [["shelf.jpg"], ["shelf2.jpg"]]
def test_split_still_requires_multiple_photos(tmp_path):
web, _ = make_client(tmp_path)
res = web.post(
"/api/split",
json={
"title_raw": "Fresh Off The Shelf",
"source_photos": "shelf.jpg",
"row_ix": None,
},
)
assert res.status_code == 400
def test_edit_title_corrects_read_and_requeues_row(tmp_path):
web, cfg = make_client(tmp_path)
# Citadels has an ambiguous matches row; correcting its read must drop
# the stale row (it was searched with the old data) and store the fix
res = web.post(
"/api/edit-title",
json={
"title_raw": "Citadels",
"source_photos": "shelf.jpg",
"title_new": "Citadels: Dark City",
"edition_hint": None,
"year": "2004",
},
)
assert res.status_code == 200
state = res.json()
titles = [c["title_raw"] for c in state["catalog"]]
assert "Citadels: Dark City" in titles and "Citadels" not in titles
corrected = next(
e
for e in json.loads(cfg.titles_path.read_text())
if e["title_raw"] == "Citadels: Dark City"
)
assert corrected["year_hint"] == 2004
assert corrected["publisher_hint"] == "Fantasy Flight" # untouched cue kept
assert not any(r["title_raw"] == "Citadels" for r in read_matches(cfg.matches_path))
(record,) = json.loads(cfg.title_edits_path.read_text())
assert record["match"] == "Citadels"
def test_edit_title_rejects_empty_and_noop(tmp_path):
web, _ = make_client(tmp_path)
assert (
web.post(
"/api/edit-title",
json={
"title_raw": "Citadels",
"source_photos": "shelf.jpg",
"title_new": " ",
},
).status_code
== 400
)
assert (
web.post(
"/api/edit-title",
json={"title_raw": "Citadels", "source_photos": "shelf.jpg"},
).status_code
== 400
)
assert (
web.post(
"/api/edit-title",
json={
"title_raw": "No Such Game",
"source_photos": "shelf.jpg",
"title_new": "X",
},
).status_code
== 404
)