diff --git a/CLAUDE.md b/CLAUDE.md index 1106766..0b68809 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -47,10 +47,11 @@ Full design lives in `bgg-shelf-pipeline-spec.md` (read it before changing pipel - Base game vs. expansion vs. new edition is the top failure mode — bias matching toward `ambiguous` over auto-match ("Wingspan Europe" must not match base Wingspan). - Editions/versions matter: Eric owns multiple editions of some games — each is a separate collection entry (keyed by `collid` on BGG). Never guess a version: no legible cues → `version_unknown` and a version-less collection entry. - Normalize titles (casefold, strip punctuation/articles, special chars like é/&/:) identically on both sides of a match; dedupe across photos but keep `source_photos` provenance. +- **Human curation is durable**: `data/title_splits.json` (titles split into per-photo copies) and `data/title_edits.json` (corrected reads/cues) are replayed on every titles.json rebuild, and both extract's and resolve's dedupe honor them forever. Row-level split decisions also persist via the `dedupe_veto` column. - **RPGs are local-only citizens**: when the board-game search runs dry, resolve falls back to `type=rpgitem` (same geekdo API/token). Matched rpgitems enrich into the library but diff routes them to `local_only` — they must never reach `to_add.csv`/upload (their collection lives on RPGGeek, out of scope). - Detailed BGG API behavior (202 queueing, collection-endpoint quirks, endpoints): use the `bgg-api` skill. **If the spec's BGG behavior changes, update the `bgg-api` skill to match** — they must not drift. ## Git - Remote is self-hosted Gitea 1.26 (`git.kestrelsnest.social/eric/bggpipe`), **not GitHub** — `gh` CLI does not work here. -- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`. +- Commit `data/matches.csv`, `data/to_add.csv`, `data/to_update.csv`, `data/upload_log.csv`, `data/titles.json`, `data/unidentified.json`, `data/unidentified_dismissed.json`, `data/title_splits.json`, `data/title_edits.json`, `data/games.json`, `data/STUB_DATA.marker` (while it applies), and the collection snapshot XMLs. Never commit `data/bgg_cache/`, `data/extract_raw/`, `photos/`, Playwright storage state, or `.env`. diff --git a/data/unidentified_dismissed.json b/data/unidentified_dismissed.json index 3a9f05a..60a0e3b 100644 --- a/data/unidentified_dismissed.json +++ b/data/unidentified_dismissed.json @@ -1,45 +1,90 @@ [ + "IMG_4501.jpeg|Bottom shelf, left portion, above 'Exploding Kittens' box|EXPANSION|Small box with partial text visible, colorful design, title cut off by frame edge", + "IMG_4501.jpeg|Bottom shelf, right side, near Exploding Kittens box|COSMIC|Colorful box spine, blue/green tones, title partially visible but not fully confirmable", + "IMG_4501.jpeg|Middle shelf, purple box beneath Joking Hazard, right side of stack|A Boardgame for 1 to 6 players, ages 8 and up.|Purple/violet box, text describing player count, title not legible from this angle", + "IMG_4501.jpeg|Right side, middle shelf, black spine sticking out between Santorini and purple box below|T|Black spine, single letter visible, rest obscured", + "IMG_4501.jpeg|Top shelf, far left edge, cut off by frame; near 'a Gentle Rain' box||Dark/black box, only a thin sliver visible", + "IMG_4502.jpeg|bottom-left corner of shelf, beneath the History of the World area, left of Dungeon!|A Boardgame for 1 to 6 players, ages 8 and up|purple/tan box with descriptive text visible but no title readable", + "IMG_4502.jpeg|far left side of shelf, stacked horizontally beneath the yellow-floral box, left of Santorini||small colorful box spines, red and white coloring, too angled/blurry to read title", + "IMG_4502.jpeg|left side of shelf, below the horizontal stack, left of Santorini and above Joking Hazard||blue box with white illustration, appears to be a game box but title not legible", + "IMG_4502.jpeg|top-left corner of shelf, above Santorini and to the left of Wiz-War|possibly 'Herbaceous' or similar, text partially cut off|cream/white box with floral yellow illustration, thin spine", "IMG_4505.jpeg|Bottom area, left side near Mantis Falls, small figure/box partially visible||Small gold/tan colored object, possibly a game piece or small box, mostly obscured", + "IMG_4505.jpeg|Top shelf, far left edge behind Cat Crimes, cut off by frame edge||Dark brown/maroon spine with faint gold decorative border, partially visible vertical box", "IMG_4505.jpeg|Top shelf, far left edge, partially cut off by frame, to the left of Cat Crimes||Dark brown/maroon box spine with gold decorative swirl design, portion visible at left edge of image", "IMG_4506.jpeg|Middle area, below and left of Santorini, dark red/maroon box spine with small square icons||Dark reddish-brown spine with small colored square icons in a row, text not legible", + "IMG_4506.jpeg|Right side of image, behind Santorini box, dark box with small visible logo||Dark/black box with a small circular or square colored logo element, mostly obscured by foreground Santorini box", "IMG_4506.jpeg|Right side, background behind Santorini box, partially visible dark box with small game piece icon||Dark background box with a small blue/white icon, mostly obscured by Santorini box in foreground", "IMG_4506.jpeg|Top shelf, above Santorini spine, partially visible box with patterned tiles/squares in yellow, blue, and cream tones||Patchwork/quilt-like pattern of colored squares (yellow, tan, blue) along the top edge of the box, style suggests a tile-based board game", "IMG_4506.jpeg|Top shelf, right side near blue corner box, small red 'X' symbols visible on light blue background|X X|Light blue box edge with red X pattern, possibly a tally or scoring game box", + "IMG_4506.jpeg|Top shelf, spine visible above Santorini box, partially cut off at top of frame||Multicolored patchwork/tile pattern spine in yellow, tan, and blue tones with geometric designs; appears to be a board game box edge but no title text is legible", + "IMG_4506.jpeg|Upper right corner of image, above and right of Santorini spine, next to unidentified patterned box||Small red and blue box corner visible, mostly cut off by frame edge, no discernible text", + "IMG_4507.jpeg|Below 'LABYRINTH', left side of shelf, partially cut off at frame edge|DRAGO...|Green/dark box, only top edge visible, text cut off", "IMG_4507.jpeg|Middle shelf area, partially hidden behind Labyrinth box and below Utter Nonsense, left side of the stack|Dragon...|Green/dark colored box spine, text cut off by adjacent boxes, only 'Dragon' prefix visible", + "IMG_4507.jpeg|Top of stack, right of center, between 'a Gentle Rain' and 'SANToRINI'|PATCH / WORK (possibly)|Colorful patchwork quilt-style pattern on spine, small box", "IMG_4507.jpeg|Upper right area near Santorini and a Gentle Rain, appears to be a narrow spine box|PATCH WORK or similar|Colorful patchwork/quilt-like pattern visible on spine, tan and multicolor squares design", + "IMG_4508.jpeg|Top shelf, center, small yellow tin sitting next to a black box near the stack of papers/magazines, above the Labyrinth box||Small yellow tin container, no visible text, could be a card game tin", "IMG_4508.jpeg|Top shelf, center, yellow tin box sitting next to the Super Mario Checkers box||Small yellow tin, no visible title text, could be a card game tin", "IMG_4508.jpeg|Top shelf, far left, behind the stack of papers/notebooks, next to the framed picture||Small black box with unreadable logo, mostly obscured by papers", + "IMG_4508.jpeg|Top shelf, far left, partially cut off by frame edge, next to a green-framed picture and above the Super Mario Checkers box||Teal/green colored box edge, mostly obscured, shape suggests a game box but no readable text", "IMG_4508.jpeg|Top shelf, right of center, wooden box with metal clasp between the checkers box and the papers stack||Wooden box with brass hardware, no visible title, appears decorative or a game storage box", "IMG_4508.jpeg|Top shelf, right side, partially visible box behind the wooden box, between the stack of papers and the edge of the frame|SCRABBLE (partial letters visible)|Red and teal colored box edge, appears to be a Scrabble-branded box but mostly obscured", + "IMG_4511.jpeg|Middle shelf, behind Superfight boxes, green/black box with skeletal figure artwork - between Superfight stack and the Rook box|...CTHULHU|Dark green box, illustrated skeletal/robed figure, likely a Cthulhu-themed game (e.g., Cthulhu Fluxx or Cthulhu Gloom) but exact title obscured", "IMG_4511.jpeg|Middle shelf, black box stacked below SUPERFIGHT, above ROOK; left side of shelf|...Go... Yourself|Black spine box, small white/colored text, similar size to Superfight box", + "IMG_4511.jpeg|Middle shelf, black spine box stacked above Rook, to the left of the dice tray|...Yourself|Black box with orange/white text, only partial word 'Yourself' legible, title obscured by angle", "IMG_4511.jpeg|Right edge of shelf, partially cut off by frame, next to Cthulhu box|HEX|Blue box with yellow/tan geometric pattern, only edge visible", + "IMG_4511.jpeg|Top right corner of shelf, box mostly out of frame, above the Hex box|Drag...|Orange and blue box, only a fragment of text visible, likely a dragon-themed game title", "IMG_4517.jpeg|Middle of shelf, between Pie Face! (left) and the blue box with bird artwork (right)|...Grenade...|Tall narrow box, dark red/brown color scheme, appears to have a hand grenade or bomb-like illustration", "IMG_4517.jpeg|Middle of shelf, right of the grenade-themed box and left of Walk the Plank!|Bird ... Bird ...|Blue box with tall bird-like cartoon character illustration, small/narrow box shape", + "IMG_4517.jpeg|Middle shelf area, second box from left, between 'Pie Face!' and 'Bird on Your Bread?!'|...renade... Nintendo?|Narrow tall box, red and tan coloring, partially obscured by shadow and angle, appears to have small figure illustration", "IMG_4518.jpeg|Between Mage Wars Academy Core Set and Saberu Merry Men||Thin black spine, largely obscured, possibly a second small game box", "IMG_4519.jpeg|Far right of shelf, white spine box next to the two red 'G' logo boxes|Before I Kill You, Mister Bond... / Kill Dr. No Lucky|White worn spine with black text, possibly James Bond themed game, 'CHEAPEST GAMES' text also visible on spine which may be a store sticker rather than title", + "IMG_4519.jpeg|Left shelf, red spine with black text between the cream D&D box and the red-orange boxes|THE BEST OF RA... (possibly 'RAMPHA' or similar)|Dark red spine with illustrated figure artwork, book-like thickness", + "IMG_4519.jpeg|Middle shelf area, two matching red boxes with orange circular 'G' logo, positioned between the white D&D boxes on the left and the white spined books on the right|G (stylized orange circular logo)|Two identical red boxes, narrow spine width, orange/yellow circular logo with letter G, taped/worn edges", "IMG_4519.jpeg|Right-center of shelf, two matching red boxes standing between the D&D group on the left and the white 'Cheapest Games' spine on the right|Orange 'G' logo visible on both spines|Two identical red boxes with worn/taped edges, orange circular 'G' emblem, appears to be same game duplicated", "IMG_4520.jpeg|Middle shelf, between Carcassonne and Evolution: The Beginning||Plain wooden-colored vertical box, no visible text, appears to be a wood grain finish game box or insert", "IMG_4521.jpeg|Top edge of frame, above the Sleeping Gods box, partially cut off||Dark spines/boxes barely visible at top, too obscured by shadow and cropping to identify", + "IMG_4521.jpeg|Top of shelf, above the Sleeping Gods box, partially cut off by frame edge||Dark spines of several boxes, indistinct due to shadow and angle, colors appear dark blue/black and tan", "IMG_4527.jpeg|Left edge of shelf, partially visible spines to the left of the main game box, on the top shelf|01, T...|Appears to be book-like spines, red and dark covers, stacked vertically, likely not games but partially obscured", + "IMG_4527.jpeg|Right edge of shelf, behind/beside the Red Dragon Inn box, partially cut off by frame|4, 3|Dark colored spine/box edge, numbers visible but title obscured", + "IMG_4528.jpeg|Top shelf, above Wiz-War box, partially obscured by rolled poster/tube on left side|SLUGFEST GAMES|Red and orange colored box with illustrated fantasy/comic style artwork, only top edge visible", "IMG_4528.jpeg|Top shelf, back corner above the Wiz-War box; only the publisher logo area is visible, partially obscured by other items|SLUGFEST GAMES|Colorful red/orange box art, only top publisher banner visible, rest of box obscured", "IMG_4529.jpeg|Below the 1000-piece puzzle box, spine showing armored warrior artwork, above the face-out Risk Lord of the Rings box|The Middle-earth|Dark battle scene with armored figures, likely same LOTR-themed game box viewed from spine angle", "IMG_4529.jpeg|Middle of stack between the puzzle box and Risk Lord of the Rings, tan/cream colored spine with Asian characters and blue castle icon|unreadable Asian characters, blue icon|Tan spine with small blue building/castle graphic and Asian script text", + "IMG_4529.jpeg|Top of stack, above the 1000-piece puzzle box, partially cut off at top of frame|V, 3|Dark spines with minimal visible text, one shows a circular 'V' logo, another shows number '3' on purple/dark background; too obscured to identify title", "IMG_4529.jpeg|Top of stack, second partial box edge, above the puzzle box|0|White/light colored box edge with a circular icon, only corner visible", "IMG_4529.jpeg|Very top of stack, box behind/above the numbered fragments, purple/dark spine||Dark purple or navy spine, mostly obscured, edge of frame", "IMG_4529.jpeg|Very top of stack, partial box edge visible above the puzzle box|3|Dark colored spine/edge, only a sliver visible, number '3' printed in white", + "IMG_4532.jpeg|Top right corner of frame, above Gloomhaven box, partially cut off|GLO...|Fiery orange/red background with dark textured lettering, appears to be another large box possibly related to Gloomhaven expansion", "IMG_4532.jpeg|Top right corner of image, partially cut off, above and behind the Gloomhaven box|GLO... (possibly part of another title starting with GLO)|Dark box with fiery orange/red background and large white curved text, only edge visible", + "IMG_4535.jpeg|Top shelf, far right edge of frame, next to the last visible Etherfields big box||Black box spine, partially cut off by frame edge, appears similar in size to adjacent Etherfields big boxes", + "IMG_4542.jpeg|Bottom right shelf area, dark spine next to unidentified train-themed box||Dark spine, illegible text, appears to be a board game box based on shape", + "IMG_4542.jpeg|Right side shelf, below CATAN, dark spine with red/white text|T...T / Ride (possible)|Dark red/maroon spine with train-like graphic element, narrow box spine", "IMG_4542.jpeg|Right side shelf, below Catan box, next to unidentified black-spined game|T...C...|Teal/orange box with train-like graphic, partially obscured, could be a train-themed game", "IMG_4542.jpeg|Right side shelf, lower row near 'Catan' box||Dark spine with red accent, text too small/blurry to read", + "IMG_4542.jpeg|Top area, small red square box with white 'M' logo, near Dragon Land|M|Small red square box with large white letter M, possibly a classic board game logo", + "IMG_4542.jpeg|Top left corner of shelf, partially cut off by frame edge, above the Skip-Bo box|...O / From the Makers of...|Red box edge with white lettering, likely a card game box, mostly obscured", "IMG_4542.jpeg|Top right shelf, partially hidden behind Dragon Land box|From the Makers of, M|Red box with a large stylized 'M' logo and small icons, likely a Mattel-published card game (possibly UNO), edge cut off by frame", + "IMG_4543.jpeg|Background, lower right area behind the hand holding Scrabble Slam, blurred and out of focus||Indistinct colorful box shapes, cannot make out title or artwork details due to heavy blur", + "IMG_4543.jpeg|Right edge of frame, middle row, next to the Scrabble Slam box|CAT...|Dark colored box spine, partially cut off at right edge of photo, red/dark background with light text", + "IMG_4543.jpeg|Top right corner of frame, behind and above the Scrabble Slam box, partially cut off by frame edge|Dogs Kitchen|Small box or book with illustrated cartoon-style cover, colorful, appears to feature a dog character; too small/blurry to confirm if it's a game", "IMG_4543.jpeg|right edge of image, behind Scrabble Slam box|CAT|Yellow/orange box edge visible, partial text 'CAT' in bold letters, rest obscured by foreground box", "IMG_4543.jpeg|top right corner of image, above and behind the main Scrabble Slam box|Dogs Kitchen|Small blue/purple box partially cut off at top edge, illustrated cartoon-style artwork", "IMG_4544.jpeg|Background shelf behind the Cat Stax box, blurred and out of focus, appears to be a row of game/book boxes||Multiple colorful spines/boxes including orange, white, and dark tones, too blurry to make out any text", + "IMG_4544.jpeg|Background shelf behind the Cat Stax box, top of frame, multiple stacked boxes with no clearly identifiable neighbors||Blurry stack of colorful boxes (orange, white, dark tones) in background, out of focus, too indistinct to read any text", + "IMG_4545.jpeg|Bottom right corner of image, yellowish box edge visible||Yellow/tan colored box spine, text not legible due to angle and cropping", + "IMG_4545.jpeg|Lower left shelf area, dark colored box partially visible behind hand holding Pocket Farkel||Dark/black box edge visible, no legible text", + "IMG_4545.jpeg|Top right shelf, box with 'Ravensburger' text visible, title obscured|Ravensburger|Colorful box, brand name visible but game title not legible", "IMG_4545.jpeg|Top shelf, far right edge, partially cut off by frame border|Ravens...|Yellow/orange box with small logo resembling a bird, likely Ravensburger branding but game title not legible", + "IMG_4545.jpeg|Top shelf, right side, above the 'Ravensburger' labeled box, partial title visible at top edge of frame|DRAGON ...and|Orange/red box with dragon and fantasy artwork, cut off at top of image", "IMG_4545.jpeg|Top shelf, upper right corner, above and behind the Pocket Farkel tin, to the right of 'Bugs in the Kitchen'|DRAGON, land|Colorful fantasy-style box art, orange/red hues, dragon imagery visible, title cut off at top edge of frame", "IMG_4547.jpeg|Top area, right of the maze-like box, small orange/red box||Small red/orange box, text illegible, largely obscured", "IMG_4547.jpeg|Top center-right, small red/orange box near top edge||Small reddish box with indistinct artwork, mostly cut off at top frame", + "IMG_4547.jpeg|Top left area, next to the '...AZE' box, second spine from left|UTTLES or similar|Bright blue box with light green accents, small toy-like image visible on top", "IMG_4547.jpeg|Top left corner of image, partially visible behind main box|AZE|Blue and green box, appears to be a puzzle or maze-themed game, mostly cut off by frame edge", + "IMG_4547.jpeg|Top left corner of shelf, above the main held box, leftmost visible spine|...AZE|Blue and green box, partial text visible, cut off at frame edge", + "IMG_4547.jpeg|Top right area, red-orange box between the dark box and top edge|Roll Pla...|Red/orange box with illustrated artwork, text partially cut off", "IMG_4547.jpeg|Top right corner of image|Dark T...|Dark blue/black box with white text, appears to be 'Dark T' something, possibly 'Dark Tower' or similar, cut off at frame edge", + "IMG_4547.jpeg|Top right corner of shelf, above and to the right of the main held box|Dark Terr... or Dark Territory|Dark navy/black box with red and white text, partially obscured by other boxes", "IMG_4547.jpeg|Top right, near 'Dark T' box||Colorful box with red/orange tones, illustration unclear, partially obscured by other boxes", "IMG_4552.jpeg|Left edge of frame, vertical spine, sharply cropped|Giv... / Ci...|Dark spine with white lettering, partially visible artwork, cropped by frame edge", "IMG_4552.jpeg|Top left corner of frame, behind and above the Dark Cults package, no clearly identified games adjacent due to cropping|'Giv...'|Dark colored box with white/red text fragment, likely a card or board game box mostly out of frame", diff --git a/src/bggpipe/config.py b/src/bggpipe/config.py index dbe3965..d834cdb 100644 --- a/src/bggpipe/config.py +++ b/src/bggpipe/config.py @@ -66,6 +66,18 @@ class Config: def games_path(self) -> Path: return self.data_dir / "games.json" + @property + def title_edits_path(self) -> Path: + # human corrections to extracted reads (misspellings, known cues) — + # replayed on every titles.json rebuild + return self.data_dir / "title_edits.json" + + @property + def title_splits_path(self) -> Path: + # titles the human declared to be MULTIPLE physical copies: extract + # and resolve dedupe must never cross-photo-merge them again + return self.data_dir / "title_splits.json" + @property def dismissed_path(self) -> Path: return self.data_dir / "unidentified_dismissed.json" diff --git a/src/bggpipe/extract.py b/src/bggpipe/extract.py index 9e075ce..9f53770 100644 --- a/src/bggpipe/extract.py +++ b/src/bggpipe/extract.py @@ -240,11 +240,93 @@ def _merge(a: dict, b: dict) -> dict: return merged -def dedupe_entries(entries: list[dict]) -> list[dict]: - """Collapse same-normalized-title sightings unless their cues conflict.""" +# the fields a human correction may override on an extracted entry +EDIT_FIELDS = ( + "title_raw", + "publisher_hint", + "edition_hint", + "year_hint", + "language_hint", + "art_notes", +) + + +def load_title_edits(path: Path) -> list[dict]: + """Human corrections to raw reads (fixed misspellings, cues the owner + knows offhand). Each record: {"match":