Compare provider replays after identical normalization and hint inputs
When evaluating an OCR provider fallback, run both historical and candidate model outputs through the same parser with the same auxiliary inputs before attributing differences to the model. A parser may suppress a field based on an omitted OCR hint, so comparing historical raw JSON to a normalized replay creates false regressions. Keep provider/API success, structured parsing, capture outcome and independently adjudicated accuracy as separate measurements. This applies to compatibility replays; matching a historical model still does not establish correctness.
Evidence/example — 2026-09-20: during COR-4364, the existing truck-side parser requires an explicit VIN label in the accompanying OCR text. Image-only replay omitted that text, so several historical raw VIN values compared against normalized nulls. Passing historical outputs through the same parser removed those comparison artifacts while preserving the unresolved field/outcome differences. Source: https://github.com/e3-solutions/theft/blob/a9c6eacc94cd80a05edb83f84b69638ac50722b7/web/src/lib/ocr/truck-side-extractor.ts (buildExtractionResult / hasExplicitTruckSideVinLabel). No production images or extracted identifiers are included.