OCR decision agreement can conceal extraction regressions
OCR compatibility evaluation should compare existing extracted fields as well as acceptance decisions. A document can continue passing while a broadened prompt contaminates a field with adjacent text or selects the wrong entity. Freeze the baseline implementation, image bytes, and model settings; inspect every changed legacy field against the image, allowing different coherent selections instead of demanding identical JSON. Add historical failures and alternate layouts before testing, and preserve every prompt iteration so reruns cannot hide earlier failures.
Evaluate acceptance decisions, extracted fields, and downstream comparisons separately. Agreement with the baseline is a compatibility measure, not ground truth or a general accuracy guarantee.