When reference labels change, invalidate benchmark claims and reuse frozen predictions

When reference labels change, invalidate benchmark claims before spending on more model inference. If the original inputs, preprocessing, prompts, output schema and saved predictions are unchanged, compare candidates again by rescoring the frozen predictions against one explicitly versioned reference release. Preserve both score versions, exclusions and denominator changes. A change to the inference contract needs a new run; a label repair alone usually does not.

Treat label publication, score validity, runtime readiness and permission to affect decisions as separate states. Local corrections are not a usable team release until another runner can restore them. A valid development score does not establish generalization; corrected labels do not make a repeatedly inspected test set fresh again. Missing accuracy evidence should not prevent independent work on a non-authoritative, resource-bounded runtime path where authorized. Conversely, a deployed service does not validate its extraction decisions.

This is a workflow recommendation for small image-model teams with retained per-case outputs, not a claim that process changes alone improve model quality. Keep one expert responsible for label/scoring semantics and one delivery owner responsible for a consumable candidate and runtime; automate routine version checks rather than requiring the expert to supervise every run. Existing scorers and manifests are preferable to a new orchestration framework.

Applies when: Several image-model experiments share evolving reference labels and existing prediction artifacts, and handoffs are slowing a release decision. Keywords: OCR benchmark label version provenance rescore cached predictions handoff shadow evaluation