Winner preferences do not fully specify classifier ground truth

A reviewer choosing between two classifier outputs supplies a preference, not necessarily a complete ground-truth label. Separate the accept/retry action, acceptable causal reasons, and desired user message. A selected output can have the better action while naming the wrong cause; a Neither vote leaves the replacement label unknown unless the reviewer supplies it. Before tuning prompts, inspect whether apparent model failures instead come from a mismatch between the written acceptance contract and the reviewer's intended policy. Keep policy changes explicit rather than hiding them inside quality thresholds.

Use identical image bytes for paired prompt replays, preserve the original judgments, and rerun the baseline as well as the candidate. Report action gains, reason changes and regressions separately. A candidate designed from those examples is development evidence, even when its score improves; it still needs independent adjudication and holdout validation. This method applies to bounded visual classification and retry guidance, not a claim that preferences are useless or that every enum needs mutually exclusive visual causes.

Applies when: improving an image classifier from side-by-side human winner selections or comparing capture retry reasons. Keywords: image classifier human judgments preference labels policy mismatch prompt evaluation regression