When a data gap is really a policy ambiguity

A zero-count mechanism gap is not automatically a training-data gap: freeze real examples and require independent target review before building the SFT packet, because the exercise may reveal that the prompt/rubric does not determine a unique action. If reviewers disagree on the desired tool for every negative example, fail closed—do not train only the easy positive controls, force consensus, or optimize labels toward the hoped-for evaluation result. Resolve product policy first, then revise the rubric and collect/review supervision under the clarified rule.

Practical gate: 1. Freeze a balanced, split-disjoint mechanism packet before labels. 2. Review inputs independently with model outputs and expected targets hidden. 3. Require exact target-set agreement on the negative mechanism, not just agreement on controls. 4. Treat systematic disagreement as a policy/rubric finding, not annotator noise. 5. Make candidate assembly fail closed when the targeted negative set has zero clear consensus.

Limits: systematic disagreement does not itself identify the correct policy, and a small mechanism-focused packet is not prevalence evidence. It only shows that retraining is not yet justified under the current specification.

Evidence/example (2026-09-11, bounded offline audit): ten real training-partition cases were frozen as five negative single-unassigned-location cases and five explicit two-endpoint controls, with call/group/digest separation from prior training selections and heldout metadata. Two blind reviewers agreed on all five controls but disagreed on all five negatives: one allowed either partial direction, while the other inferred the first requested side. Neither selected the proposed clarification action. The training packet was therefore not materialized; existing training data remained unchanged, and no evaluation data, inference, or training run was used.