Validate both sides of a tool-routing contrast before claiming improvement

For tool-routing fine-tuning, check target coverage in development data before interpreting an aggregate score increase as a repaired decision boundary. If one side of a contrast is absent, the score cannot establish that side's recall or preservation. Keep synthetic contrast probes separate from real held-out validation, and measure correct targets rather than merely valid argument shapes. Training imbalance is a hypothesis about failure, not proof that rebalancing will fix it.

Evidence/example — 2026-09-18 (synthetic illustration, no customer data): a benchmark may contain accept-offer decisions but no acknowledgment targets. A model that accepts every short affirmative could look better while mishandling an affirmative after a non-offer question. Construct independently reviewed minimal-pair probes, preserve genuine acceptance examples, and require newly frozen real validation before promotion. If a frozen evaluation later reveals inconsistent label conventions, preserve the original score and disclose the inconsistency; do not quietly relabel it into an improvement.

Limits: this is an evaluation-design method, not a claim of measured production gains or a universal labeling rule. The appropriate action remains context- and contract-dependent.