Protect evaluation families by discourse mechanism, not surface variation

A protected authored evaluation set should define separation on the mechanism that makes a decision hard—turn topology, semantic role of numbers, repair scope, temporal contrast, and mixed-intent precedence—not on new digits, punctuation, synonyms, or assistant-prompt variants. Apply this when synthetic training and evaluation share the same task contract: inventory exposed families first, explicitly reject surface-only variants, store inputs apart from labels, freeze exact scoring before inference, and require every admitted family to pass rather than relying on an aggregate. Limits: this establishes only bounded separation from named source artifacts; it does not create a fresh natural-data holdout, prove population representativeness, or imply universal semantic disjointness.

Evidence/example — 2026-09-10

For one carrier-identifier routing packet, two independent read-only reviews rejected plain fragments, ordinary complete identifiers, whole-sequence restarts, and direct control paraphrases because those mechanisms were already exposed. They admitted only multi-turn latest-turn grounding, plausible-length incidental-number disambiguation, explicit alternate-identifier elicitation, contrastive wait/callback negation, and mixed-intent precedence. The first review caught two false separation claims: incidental values were too short to force semantic-role reasoning, and alternate-identifier wording did not explicitly satisfy the contract. After correction, both reviewers passed 12 cases across five families. This is authored diagnostic evidence against four pinned local cohorts, not a production or model-quality result.