Split evaluations by causal family and provenance, not identifier novelty

Evaluation separation must be defined at the causal family/provenance level, not by changed identifiers or exact-input hashes. When authored examples reuse a contract, generator logic, template grid, failure observations, or previously inspected evaluation inputs, they may be clean training or challenge cases but are not an independent representative holdout. Limits: family separation reduces direct and structural leakage; it does not establish production representativeness, semantic independence, or future-distribution performance.

Recommended checks: - Exact canonical input digest, normalized latest-turn text, and assistant-anchor overlap. - Canonical numeric equivalence across glyphs, spoken digits, oh/o, repetition words, punctuation, leading zeros, substrings, and small edit distance. - Typed template skeletons after replacing identifiers/entities with placeholders; hold template IDs and generator versions together. - Near-neighbor checks on caller text and full pre-target conversation using token n-grams/MinHash plus semantic-neighbor review. - Indivisible provenance groups for source call/entity, authoring prompt, generator/version, template, seed, failure-analysis source, and reviewer. - Separate contract-matched synthetic interpolation from held-out-template, stateful, grounding, ASR, and mixed-intent challenges. - If a suite was designed after inspecting failures or evaluation inputs, report it as adaptive challenge evidence and retire those source families from confirmatory claims.

Evidence/example — 2026-09-09: Input-only review of the retained original195 TRAIN packet and proposed authored192 packet found 0 exact full-input overlaps and 0 multi-digit maximal-sequence overlaps, but 20 single-digit-overlap rows and 5 normalized single-digit utterance overlaps against supplied cohorts. Against TRAIN195 alone, normalized caller text had three unique exact overlaps represented by four proposed rows. TRAIN195 already contains authored short-fragment, complete-identifier, same-turn restart, and control families; authored192 expands the same capabilities through a quota-balanced template grid and adds callback-later. Its builder explicitly consulted supplied195+78+48 inputs, all rows declare non-production parity, and the fixed system/tool contract itself states the short-fragment rule. Therefore authored192 is defensibly an authored synthetic capability/training suite, not a fresh representative holdout. Counter-attempt state, valid-length ungrounded context, clarification delivery, and a later complete identifier after clarification are absent and remain separate stateful/system challenges. Provenance: local pinned input files and construction receipt only; protected labels were not inspected and no inference or external data was used.