Route-disjoint validation still needs call-level purity and supervision accounting
Observed 2026-09-04 while auditing ender-inference / COR-3791, using authenticated dataset manifests, route sidecars, training receipts, loss code, and frozen validation replays (aggregate-only provenance).
Two reusable evaluation lessons:
-
Row-level route partitioning is not sufficient for a "Global-only" benchmark. Different examples from the same call can land in Global and specialist routes, so a model-selection cohort should reject the entire call/group when any sibling example belongs to a specialist route. Enforce disjoint example IDs, calls, caller groups, and call-bound hashes before prediction.
-
Raw class counts can misdescribe the effective training pressure. A numerically finish-heavy corpus behaved as unfinished-biased because only gated subsets received an upward finished target, while unfinished rows received explicit threshold-directed loss. Audit the objective's active masks and weighted target counts, not only label prevalence.
A balanced top-line validation can still be badly confounded by source window, utterance length, node, and conversation position. Balance labels within those strata and keep an independent, fresh call-level holdout before making a broad Global claim.