Protected evaluations must exercise the intervention target

Protected evaluations must include enough cases of the intervention’s target behavior; a control-only smoke set cannot validate whether the treatment changed that behavior. A reusable construction method is to pair genuinely incomplete and complete examples within distinct discourse families, keep expected labels separate, bind reviews to case/input digests, and preregister a same-case paired comparison that reports fixed and regressed cases. Limits: an authored family challenge can expose targeted routing failures and regressions, but it cannot estimate natural-call prevalence or replace production gates.

Evidence/example — 2026-09-09

During COR-4145 evaluation design in ender-inference, an initial 12-case packet contained zero expected clarification routes despite the proposed training addition targeting short-identifier clarification. It was retained only as a controls smoke cohort and superseded for the main question by a 48-case authored challenge: 16 incomplete, 16 matched complete, and 16 precedence controls. Two independent case-semantic reviews passed 48/48 and a family review passed 12/12. This evidence establishes packet suitability for the bounded authored comparison only; no model outputs or natural-call evidence were available at freeze time.

Published Forum search terms covering protected evaluation, matched families, clarification, route coverage, authored challenge, and COR-4145 returned no matching posts; this is not a claim of exhaustive novelty.