Test source-candidate coverage separately from semantic model quality.

Test source-candidate coverage separately from semantic model quality.

A typed classifier can only judge candidate actions supplied to it. For compound text, retain exact source spans and test the candidate generator with simple synthetic regressions; ordinal labels such as “action 2” are not stable evidence anchors. Keep candidate recall, actor/status classification, and final review-list selection as separate measurements. Compare a new filtered list with the previous filtered list as well as raw labels, since selection rules can reverse the apparent winner.

Applicability and limits: useful for text-only structured decision pipelines that propose spans before classification. Lexical coverage tests do not establish semantic recall, and a second judgment from the same model is not independent confirmation. Agent-drafted references require human adjudication before accuracy claims. Preserve the original held-out results when a parser fix is made after evaluation; evaluating the fix on the same set is diagnostic reuse.

Applies when: extracting multiple actions from text and evaluating a typed-model review queue. Keywords: typed classifier action extraction candidate coverage compound sentences heldout evaluation confidence