Paired exact-match ties can hide meaningful error transitions

Paired exact-match evaluation should keep its binary primary endpoint while separately classifying error transitions; “both incorrect” is not evidence that two systems behaved the same. Component metrics must also be conditioned on prerequisites: an argument object is not “correct” for the expected action when the model selected a different action, even if the JSON happens to be identical. This method applies to structured-action comparisons with fixed gold labels; it does not establish that one wrong output is behaviorally better without a predeclared route-specific validator.

Recommended method: retain exact-match McNemar/fixes/regressions; add mutually exclusive per-arm outcome classes (request failure, protocol invalid, predicted-call schema invalid, wrong tool, correct tool/wrong arguments, exact correct), a full paired transition matrix, and tool-only paired discordance. Report raw argument-object equality separately from arguments_exact = tool_correct AND raw_equal. Recompute classes from raw captures rather than trusting submitted summaries.

Evidence/example — 2026-09-08, synthetic local inspection of ender-inference commit 3167717: a schema-valid wrong tool with the same argument object as the gold tool yielded tool_correct=false, arguments_correct=true, exact_match=false; different no-argument routes make this especially easy. The existing strict exact result remained correct, but the component label and a single undifferentiated both-wrong bucket could support false partial interpretations. Source provenance: repository training/mc_pilot/baseline.py plus synthetic responses only; no private calls or production results.