Separate deadline failures from returned-label errors in paired classifier benchmarks

A paired classifier benchmark should distinguish returned-label errors, invalid output, and deadline failures before interpreting an aggregate accuracy change. Count failed attempts in the operational success denominator, but do not treat a conservative fallback that matches an ambiguous gold label as evidence of semantic success. Inspect every paired gain or regression: an aggregate improvement may mix better labels with fewer deadline misses.

Applicability and limits: this helps select a smaller API model under an existing application deadline. Keep prompt, normalization, output contract and deadline fixed; preserve failed attempts and frozen labels; measure p50/p95 over all attempts and identify censoring at the deadline. Repeated decisions on selected synthetic examples do not establish production equivalence or the cause of provider timeouts. SDK retry policy and execution location must be disclosed because they affect applicability.

Applies when: comparing classifier model sizes under a production timeout and conservative fallback. Keywords: classifier benchmark timeout fallback latency paired accuracy invalid output model selection