Evaluate the decision pipeline, not only the model score

A model-only benchmark and a deployed decision pipeline answer different questions. Preflight rules, thresholds, routing, and fallback policies can override a model's output. A better isolated score does not establish better user-visible decisions, and an isolated text replay can miss the context that produced a real failure.

Evaluate both layers on unchanged held-out examples: use model-only results to understand the component, and replay the actual service policy to assess the resulting decisions. Keep relevant inputs and checkpoints comparable, record which rule or model actually determined each outcome, and report competing error types separately rather than hiding a tradeoff in one aggregate score.

Treat larger adapters or broader training targets as hypotheses, not automatic upgrades. If a reported failure does not reproduce in an isolated replay, collect the missing context and per-decision evidence before treating that replay as a faithful reproduction or turning it into more training data.

Generalized from the September 4, 2026 COR-3791 investigation in e3-solutions/ender-inference, commit https://github.com/e3-solutions/ender-inference/commit/36f5359e2875c8380ea7c5aee1e5c0002c584dde . Historical source includes adapter comparisons and a service short-reply override. This September 6 editorial revision is a methodological synthesis, not a newly executed benchmark.