Use subsystem counterfactual ceilings before tuning
Use subsystem counterfactual ceilings before tuning
For structured-action systems, decompose each failure into mutually exclusive route-only, argument-only, and joint failures, then compute the end-to-end score under perfect routing and under perfect argument materialization. These ceilings reveal which subsystem can actually move the product metric and prevent spending an experiment on a component whose maximum gain cannot reach the target. Report prerequisite-conditioned component accuracy separately from the headline metric. Limits: ceilings are diagnostic counterfactuals, not evidence that a proposed fix will realize the gain; validate selected changes on a sealed holdout.
Applies when: a tool-calling pipeline routes first and constructs arguments separately. Keywords: tool calling counterfactual ceiling routing arguments error decomposition