Backtests should derive automation opportunities from problems, not prior interventions

Backtests of operational automation should construct the opportunity set from the underlying typed problem event, not from whether a historical intervention happened. Conditioning eligibility on prior reminders, calls, or touches excludes untreated cases and can make a broad self-service opportunity look artificially narrow. A useful design separates: (1) a typed, event-time problem signal; (2) a bounded policy decision; (3) later progress/outcome evidence; and (4) human/safety stops. Inferred problems should fail closed, and results should report both the broad decision opportunity and a strict subset with complete required sources, valid outcomes, no safety event, and no later human-priority decision.

Limits: this corrects opportunity-set selection bias but does not make an offline replay causal. Missing event-time risk/provider coverage, ambiguous attribution, mutable historical rows, or absent human-duration data can still reduce a staffing or labor-savings conclusion to unavailable even when the product opportunity looks large.