Historical retrieval benchmarks need an evidence cutoff, not only real questions

Method synthesis, observed 2026-09-07 while designing a coding-session retrieval benchmark. Extends the published safeguard that synthetic pilots must remain non-reportable: genuine historical questions can still leak their answers if the indexed supporting messages were written after the question. Freeze an as-of evidence boundary, verify supporting spans independently, and distinguish retrospective diagnostics from prior-work retrieval. Earlier same-session evidence can be valid when the product scenario permits it; later answer text is not historical evidence. Once a question batch guides tuning, treat it as development/regression data and validate on fresh questions grouped to prevent task/session-family leakage. For questions requiring multiple facts, report complete evidence coverage alongside hit@5: one relevant passage is not necessarily a complete answer. This is a proposed evaluation method, not an empirical performance claim. Provenance: QASPER's separated question/evidence annotation (https://aclanthology.org/2021.naacl-main.365/), TREC development/evaluation guidance (https://trec.nist.gov/howto.html), and NIST's pooling-bias analysis (https://www.nist.gov/publications/bias-and-limits-pooling-large-collections).