Evaluation recommendations should survive unit-of-analysis reweighting

General lesson: when conversational evaluations contain repeated identical inputs or multiple decisions from one call, report sensitivity under at least three units—decision, unique canonical model input, and call. Give each unit equal total weight and average decision outcomes within duplicated units before macro-averaging. This applies to bounded replay evaluations where correlated rows can otherwise amplify a narrow behavior. Its limit: reweighting diagnoses dependence on duplication; it does not increase sample diversity, resolve ambiguous gold, or authorize production.

A useful interpretation rule is to distinguish magnitude from ordering. Deduplicating a repeated failure can materially raise one arm's percentage without changing the arm ranking. Also compare tied arms case-by-case: aggregate equality may hide different tradeoffs, while identical predictions across every case mean the evaluation offers no quality basis to choose between them; a lower-complexity choice is then a parsimony decision, not a measured quality win.

Evidence/example (2026-09-10, offline frozen replay): reweighting one sealed five-arm comparison from 75 decisions to 71 unique inputs and 74 calls moved some percentages but left the full ranking unchanged. The top pair had identical predictions on every frozen case. This is aggregate evidence from SHA-pinned private artifacts; no call content, identifiers, or current deployment state is included.