Durable shadow/canary analytics should persist decisions before side effects

General lesson

For shadow/canary automation analytics, persist one append-only policy-decision record per evaluated candidate before any external side effect. Keep downstream delivery evidence separate. If the decision record cannot be persisted, fail closed to the existing human path. Shadow mode can then exercise selection and policy logic without sending messages or mutating the queue.

Applicability and limits

This pattern applies when an operator needs durable coverage and decision counts from both shadow and live runs, and must preserve the legacy path when disabled or degraded. A per-candidate decision table does not by itself prove end-to-end delivery, causal recovery, labor savings, or complete run coverage. Aggregate views should remain explicitly partial unless a durable run-level receipt proves that a whole scan completed.

Evidence/example — 2026-09-16

In Shield COR-4280 / PR #903, dashboard operational analytics were moved from optional log-query credentials to a direct database aggregate over an append-only evaluation table. The runtime writes a bounded typed outcome for every candidate, writes before live claiming/dispatch, and falls back to human handling if persistence fails. Shadow evaluation records decisions but performs no send. A final review caught and corrected two risks: falsely labeling populated aggregates as complete without a run receipt, and logging persistence failure below error severity. Focused tests, type checks, migration validation, static checks, unit/integration checks, local Postgres evidence, and the CI production build passed.

Related prior work

The append-only evaluation-history guidance supports immutable policy-decision records. The crash-safe idempotency post is related to the ordering and uniqueness boundary, while this example adds the shadow/no-side-effect and explicit partial-coverage constraints.