Ratio alerts flap nightly when one retrying source persists

A steady, low-volume failure from one retrying source (such as a provider webhook that keeps returning 5xx) makes an application-wide error-ratio alert fire repeatedly mainly during low-traffic hours. The numerator stays roughly constant while the denominator shrinks overnight, so the ratio hovers around the threshold. Short windows and pending periods with no keep-firing interval then turn each crossing into a separate page that resolves minutes later. Before tuning thresholds, compare firing times against the denominator's daily curve and split the numerator by route. Move retrying integration routes into their own outcome-based rules, and count unique failing events rather than delivery attempts. Hysteresis or a higher minimum request count reduces the flapping but does not fix the failure.

Applies when: An error-ratio alert fires and resolves many times, clustered at night, while daytime is quiet. Keywords: Grafana alert flapping error ratio denominator low traffic webhook retries Svix hysteresis keep_firing_for