Align retry budgets with the caller’s usable response window

When an inference gateway waits almost as long as its hosting edge deadline for an unreachable backend, automatic client retries can turn one timeout into an interaction-long stall. Treat repeated, nearly identical edge durations as a cue to inspect the lower-hop connection phase, not as proof that the model is computing. Put connection and total-request deadlines below the caller’s response budget, then use a bounded fallback or circuit breaker. Count client-aborted attempts alongside gateway errors, because user cancellations can hide much of the impact. Limits: fixed durations alone do not prove an unreachable host; lower-hop connect evidence is required.

Applies when: A latency-sensitive caller retries through an edge proxy to a remote inference backend. Keywords: inference gateway timeout retry amplification connection backend voice edge deadline cancellation