The full picture
- Retry amplification is multiplicative across layers — say the arithmetic out loud: if every layer retries 3×, a three-layer call chain turns 1 user request into up to 27 downstream calls at exactly the moment the downstream is least able to serve them. The fix is to retry at one layer only (usually the outermost that can still be useful) and to make the others fail fast.
- Jitter is not a nicety, it's the fix: synchronized exponential backoff makes every client retry at the same instants, producing traffic spikes that re-break a recovering service. Full jitter (
sleep = random(0, base·2^n)) spreads them out — AWS's architecture blog on this is the standard reference and it's worth naming. - Timeout budgets must shrink down the chain: if the user-facing request has a 2s budget, a service 3 hops deep cannot have a 5s timeout — it will be working on a request nobody is waiting for. Propagate a deadline (gRPC does this natively) rather than configuring independent timeouts that don't add up.
- Circuit breakers convert a slow failure into a fast one, which is the actual goal — threads freed, pools healthy, a fallback served. Understand the states (closed → open after a failure threshold → half-open probe) and the parameter that matters most: how many probes you allow in half-open, since a stampede of probes re-kills a recovering service.
- Bulkheads and load shedding are the underrated pair: separate connection pools per dependency mean a slow recommendations service can't starve checkout; and shedding low-priority traffic at the edge (with a 429 and a
Retry-After) protects the core. Graceful degradation is a design output, not an accident.
| Pattern | Prevents | Watch out |
|---|---|---|
| Timeout + deadline propagation | unbounded resource hold | budgets must shrink per hop |
| Backoff + full jitter | synchronized retry spikes | cap total attempts and elapsed time |
| Circuit breaker | slow failure consuming threads | half-open probe stampede |
| Bulkhead (pool per dependency) | one dependency draining all threads | more pools to size and monitor |
| Load shedding / rate limit | total collapse under overload | must shed the right traffic |
| Idempotency | duplicates created by retries | requires stable keys — see delivery semantics |