27×
database queries for one tap
App, gateway and order service each try three times, so 3 × 3 × 3. The multiplier sleeps on good days and wakes up when the database is failing.
Do more retries mean more availability?
Bad: “Retry everything, three times”
If a dependency fails from overload, each retry lands on it.
Good: Retries fix independent faults
1% random failures, one retry: 0.01% fail, for 1% more traffic.
The trigger ends. The outage doesn’t.
At 280 QPS, a 10 s blip pushes latency past the 1 s timeout. Each request now costs 2 queries, 560 against a 300-QPS database, and goodput stays at zero.
Detect it: errors first, then the traffic jumps
Attempts per request climb past 1.5 while edge traffic stays flat. Traces show a staircase of calls, each cut off at the timeout.
Make retries scarce
- One layer owns retries; the rest try once
- Cap at 2–3 attempts, with jittered backoff
- A 10% retry budget per client
- Shed load early; propagate deadlines
1.1×
worst-case load with a 10% retry budget
Google caps attempts at 3 and retries at 10% of a client’s traffic. In a 2,000-request-a-second fault test, the database sees about 2,200, not 54,000.
The failover is fixed, yet the service stays down. Autoscaling won’t save it.
The way out
Cut load hard at the edge, turn retries off, drop stale queued work, then ramp back 10% → 25% → 50% → 100%.
Remember
Retries are extra load on whatever is failing. One layer owns them, on a budget, with jitter, and servers shed early. A retry is fine when it’s bounded, spread out, safe and owned.