80.4%
of requests fail when just 5% hang
1,000 threads serve 1,000 requests a second at 100 ms. If 5% hang against a 100 s deadline, they want 5,000 threads. Only 19.6% get served.
Put each defence where its failure lands
No single box holds them all. The edge admits, pools contain damage, every call bounds its wait, and a queue absorbs what can wait.
A slow provider gets its own pool, a timeout and a breaker
Only its pool fills. A 2-second timeout bounds each wait; when failures burst, the breaker fails fast. Database and events keep serving.
Retry harder?
Bad: Every layer retries
Four tries at browser, frontend and backend: one tap, 64 database queries.
Good: One layer, budgeted
Only the layer above the database retries: twice at most, full jitter, a 10% budget.
A debit call times out. That means “unknown”, not “failed”: the debit may have happened.
The trick
Retry with one idempotency key per operation, resent every time. The server stores key → result and replays it.
Past capacity, more traffic means fewer useful answers
Late replies are wasted work. Shed early and cheaply, lowest criticality first: in a tatkal rush, checkout survives and recommendations don’t.
Safe alone, risky together
- Stop retrying when the breaker opens
- Fit inner timeouts and retries inside the deadline
- Keep liveness checks local, off the database
- Run fallbacks regularly, not first in an outage
Remember
Walk the request path and put a pattern where each failure lands: shed at the edge, a timeout, pool and breaker per call, retries at one layer with idempotency keys.