Name a failure by what the caller sees
Lanes are replicas. A shut one is easy, because checks catch it and traffic moves. A slow one still passes liveness checks, so nothing replaces it.
80.4%
of requests fail when only 5% hang
1,000 threads serve 1,000 req/s at 100 ms. Hanging calls wait out a 100 s deadline and want 5,000 threads. Failing fast, it would be 5%.
Gray failure: checks green, users failing. Who’s right?
Bad: The health check
Its probe takes another path and misses the broken one.
Good: The callers
Compare caller-side errors per replica with health status, and alert on the gap.
One replica of four dies. Its load doesn’t.
Four share 1,200 req/s, each serving at most 400. Lose one and the rest run at 100%, zero headroom. The next wobble ejects another, and it cascades.
The 10-second blip ended. Why is the database still down?
A metastable loop
Each query times out and is retried, so 280 × 2 = 560 req/s against a limit near 300. Fix the loop, not the trigger.
Cloudflare, 2019: one regex, every HTTP core near 100%
A query of death crashes or wedges each replica it reaches. A config push reaches every one in seconds, so stage config like code.
Forgotten dependencies fail everywhere at once
- DNS: Facebook’s servers withdrew their own routes
- Certificates: every handshake fails, nothing deployed
- Clocks: a leap second made durations negative
- Quotas: a signed 32-bit ID stops at 2,147,483,647
The interview line
Name failures stop, slow or wrong. Make slow look like down, watch callers as well as checks, keep headroom and budget retries.