0.4 s
until all 200 of checkout’s threads are stuck
Each call waits out the 2-second timeout: 500/s × 2 s needs 1,000 threads. Cart and payment requests then queue behind them and fail too.
A breaker can’t make a slow dependency fast. So what does it actually change?
The trick
How calls fail: in about 0 ms with a clear error, not after 2 s with a thread held hostage.
Three states, and the only way back is a test
Closed counts outcomes. Open rejects every call instantly for a cool-off. Half-open lets a few probes through: success closes it, failure reopens it.
When does it trip?
Bad: Failures in a row
Any success resets it; one dead replica of two never trips it.
Good: A rate over a window
Trip when 50% fail or crawl, over at least 20 calls.
Count only what says the dependency is sick
- Count timeouts, connection errors and 5xx
- Count 429 and 503; honour Retry-After
- Ignore 4xx validation, auth and business errors
- Count everything, and your own bad deploy trips it
Shard B is down. Where does the breaker go?
One breaker sees 25% failing: it never trips at 50% and blocks healthy shards at 10%. One per shard sees 100% on B, 0% elsewhere.
What’s the fallback?
Bad: The database behind the cache
Amazon did, around 2001: caches failed together, the database locked up, the website went down.
Good: Cheap and local
Stale data from memory or a static default, exercised often.
Remember
A breaker changes how calls fail: fast and honest. One per dependency or shard, real faults only, a cheap fallback, an alert.