1.2 s
until 200 shared threads are all stuck
Cooks are threads. Recommendations slows to its 2 s timeout: 150 calls/s × 2 s wants 300 threads. Checkout fails too, though payments is healthy.
Same 200 threads, but recommendations may hold only 30
Payments gets 24, the database pool 20. When recommendations hangs, only its compartment floods, and at least 126 threads stay free.
Recommendations is full. Call 31 arrives.
Bad: Wait for a free permit
That just moves the pile-up one step earlier.
Good: Reject in microseconds
Serve the cached “popular near you” list, or a fast, honest error.
Size it with Little’s Law
- Healthy: 150 req/s × 0.12 s p99 = 18 permits
- Add headroom: 18 × 1.5, rounded up to 30
- Keep it under rate × timeout (45) so slowdowns trip it
A 300 ms timeout already caps each call. Why add a bulkhead?
The catch
A cricket-final spike to 450/s wants 135 threads for one widget. The timeout bounds latency; only the bulkhead caps calls in flight.
The same wall works from threads to regions
Each rung down contains a bigger, rarer failure at a higher price: a slow call at the top, a lost region at the bottom.
120
slots, where one shared pool needs only 70
Three dependencies each peak at 40, never together. Compartments cover the sum of peaks, mostly idle: isolate only risky or critical paths.
Remember
Cap what each dependency may hold and reject fast when full. A bulkhead limits damage while it’s slow; a breaker stops calls once broken.