A blocked thread waits, but nobody else can use it
Each checkout needs 2 ms of CPU and 48 ms of waiting on the database and payments. Its thread is held for all 50 ms.
Payments slows from 40 ms to 190 ms. It’s not down. Safe?
The cliff
No. At 1,000 requests a second, 200 threads fill once requests take 200 ms. It isn’t down, yet it saturates you.
Just add more threads?
Bad: 4,000 threads
Each reserves a 1 MB stack, and the slow dependency gets more calls.
Good: Hold no thread while waiting
Async I/O or virtual threads; waiting costs no thread.
Detect it: busy threads up, CPU down
Graph busy threads, the pool maximum and CPU on one panel. Thread dumps then show hundreds of identical stacks in a socket read.
Bulkhead the slow dependency
Normally 40 payment calls are in flight; cap them at 100. When payments takes 1 s, 100 checkouts a second succeed, the rest fail fast, and the cart keeps its threads.
Never hold a scarce thread while you wait
- Async I/O end to end, or virtual threads
- A timeout on every remote call
- A bulkhead per dependency
- Cache, queue or parallelise the wait
1,250
queries a second through 10 pooled connections
Async removes the thread limit, not the work. More calls reach the database, and its pool becomes the next ceiling: 10 connections ÷ 8 ms.
Remember
Throughput is threads ÷ latency, so never hold a thread while you wait. Go async or use virtual threads, then bound each dependency with a timeout for how long and a bulkhead for how many.