Bulkheads: stop one failure sinking the ship

A bulkhead splits a shared resource (threads, connections, queue slots, CPU, whole fleets) into separate compartments, so that when one dependency slows down or one tenant floods you, it exhausts only its own compartment and everything else keeps working. You pay with capacity that sits idle in one compartment while another is full, and with more limits to size and watch.

Picture it first: one kitchen, ten cooks

A food-court kitchen has ten cooks and three counters: dosa, chaat and biryani. Any cook takes any order, so idle cooks help whichever line is long.

Then the tandoor starts misbehaving, and each biryani takes twenty minutes instead of five. Every cook who picks up a biryani order is stuck at the tandoor until it is done. Orders keep arriving, so soon all ten cooks are standing there, and nobody makes a dosa on a perfectly good griddle. One broken oven has sunk the kitchen.

The fix: four cooks on dosa, three on chaat, three on biryani. A slow tandoor can now tie up at most three cooks. Biryani customers wait, or hear “biryani is 30 minutes today”, but the other lines move as usual. The price: on a quiet evening, a bored chaat cook can no longer help a busy dosa counter.

That is a bulkhead, named after the watertight walls in a ship’s hull: a breach floods one compartment, not the whole ship. Map it back:

• Cooks are threads, connections or workers: anything held while work waits.
• Counters are dependencies, request types or tenants.
• The broken tandoor is a dependency that is slow, not down.
• The fixed split is the compartment; the bored chaat cook is stranded capacity.

Why a slow dependency sinks everything

A dependency that is down is, oddly, the easy case: connections are refused within milliseconds and the thread moves on. A slow dependency is the dangerous one, because every call holds a resource for the whole time it waits.

Little’s Law says how many. In any stable system, the average number of requests in flight equals the arrival rate times the time each one spends inside:

in-flight = arrival rate × latency

A service calling recommendations 150 times a second at 0.12 s per call keeps about 150 × 0.12 = 18 threads busy. If recommendations slows down and every call waits for the 2-second client timeout, the same traffic needs 150 × 2 = 300 threads. With 200 worker threads shared by every endpoint, they run out, and requests for every endpoint start to queue.

Three things make it worse:

• It is fast. Stuck threads pile up at the arrival rate, so a 200-thread pool is gone in about a second, long before anyone sees an alert.
• The victim looks guilty. Checkout’s error graphs light up although payments is healthy. The cause is a shared resource, not checkout’s code.
• Retries multiply it. Callers that time out retry, raising the arrival rate on the stuck path: a retry storm.

The Google SRE book traces this chain to cascading failure: more requests in flight eat threads, memory and file descriptors until health checks fail and load moves to the next server. A bulkhead breaks the chain at the first link.

The fix: one compartment per dependency

A food-delivery order-api runs as pods with 200 worker threads each. At dinner peak, one pod sees:

About 40 of the 200 threads are busy. Then a bad deploy makes recommendations hang until the client’s 2 s timeout, and home-feed calls want 300 threads. The other endpoints hold about 6 + 16 = 22, so the pool fills in roughly (200 − 22) ÷ 150 ≈ 1.2 seconds. Then checkout requests queue behind home-feed requests and time out, while payments sits idle and healthy.

The fix keeps the same 200 threads but caps what each dependency may hold: 30 permits for recommendations (with its client timeout cut to 300 ms), 24 for payments, and the existing 20-connection database pool. Even with all three full, 200 − 30 − 24 − 20 = 126 threads stay free. When recommendations hangs, the 31st concurrent home-feed call is refused in microseconds and served a cached “popular near you” list.

Two details make the after-picture work:

1. The compartment rejects instead of waiting. Queuing for a permit just moves the pile-up one step earlier.
2. The rejected request has somewhere to go: a cached list, a hidden widget, or, for checkout, which has no sensible fake payment, a fast and honest error.

Thread pools vs semaphores

Inside one process there are two ways to build a compartment. Netflix’s Hystrix popularised both; it is now in maintenance mode, and Netflix points new projects at Resilience4j, which offers the same pair.

Semaphore isolation is a counter. The request’s own thread takes a permit before calling; if none is free, it is rejected at once. No extra threads, no hand-off. The catch: the caller does its own waiting. If the dependency hangs, the thread holding the permit hangs with it until the network timeout. The semaphore stops more threads joining the pile but cannot rescue those already in it.

Thread-pool isolation runs the call on a small pool dedicated to that dependency. The request thread waits on a future with its own deadline, so it can walk away and serve a fallback. You pay for the hand-off: queueing, scheduling and a context switch. Netflix measured it on one command at 60 requests a second: nothing at the median, about 3 ms at p90 and 9 ms at p99. Their API servers ran more than 40 such pools per instance, mostly 10 threads each, across over 10 billion executions a day.

Both Resilience4j types throw BulkheadFullException when full. Here is the food-delivery service with one of each:

// Semaphore bulkhead: at most 30 concurrent recommendation calls, never wait for a permit
Bulkhead recs = Bulkhead.of("recommendations", BulkheadConfig.custom()
        .maxConcurrentCalls(30)
        .maxWaitDuration(Duration.ZERO)
        .build());

List<Dish> homeFeed(String userId, String cityId) {
    try {
        // recsClient has its own 300 ms connect and read timeouts
        return recs.executeSupplier(() -> recsClient.topPicks(userId));
    } catch (BulkheadFullException full) {
        return popularNearby.get(cityId);   // in-memory cache, no network call
    }
}

// Thread-pool bulkhead: payment calls run on 24 dedicated threads; 4 more may wait
ThreadPoolBulkhead payments = ThreadPoolBulkhead.of("payments", ThreadPoolBulkheadConfig.custom()
        .coreThreadPoolSize(24)
        .maxThreadPoolSize(24)
        .queueCapacity(4)
        .build());

CompletableFuture<Receipt> charge(Order order) {
    return payments.executeSupplier(() -> paymentsClient.charge(order))
            .toCompletableFuture()
            .orTimeout(1500, TimeUnit.MILLISECONDS);   // the caller walks away here
}

Two traps hide in the thread-pool settings. The default queue of 100 is long next to a pool of, say, 10 threads: at 300 ms per call, the 100th task waits about 100 ÷ 10 × 0.3 s = 3 s before it starts. And the pool is a Java ThreadPoolExecutor, which prefers queuing to adding threads once the core threads are busy, so threads above the core size start only when the queue is full. Keep the queue short and core equal to max.

Sizing a compartment with Little’s Law

Little’s Law gives the starting number. Hystrix’s documented rule says the same in words: peak requests per second when healthy, times p99 latency in seconds, plus some breathing room. For recommendations:

• Peak rate per pod: 150 req/s. Healthy p99: 0.12 s.
• Permits needed: 150 × 0.12 = 18.
• Headroom for bursts: 18 × 1.5 = 27, rounded up to 30.

Use p99, not the mean, so a normal day sees no rejections. Then check the failure case. With a 300 ms client timeout, a call holds its permit for at most 0.3 s, so at most 30 ÷ 0.3 = 100 req/s are admitted and the other 150 − 100 = 50 req/s get the cached list straight away. Recommendations never holds more than 30 threads.

So you want a timeout and a bulkhead. The timeout bounds latency (W); the bulkhead bounds in-flight work (L), whatever the arrival rate does. With only the 300 ms timeout, a cricket-final spike that triples home-feed traffic to 450 req/s would want 450 × 0.3 = 135 threads for one widget, and retries could multiply that again. The bulkhead holds at 30.

Re-derive the numbers when traffic or latency shifts, and err small: Netflix kept more than 30 of its pools at 10 threads, treating the pool as its main load-shedding tool.

When a compartment is full

You have three moves, chosen per compartment in advance:

Reject fast. Return an error or a fallback at once: BulkheadFullException, an HTTP 503 or 429 with Retry-After, a cached answer. This is the default. Rejection costs microseconds, so the requests you accept still finish in time.

Queue, with a short bound. A small queue absorbs millisecond bursts. For steady traffic the Google SRE book suggests queues of half the thread count or less, and shows why: with a queue ten times the thread count and 100 ms of work per request, the last request in a full queue takes about 1.1 s, nearly all of it waiting. A queue longer than the caller’s timeout serves answers nobody is waiting for.

Shed the lowest priority first. If a compartment serves mixed traffic, drop the least important work first (next section).

Count rejections per compartment and alert when they persist, but keep them out of success-latency percentiles: as the Amazon Builders’ Library notes, fast rejections make the median look wonderful while real requests suffer. And stop callers retrying a rejection instantly. See also backpressure and load shedding.

Compartments by priority and tenant

So far each compartment guards a dependency. You can also split by who is asking: a critical queue for checkout and payments, a standard one for browsing, and a best-effort one for reports, crawlers and exports, each with its own workers.

This is the bulkhead for floods rather than slow dependencies: a big tenant’s export, a buggy client in a loop, a crawler at peak hour. Real systems do it at scale:

• Google’s RPC stack tags each request with one of four criticalities (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE), propagates it downstream, and under overload rejects lower ones first (SRE book, Handling Overload).
• The Kubernetes API server, through API Priority and Fairness (APF), splits its concurrency limit (by default --max-requests-inflight 400 plus --max-mutating-requests-inflight 200, so 600 seats) among priority levels, so a pod flooding the API cannot block leader election or the built-in controllers. Within a level, shuffle-sharded fair queuing stops one noisy client starving its peers.
• Envoy keeps separate limits, and separate connection pools, for each routing priority (DEFAULT and HIGH) of every upstream cluster.

The API server adds a twist worth copying: priority levels lend unused seats to busy levels, within bounds, and reclaim them when their own demand returns. That attacks the main cost of bulkheads, idle capacity, at the price of a more complex controller.

For tenants, the same split means pools per tier, per-tenant limits in a shared pool, or a dedicated fleet for the largest customer; see noisy neighbour and multi-tenancy and isolation.

The ladder: from threads to regions

Everything so far lives inside one process. The idea works at every level of the stack, and each step down contains a bigger, rarer failure at a higher price.

The figure and the table below cover the ladder; three rungs deserve a note:

• Split connection pools are the rung teams most often miss. Give the nightly report its own small pool, ideally against a replica, so it cannot take checkout’s connections; the Azure guidance uses one pool per downstream service as its main example. See connection pooling.
• Dedicated instance pools give checkout its own deployment and machines, so a crash loop or bad release in browse cannot reach checkout’s hosts.
• Cells and shuffle sharding apply the idea to whole stacks. AWS’s Well-Architected guidance gives the arithmetic: with 10 cells, a failure in one leaves 90% of requests untouched. Shuffle sharding gives each customer a random pair of workers: 8 workers make 8 × 7 ÷ 2 = 28 pairs, so a customer that takes down its pair fully affects about 1/28 of customers, against 1/4 with plain 4-way sharding. Details in cell-based architecture; regions are covered in multi-region and active-active.

Bulkheads you configure: Envoy and Kubernetes

The Azure guidance makes a useful point: prefer the isolation your platform already has over rebuilding it in application code.

Envoy’s per-cluster limits

Envoy calls these “circuit breakers”, but they are concurrency limits: bulkheads at the proxy. Each upstream cluster gets four:

• max_connections to all hosts in the cluster (default 1024).
• max_pending_requests waiting for a free connection (default 1024).
• max_requests outstanding at once (default 1024).
• max_retries outstanding at once (default 3). Envoy recommends a retry budget instead, by default 20% of active plus pending requests.

On overflow, Envoy fails the request at once, bumps a counter such as upstream_rq_pending_overflow, and sets the x-envoy-overloaded header on HTTP responses. Limits are per cluster and per priority, and the application code does not change.

# One order-api sidecar's view of the payments cluster (excerpt)
- name: payments
  connect_timeout: 0.25s
  circuit_breakers:
    thresholds:
    - priority: DEFAULT        # refunds, reconciliation, admin tools
      max_connections: 10
      max_pending_requests: 4
      max_requests: 10
      max_retries: 1
    - priority: HIGH           # routes marked high priority, such as checkout
      max_connections: 24
      max_pending_requests: 4
      max_requests: 24
      max_retries: 2

Two facts shape the sizing. Envoy enforces limits per proxy, with no coordination: 50 calling pods with max_requests: 24 can put 50 × 24 = 1,200 requests on payments, so size each proxy’s limit as what the upstream can take divided by the number of callers. And the 1024 defaults exceed what most services can absorb, so an untuned cluster effectively has no bulkhead.

Kubernetes requests, limits and quotas

At the container level, the Linux kernel partitions through cgroups:

• Requests are what the scheduler reserves: a pod lands only on a node with that much unreserved CPU and memory. Under contention, CPU time is weighted by requests.
• Limits are ceilings. A container at its CPU limit is throttled; one over its memory limit may be OOM-killed. Either way the damage stays in that container.
• ResourceQuota caps the total requests, limits and object counts of a namespace; anything that would exceed it is rejected with 403 Forbidden.
• LimitRange sets per-container defaults and maximums, and QoS classes set eviction order under node pressure: BestEffort, then Burstable, then Guaranteed.

A quota turns the batch team’s namespace into a compartment that cannot crowd production off a shared cluster:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: batch-ceiling
  namespace: batch-jobs
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 160Gi
    limits.cpu: "60"
    limits.memory: 200Gi
    pods: "200"

A cgroup partitions CPU and memory, not everything: pods on one node still share its network interface, disk bandwidth and kernel. For a critical API, add a dedicated node pool, one rung further down the ladder.

What bulkheads cost

Stranded capacity. A shared pool is sized for the peak of combined demand; compartments for the sum of each one’s peak. Say three dependencies each peak at 40 slots, but never together, and combined demand never exceeds 70. A shared pool of 70 covers it; three compartments need 3 × 40 = 120, about 70% more, mostly idle. At fleet level it means a minimum fleet per pool.

More tuning. Amazon’s experience with a single “maximum connections” setting is a fair warning: too low, it turned away traffic the service could handle; too high, servers slowed to a crawl; just right, the workload shifted and it was wrong again. Bulkheads multiply that by the number of compartments.

More to watch. Each compartment needs utilisation, rejection and latency metrics, and an owner. A bulkhead quietly rejecting 5% of requests is an outage nobody has noticed.

Three things keep the bill down: lending idle capacity within bounds, as the Kubernetes API server does; adaptive limits computed from measured latency, the direction Netflix took after Hystrix; and few, coarse compartments, isolating only the risky or critical dependencies.

When a shared pool is fine. There is one dependency, or several equally critical ones, so the request fails anyway if any fails. Or the service is small, with tight timeouts and steady traffic. Or it uses non-blocking I/O, where a waiting request costs kilobytes rather than a thread, though you should still cap in-flight calls per dependency. The Azure guidance gives the same two reasons to skip the pattern: you cannot accept the less efficient use of resources, or the complexity is not needed.

Going deeper: how bulkheads fail

No timeout inside the compartment. If calls can hang forever, the compartment fills once and stays full: the failure is contained, but that feature is down until a restart. Pair every bulkhead with timeouts the client actually honours.

The real shared resource is elsewhere. Thread pools that share one database pool, one HTTP connection pool, one CPU or one heap are not isolated. That was much of Hystrix’s motivation: client libraries are black boxes with their own pools, retries and caches. Find what a stuck request actually holds (thread, connection, memory, file descriptor) and partition that.

Limits multiply with replicas. If autoscaling doubles the callers in a spike, the concurrency allowed onto the downstream doubles too. The downstream needs its own per-client limits.

The rejection path is costly or untested. A fallback that makes a network call can fail the same way; Hystrix advises fallbacks with no network dependency. And code that runs only in incidents rots, so the SRE book suggests regularly running a few servers near overload.

Shared layers above the compartments. Cell routers, load balancers and deploy pipelines stay shared; AWS’s cell guidance says keep the router thin and deploy cell by cell.

Bulkheads and their neighbours

These patterns are easy to blur together in an interview. Each bounds a different number:

They compose: a rate limiter at the edge, a per-dependency bulkhead with a timeout inside it, and a circuit breaker around the call so a dependency that keeps failing is not even attempted. The bulkhead covers what the breaker misses: the seconds before it trips, and slowness under the timeout that an error-counting breaker never sees. Retries go inside, with backoff and a budget; see timeouts, retries and backoff with jitter and rate limiting.

Cheat sheet

How to say it in the interview

“Every request that calls a dependency holds a thread while it waits, so by Little’s Law the threads in use are rate times latency. If recommendations goes from 120 milliseconds to a 2-second timeout at 150 requests a second, it wants 300 threads, and a shared 200-thread pool is gone in about a second, taking checkout with it. So each downstream gets its own bulkhead, sized at peak rate times p99 plus headroom: about 30 permits for recommendations, with a short timeout inside. When it’s full we reject in microseconds and show a cached list. Checkout also gets its own deployment and high priority in the mesh. The cost is idle capacity and more limits to tune, so I’d alert on rejections per compartment.”

How is a bulkhead different from a circuit breaker? A bulkhead caps how many concurrent calls a dependency can hold and is always on; it limits damage while the dependency is slow. A circuit breaker counts failures and, past a threshold, stops calling for a while. The bulkhead covers the seconds before the breaker trips and slowness that never becomes an error.

How do you size one? Peak rate × p99 latency, plus headroom: 150 × 0.12 = 18, so 30 permits. Then check the slow case: with a 300 ms timeout, 30 permits admit 30 ÷ 0.3 = 100 req/s and the rest get the fallback. Confirm with a load test.

Thread pool or semaphore? A semaphore for in-memory or non-blocking calls where overhead matters; a thread pool for blocking clients you don’t trust, because the caller can leave at its deadline. Either way the client needs real timeouts, since a pool thread cannot be killed.

What about a noisy tenant? Bulkhead by tenant or tier: per-tenant limits with fair queuing in a shared pool, a separate fleet for the biggest tenants, and at scale cells or shuffle sharding.

Doesn’t this waste capacity? Yes: 120 slots against 70 in the example above. Isolate only the risky or critical paths, let idle compartments lend within bounds, or use adaptive limits.

Where to go next

• Circuit breaker: stop calling a dependency that keeps failing.
• Timeouts, retries and backoff with jitter: the timeout that bounds how long a permit is held.
• Cell-based architecture: deployment stamps and geodes: bulkheads the size of whole stacks, and shuffle sharding.
• Noisy neighbour: the tenant-flood version of this problem.
• Connection pooling: sizing and splitting pools in front of a database.
• Backpressure and load shedding: what to do with rejected work.
• Multi-tenancy and isolation: keeping tenants apart.
• Retry storm: why rejected requests must not come straight back.

References and further reading

1. Bulkhead pattern (Azure Architecture Center)learn.microsoft.comThe pattern’s definition, consumer-side and service-side partitioning, per-service connection pools, and when the lower resource efficiency is not worth it.
2. Resilience4j documentation: Bulkheadresilience4j.readme.ioThe semaphore and fixed thread-pool bulkhead implementations, their configuration properties and default values.
3. Hystrix wiki: How it Works (isolation, threads and semaphores)github.comNetflix’s case for per-dependency thread pools, the measured cost of thread isolation, and why semaphores cannot walk away from a latent call.
4. Hystrix wiki: Configurationgithub.comThe thread-pool sizing rule (peak rate × p99 latency + breathing room), default pool and semaphore sizes, and Netflix’s production pool sizes.
5. Envoy documentation: Circuit breakingenvoyproxy.ioPer-cluster, per-priority limits on connections, pending requests, requests and retries, their overflow counters and uncoordinated enforcement.
6. Kubernetes documentation: Resource Management for Pods and Containerskubernetes.ioHow requests drive scheduling and CPU weighting, and how cgroups enforce CPU limits by throttling and memory limits by OOM kills.
7. Kubernetes documentation: Resource Quotaskubernetes.ioPer-namespace caps on total requests, limits and object counts, enforced at admission with 403 Forbidden.
8. Kubernetes documentation: API Priority and Fairnesskubernetes.ioA production bulkhead: the API server’s concurrency split into priority levels that can lend and borrow seats, with shuffle-sharded fair queuing inside each level.
9. Workload isolation using shuffle-sharding (Amazon Builders’ Library)aws.amazon.comSharding versus shuffle sharding with eight workers, the 28-combination arithmetic, and Route 53’s use of the technique.
10. Addressing Cascading Failures (Google SRE book)sre.googleHow overload and resource exhaustion cascade, and the guidance on short queues relative to thread pools and early, cheap rejection.