Circuit breaker

A circuit breaker is a small state machine wrapped around calls to one dependency. While calls succeed it stays closed. When too many recent calls fail or crawl, it opens and rejects new calls instantly for a cool-off period, then goes half-open and lets a few probes test whether the dependency has recovered. It turns a slow, resource-eating failure into a fast, honest one, protecting both your service and the struggling dependency.

Picture it first: the kitchen that stops taking orders

It is the last over of a cricket final and a cloud kitchen’s only tandoor has died. The delivery app keeps sending orders, and riders keep arriving to wait for food that will not come. Twenty minutes later a dozen riders are stuck there, and orders from other restaurants have nobody to carry them.

A sensible app notices that most of this kitchen’s recent orders were cancelled for delays and marks it “not accepting orders”. Customers see that in a split second instead of after a 40-minute wait, riders go where they are useful, and the kitchen fixes its oven in peace. Later, the app lets one or two test orders through. If they come out on time, the kitchen reopens; if not, it stays closed a while longer.

Map it back:

• The kitchen is a dependency: a service, a database, a third-party API.
• The riders are your caller’s scarce resources: threads, connections, memory.
• “Not accepting orders” is the open state: fail fast, without making the call.
• The test orders are half-open probes.
• “Most recent orders cancelled” is a failure rate over a sliding window.

The name comes from the breaker in your fuse box. Michael Nygard popularised the software version in his book Release It!, and Martin Fowler’s short write-up made it a standard pattern.

Why a slow dependency is worse than a dead one

A dependency that is down usually fails fast: connection refused, an instant error. A slow dependency is far more dangerous, because every call to it holds one of your resources while it waits.

Little’s law makes this concrete: calls in flight = arrival rate × time per call. Take a checkout instance that serves 500 requests a second, each calling a recommendations service, with 200 worker threads.

• Normal day. Recommendations answers in about 20 ms, so 500 × 0.02 s = 10 threads are busy with it.
• Bad day. Every call waits for the 2-second timeout. You need 500 × 2 s = 1,000 threads and have 200.
• How fast it breaks. The pool fills in 200 ÷ 500 = 0.4 seconds.

Once the pool is full, cart and payment requests that never touch recommendations queue behind the stuck threads and fail too. Hystrix’s documentation warns that under high traffic one slow dependency can saturate every server within seconds.

The breaker attacks both halves of the problem:

• It protects the caller. A rejected call costs microseconds and holds no thread, so the rest of the service keeps working.
• It protects the dependency. An overloaded service that keeps receiving full traffic, plus retries, may never recover. Cutting its load to a trickle gives it room to restart, warm caches or drain queues.

Three states and the moves between them

A breaker is a state machine with three states.

Closed is normal. Calls pass through and the breaker records each outcome (success, failure, slow) in a sliding window. If the failure rate crosses the threshold, it trips to open.

Open is the protective state. Every call is rejected at once with a distinct error, without touching the network: Resilience4j throws CallNotPermittedException, Polly a BrokenCircuitException. A timer starts. Nothing is measured while open, because nothing is sent.

Half-open is the experiment. When the timer expires, the breaker lets a fixed number of probe calls through and rejects everything else until they finish. Hystrix allowed one probe; Resilience4j allows 10 by default; Azure’s guidance requires a number of consecutive successes. If the probes succeed, the breaker closes and its counts reset. If they fail, it reopens and the timer restarts.

Two details matter in practice:

• The open timer can grow. Azure suggests seconds for the first opening and minutes if the fault persists; Polly accepts a function that computes each break duration, and Envoy multiplies a host’s ejection time by its consecutive ejections. Growth stops a breaker probing every few seconds through an hour-long outage.
• Operators need overrides. Fowler says operations staff should be able to trip or reset breakers. Resilience4j adds FORCED_OPEN, DISABLED and METRICS_ONLY states, and Polly can isolate a circuit by hand. Forced open doubles as a kill switch.

When should it trip?

Consecutive failures

The simplest rule: open after N failures in a row, and let any success reset the count. Fowler’s example uses 5, as does Envoy’s per-host outlier detection by default.

It is cheap and fine for low-traffic callers, but one success wipes the slate. Round-robin between two replicas, one of them dead, and outcomes alternate fail, success, fail: a 50% failure rate, yet the count never passes 1.

Failure rate over a sliding window

Most libraries instead compute a failure rate over a sliding window, with three knobs:

• Window: the last N calls (count-based) or the last N seconds (time-based).
• Threshold: the failure percentage that trips the breaker.
• Minimum number of calls: no verdict until the window holds this many. Without it, 2 failures in 3 calls at 3 a.m. is a 67% failure rate.

Well-known libraries disagree on defaults, a hint that there is no universal answer:

A count-based window covers however long your traffic takes to make N calls, so the same setting behaves very differently on busy and quiet callers.

Slow calls are failures too

A dependency can return 200s and still hurt you if each takes 3 seconds. Resilience4j therefore tracks a slow-call rate: calls longer than slowCallDurationThreshold count as slow, and the breaker opens when their share reaches slowCallRateThreshold, cutting load before the dependency becomes unresponsive. The defaults (100% of calls slower than 60 seconds) effectively switch this off; start with a duration near the normal p99.9 and a rate of 50–80%.

Which errors count

Count only outcomes that say something about the dependency’s health:

• Count: timeouts, connection refused or reset, 5xx responses (especially 502, 503, 504), and rejections by your own bulkhead. Hystrix counted timeouts and thread-pool rejections too.
• Count, and consider tripping at once: 429 or 503 with Retry-After. Azure’s guidance notes that an explicit “overloaded” reply can justify opening immediately for a minimum time.
• Don’t count: 4xx validation errors (400, 404, 409, 422), auth failures from your own expired credentials, business errors such as “insufficient balance”, and calls your own caller cancelled. These prove the dependency is up.

Defaults are blunt: Resilience4j counts every exception until you configure recordExceptions and ignoreExceptions, and Polly every exception except OperationCanceledException.

Here is the whole mechanism as a single-threaded, count-based Python sketch:

import time
from collections import deque

class BreakerOpen(Exception):
    """Raised at once while the breaker is open. Never retry it."""

class DependencyFailure(Exception):
    """Timeouts, connection errors and 5xx: the outcomes that count."""

class CircuitBreaker:
    def __init__(self, window=100, min_calls=20, max_failure_rate=0.5,
                 open_seconds=30.0, probes=3, clock=time.monotonic):
        self.results = deque(maxlen=window)          # count-based window
        self.min_calls, self.max_failure_rate = min_calls, max_failure_rate
        self.open_seconds, self.probes, self.clock = open_seconds, probes, clock
        self.state, self.opened_at = "closed", 0.0

    def call(self, fn, *args):
        if self.state == "open":
            if self.clock() - self.opened_at < self.open_seconds:
                raise BreakerOpen()                  # fail fast: no network call
            self.state, self.slots, self.passed = "half_open", self.probes, 0
        if self.state == "half_open":
            if self.slots == 0:
                raise BreakerOpen()                  # probes already in flight
            self.slots -= 1
        try:
            result = fn(*args)
        except DependencyFailure:
            self._record(ok=False)
            raise
        except Exception:
            self._record(ok=True)                    # a 404 proves it is up
            raise
        self._record(ok=True)
        return result

    def _record(self, ok):
        if self.state == "half_open":
            if not ok:
                self._trip()                         # one bad probe: reopen
            else:
                self.passed += 1
                if self.passed == self.probes:       # every probe passed
                    self.state = "closed"
                    self.results.clear()
        elif self.state == "closed":
            self.results.append(ok)
            failures = self.results.count(False)
            if (len(self.results) >= self.min_calls
                    and failures / len(self.results) >= self.max_failure_rate):
                self._trip()

    def _trip(self):
        self.state, self.opened_at = "open", self.clock()
        self.results.clear()

A production library adds atomic state updates, time-based windows, slow-call tracking, events on every transition, and running totals so a check costs O(1). The injectable clock lets you test transitions without sleeping.

When it is open: fail fast, then fall back

Opening decides when to stop calling. You still decide what the caller gets instead.

Fail fast with a clear error. Raise a distinct error type, or, as a server passing the failure upstream, return 503 with Retry-After set to the remaining open time (Polly’s exception carries a RetryAfter value). Retries must recognise it and stop, and dashboards must separate “short-circuited” from “dependency failed”.

Then, if you can, fall back. Fowler’s examples are queueing a card authorisation for later and showing stale data that is good enough.

Fallbacks have a dark side. The Amazon Builders’ Library explains why Amazon almost never uses them: fallback code runs rarely, so it is hard to test, can fail itself, and often widens an outage. Its example is from around 2001: web servers read shipping estimates through a local cache process and, if it failed, queried the supply-chain database directly. Months later the caches failed together, every web server hit the database, and it locked up, taking down the whole website and fulfilment centres worldwide.

Two rules follow:

• A fallback must be cheaper and more reliable than the primary path. Local memory and static defaults qualify; a remote system that normally carries none of this load does not.
• Exercise it continuously. AWS suggests turning fallback into failover by using both paths routinely.

One breaker per what?

Each breaker should cover one independent failure domain: a set of calls that tend to fail together.

• Per dependency is the minimum. Payments failing must never open the breaker for recommendations.
• Per operation when endpoints behave differently: a slow /search should not block GET /items/{id}.
• Per shard or host when the dependency is partitioned. Azure’s guidance warns that one shard can fail while the others are fine.

With one shard of four down, a single breaker sees a 25% failure rate. At a 50% threshold it never trips, so every shard-B call still waits for its timeout; at Polly’s default of 10% it trips and blocks three healthy shards. Per-shard breakers see 100% on B and 0% elsewhere.

Finer is not free: each breaker needs traffic to fill its window. Spread 1,000 calls/s over 64 shards and each breaker sees about 16 calls/s, plenty. Spread 20 calls/s over 64 and each sees about 3 calls per 10-second window, so it never reaches a minimum of 20.

Local or shared state?

Almost every library keeps breaker state in each caller instance’s memory, and Envoy calls its limits “fully distributed (not coordinated)”. That is usually right:

• A broad failure looks the same everywhere. If 40 instances each send 100 calls/s to a dead service, all of them trip on their own within a second.
• Local views catch local faults. If only one zone’s network path is broken, only that zone’s instances should open.
• Shared state is another dependency. A breaker that reads Redis adds a network hop to every call and a new failure mode.

Share state only for callers too quiet to fill a window alone; push “force open everywhere” through configuration.

Fitting it together with timeouts, retries and bulkheads

A breaker is one layer in a small stack, and order matters. Resilience4j’s Spring integration applies Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( call ) ) ) ) ), a good default to reason from. (Timeouts and retries and bulkheads have their own articles.)

• The timeout is the sensor. A breaker learns only from finished calls; without a timeout a hung call never counts, and the breaker stays closed while your threads drain. Azure warns that long timeouts defeat a breaker. AWS sets timeouts by picking an acceptable false-timeout rate, such as 0.1%, and using the matching latency percentile, p99.9.
• The bulkhead caps damage before the breaker trips. A window takes time to fill. A cap of, say, 20 calls in flight means only 20 threads can be stuck meanwhile. Resilience4j stresses that its sliding window is not a concurrency limit.
• Retry outside the breaker. Each attempt passes through the breaker, so it sees the true failure rate and trips quickly. Once open, the retry gets an instant short-circuit and must give up; Azure says retry logic should stop when the breaker reports a non-transient fault.
• Retry inside the breaker is the trap. The breaker sees one outcome per three attempts and trips late, while the retries triple the load on a struggling service.

Retries also multiply across layers. AWS describes a five-deep call stack with three tries per layer: when the bottom database fails, its load grows 3^5 = 243×. Retry at one layer, cap attempts and keep a budget. Google’s SRE book allows three attempts per request and retries only while retries are under 10% of a client’s requests.

In Python-style pseudo-code (the wrappers are illustrative, not a real library):

fetch = recs_client.get                            # the raw network call
fetch = bulkhead(max_in_flight=20)(fetch)          # innermost: cap concurrency
fetch = timeout(seconds=0.8)(fetch)                # a hung call becomes a failure
fetch = breaker.wrap(fetch)                        # records every attempt
fetch = retry(attempts=3, backoff=jittered(0.1),   # outermost
              retry_on=DependencyFailure)(fetch)   # BreakerOpen is never retried

def recommendations(user_id):
    try:
        return fetch(user_id)
    except (BreakerOpen, DependencyFailure):
        return POPULAR_ITEMS                       # in memory, refreshed hourly

Cousins: adaptive throttling and Envoy

Client-side adaptive throttling

Google’s SRE book describes a softer alternative for clients of overloaded backends. Each client task remembers, over the last two minutes, how many requests the application attempted (requests) and how many the backend accepted (accepts), and rejects each new request locally with probability:

P(reject) = max(0, (requests − K × accepts) / (requests + 1))

The book prefers K = 2. Say the application attempted 1,000 requests and the backend accepted 300: (1,000 − 2 × 300) / 1,001 ≈ 0.40. The client drops 40% locally and sends about 600, which is K × accepts. In a heavy overload the backend ends up rejecting roughly one request for every one it serves, as the book reports.

import random

def reject_locally(requests: int, accepts: int, k: float = 2.0) -> bool:
    """SRE-book adaptive throttling; counts cover the last two minutes."""
    p = max(0.0, (requests - k * accepts) / (requests + 1))
    return random.random() < p

How it differs from a breaker:

• A dial, not a switch. Local rejection rises smoothly with overload instead of jumping from 0% to 100%.
• No half-open phase. Some traffic always flows, so clients notice recovery fast. The book prefers K = 2 over a stricter 1.1 for this reason: a little wasted backend work buys quicker news of recovery.
• Built for overload. It reacts to rejections such as “out of quota”, and works poorly for clients that call a backend only sporadically.

Envoy’s “circuit breaking” is a concurrency limit

In Envoy, and meshes built on it, circuit breaking is not the state machine above. It is a set of caps per upstream cluster on connections, pending requests, active requests and active retries, by default 1,024, 1,024, 1,024 and 3. On overflow Envoy fails the request at once and sets x-envoy-overloaded. That is a bulkhead plus load shedding: it reacts to concurrency, not error rate, with no timer or probes.

The breaker-like part is outlier detection, a passive health check per host. By default a host is ejected from the load-balancing pool after 5 consecutive 5xx or connection failures, for 30 seconds times its consecutive ejections (up to 300 seconds), and at most 10% of a cluster’s hosts can be out at once, a guardrail most libraries lack. Ejection is the open state; returning the host is the probe.

Istio configures both in one DestinationRule. A mesh protects every service without code changes but cannot serve an application fallback; a proxy does not know what “show popular items” means. Many teams use both: the mesh for per-host ejection and caps, a library for per-operation breakers with fallbacks.

Going deeper: how breakers go wrong

Flapping

A breaker that opens and closes every few seconds is worse than none. The causes are a tiny minimum number of calls, a window of a second or two, or a threshold close to the normal error rate. Use at least 20 calls per window, a window of 10–60 seconds, a threshold well above normal (if 2% of calls fail on a good day, don’t trip at 5%), and open periods that grow while the fault persists.

The half-open stampede

Breakers on many instances see the same failure, open in the same second and, with the same fixed wait, go half-open together. With 300 instances each allowing Resilience4j’s default of 10 probes, 300 × 10 = 3,000 requests hit a service that has only just recovered; if they pass, every breaker closes and full traffic returns at once.

Jitter the open period: a 30-second wait randomised by ±20% (24 to 36 seconds) spreads those probes over 12 seconds, about 3,000 ÷ 12 = 250 a second. Envoy’s outlier detection has a max_ejection_time_jitter setting for exactly this thundering herd. Keep probes few and, where your stack allows, ramp traffic back gradually after closing.

Breakers that hide outages

An open breaker with a good fallback is silent: pages render, error rates look normal, and a dependency can stay broken for days. Fowler advises logging every state change. Emit per-breaker metrics for state, short-circuited calls and fallback rate, and alert when a breaker stays open for more than a couple of minutes or keeps flipping.

Modal behaviour and slower recovery

The Amazon Builders’ Library is openly sceptical: breakers add modal behaviour (the system runs differently during an incident), which is hard to test, and they slow recovery. With Resilience4j’s 60-second default wait, a dependency that recovers one second after the breaker opened sees no traffic for another 59 seconds. AWS instead limits retries with a local token bucket, built into its SDKs since 2016. Netflix, whose Hystrix made breakers mainstream, shifted to adaptive concurrency limits and put Hystrix into maintenance mode.

The lesson: keep open periods short (5–30 seconds), prefer proportional controls where you can, and test the open mode on purpose.

Testing breakers

• The state machine, with a fake clock: feed it outcomes and assert every transition, including the minimum-calls rule and the probe limit.
• The integration, with fault injection: a proxy that adds latency or returns 503s in staging; check that the breaker opens within one window, the fallback renders and the alert fires.
• Production, deliberately: force a breaker open in a game day. If that switch feels scary to flip, the fallback has never really been tested.

Cheat sheet

How to say it in the interview

“Every remote call goes through a circuit breaker, one per dependency and operation, and per shard where data is partitioned. It records outcomes in a 10-second window; if at least 20 calls were seen and half failed or were slow, counting timeouts, connection errors and 5xx but never 4xx, it opens. Calls then fail in microseconds with a distinct error, threads stay free, and we serve a cheap fallback like cached popular items. After about 30 seconds, jittered, it lets a few probes through: success closes it, failure reopens it for longer. Retries sit outside the breaker and never retry a short-circuit; the timeout and a bulkhead sit inside. A breaker open for more than two minutes pages someone.”

Why not just use timeouts and retries? A timeout bounds one call, but at 500 calls a second and a 2-second timeout you still need 1,000 threads, and retries add load to a failing dependency. Only the breaker stops traffic altogether.

Do you put the retry inside or outside the breaker? Outside. The breaker records every attempt and trips on the real failure rate, and once open, the retry gets an instant short-circuit it must not retry. Inside, the breaker sees one outcome per several attempts and trips late while retries multiply load.

Should breaker state be shared across instances, say in Redis? Usually not. A real outage shows up on every instance within a second, local state catches zone-local faults, and a shared store adds latency and a failure mode to every call. I’d share only for very quiet callers.

How is this different from Envoy’s circuit breaking? Envoy’s “circuit breakers” are per-cluster caps on connections, pending requests, requests and retries, so really bulkheads. The breaker-like part is outlier detection, which ejects a host after consecutive 5xx for a growing time, with at most 10% of hosts ejected by default. Neither can serve an application fallback.

What happens when hundreds of instances go half-open at once? 300 instances × 10 probes is 3,000 requests on a service that has just recovered, followed by full traffic when they all close. Jitter the open period, keep probes few and ramp traffic back. Adaptive throttling avoids this entirely because it never stops traffic completely.

Where to go next

• Timeouts, retries and backoff with jitter: deadlines and safe retries, the breaker’s closest partners.
• Bulkheads: stop one failure sinking the ship: concurrency caps that work before a breaker trips.
• Retry storm: what happens when retries are not budgeted and nothing stops them.
• Backpressure and load shedding: the server side, refusing work before overload.
• Backend failure modes: the slow, partial and cascading failures breakers are built for.
• Sidecar, ambassador and service mesh: per-host ejection and concurrency caps without code changes.
• Health checks done right: active health checks, the partner to passive outlier detection.
• Reliability patterns cheat sheet: every resilience pattern on one page.

References and further reading

1. CircuitBreaker (Martin Fowler, bliki)martinfowler.comThe classic description: tripping on a failure threshold, the self-resetting half-open state, monitoring breaker state and letting operators trip or reset it.
2. Circuit Breaker pattern (Azure Architecture Center)learn.microsoft.comThe three states, growing open timers, retries that respect the breaker, per-shard breakers, manual override and the danger of long timeouts.
3. Resilience4j documentation: CircuitBreakerresilience4j.readme.ioCount- and time-based sliding windows, the slow-call rate, minimum number of calls, half-open probes, special states, and the default values quoted here.
4. Hystrix wiki: How it Works (Netflix)github.comError-percentage tripping above a request-volume threshold, a single half-open probe, and counting timeouts and thread-pool rejections; the README records the project’s move to maintenance mode.
5. Polly documentation: Circuit breaker resilience strategypollydocs.orgA second set of library defaults (10% over 30 seconds, 100 calls, 5-second break), dynamic break durations and manual isolation.
6. Envoy documentation: Circuit breakingenvoyproxy.ioEnvoy’s per-cluster limits on connections, pending requests, requests and retries, their 1,024 defaults, and their uncoordinated, per-proxy nature.
7. Envoy documentation: Outlier detectionenvoyproxy.ioPer-host passive health checking: consecutive-5xx ejection, growing ejection times and the cap on ejected hosts (defaults from the outlier detection API reference).
8. Handling Overload (Google SRE book, chapter 21)sre.googleClient-side adaptive throttling and its rejection-probability formula, plus per-request and per-client retry budgets.
9. Avoiding fallback in distributed systems (Amazon Builders’ Library)aws.amazon.comWhy rarely exercised fallbacks are dangerous, the 2001-era cache-fallback outage, and turning fallback into failover.
10. Timeouts, retries, and backoff with jitter (Amazon Builders’ Library)aws.amazon.comChoosing timeouts from downstream percentiles, the 243× multi-layer retry blow-up, and AWS’s caution that breakers add modal behaviour and recovery time.