Reliability patterns are reusable answers to the specific ways distributed systems fail: a dependency hangs, retries pile on, a spike outruns capacity, a deploy goes bad. This page puts them on one sheet in seven families, each with the failure it handles, how it works, the numbers to quote, its cost and the classic mistake, plus a symptom-first guide for choosing. The theory lives in fault tolerance, failover and single point of failure; this is the revision sheet.
Picture it first: a railway’s safety rulebook
A busy railway runs on a rulebook, not on hope, and almost every rule has a software twin:
- A train held too long at a signal gets reported, and control stops sending trains onto a failing section until an inspection clears it. Timeouts and circuit breakers: fail fast.
- Suburban trains have their own tracks, so a derailed goods train doesn’t strand commuters, and each zone has its own control room. Bulkheads and cells: isolation.
- On festival rush days the station stops selling platform tickets and admits passengers in batches from a waiting hall. Load shedding and queue leveling: overload protection.
- A spare engine waits at the junction and a relief crew can take over. Redundancy and failover.
- Every booking carries a PNR, so re-entering it after a dropped line doesn’t sell the berth twice. Idempotency: data safety.
- A new timetable runs on one division first. Canary and rollback: change safety.
- Fire and derailment drills happen when nothing is wrong. Chaos engineering and game days: verification.
Trains are requests, tracks are threads and connections, junctions are dependencies. And the rules interact: a train held too long at one signal blocks every train behind it.
Where each pattern acts
Most patterns live at a specific point on the request path.
Fail fast: bound every wait
Most outages start as slowness. A caller waiting on a slow dependency holds a thread, a connection and memory, and enough waiting callers exhaust them. These patterns turn “slow” into a quick, clean failure. Deep dives: timeouts, retries and backoff, circuit breaker, idempotency.
| Pattern | Failure it handles | How it works, key numbers | You pay | Classic mistake |
|---|---|---|---|---|
| Timeout | A dependency that hangs or crawls, pinning your threads | Connect and request timeouts on every remote call. Accept, say, 0.1% false timeouts and use the callee’s p99.9; pad it if p99.9 is near p50 | A few false timeouts | None at all (some defaults wait forever), or one longer than your caller waits |
| Deadline propagation | Work for a caller that already gave up | One deadline at the edge; pass the time left: 30 s, then 23 s after 7 s of work, then 19 s after 4 more. Drop expired work | A header on every call | Each hop inventing a fixed timeout |
| Retry with backoff and full jitter | Transient faults: a dropped packet, a restarting node, a brief throttle | sleep = random(0, min(cap, base × 2^attempt)); at most two retries, of timeouts, 5xx and throttles, never validation errors |
Latency; extra load during failures | Instant retries in a loop, or no jitter, so clients return in lockstep |
| Retry budget | Retries turning overload into meltdown | At most 3 attempts per request and retries at most 10% of a client’s requests: Google’s worst case falls from nearly 3× load to about 1.1× | Some recoverable errors reach users | A per-request cap alone, which still nearly triples load |
| Circuit breaker | A dependency down longer than retries can wait out | Count failures in a rolling window; past a threshold, fail instantly for a cool-down, then let trial calls decide | Modal behaviour, hard to test | One breaker for many independent back-ends, so one bad shard blocks all |
| Idempotency key | Duplicate side effects when a timed-out call actually succeeded | The client makes one key per operation and resends it on every retry; the server stores key → result and replays it | A key store with a TTL | A new key per attempt |
Three numbers to remember. Retries multiply across layers: Amazon’s example of three tries at each of five layers is 3⁵ = 243× the load on the database, so retry at one layer. Backoff needs jitter: in Marc Brooker’s simulation of 100 clients contending for one row, adding full jitter more than halved the calls made. And budgets can be built in: AWS added token-bucket retry limiting to its SDKs in 2016, so retries flow while tokens last and slow to a trickle after.
The circuit breaker is the pattern you’ll be asked to draw. Its failure counter is time-based, so scattered errors never trip it; only a burst inside the window does.
In the interview
Name the failure before the pattern: “a slow payment provider would pin our threads, so it gets its own pool, a 2-second timeout and a breaker.” Patterns without the failure they handle sound like a buzzword list.
Isolation: keep a failure small
Fail-fast patterns limit how long a failure holds you; isolation limits how many users it reaches. Deep dives: bulkheads, cells, rate limiting, noisy neighbours.
| Pattern | Failure it handles | How it works, key numbers | You pay | Classic mistake |
|---|---|---|---|---|
| Bulkhead | One slow dependency or tenant draining every thread | A bounded pool per dependency or tenant class. Little’s law, threads ≈ rate × latency: 100 calls/s at 100 ms needs 10 threads; at 10 s, 1,000 | Idle slack per pool | One shared pool, or pools too big to bound anything |
| Per-client quota | A runaway client or batch job starving everyone | A token bucket per API key at the edge; reject with 429 and Retry-After. Quotas may oversubscribe capacity, since clients rarely peak together |
Legitimate bursts throttled | Only a global limit, which one client can use up |
| Critical vs best-effort pools | Batch jobs or crawlers drowning user traffic | Batch goes through its own fleet or a proxy that acts as a fuse; cap any one client’s share of threads (Google’s example: 25%) | Duplicate capacity | One fleet, “prioritised” only in a queue |
| Cells | A bad deploy, poison request or overload hitting every customer | Independent copies of the stack, each serving a slice of customers; a failure reaches 1/N | A router; N deployments | A shared router that does real work |
| Shuffle sharding | One poison customer taking down a whole shard | Each customer gets a random pair from 8 workers: 28 combinations, so a poison customer fully overlaps about 1/28 of the others | Clients retry across their set | Customers sharing identical sets |
The single point of failure page works through the cell and shuffle-sharding arithmetic. Order of cost: pools are nearly free, quotas are cheap, cells are an architecture decision.
Overload protection: say no early
Past some load, a server does less useful work, not more: latency passes the client’s timeout, the client retries, and the server spends its CPU on answers nobody reads. Amazon calls the useful part goodput: requests answered correctly and in time. These patterns protect it. Deep dives: backpressure and load shedding, message queues, autoscaling.
| Pattern | Failure it handles | How it works, key numbers | You pay | Classic mistake |
|---|---|---|---|---|
| Load shedding | More requests than capacity, so every request gets slow | Reject excess cheaply (503 or 429) when in-flight count, CPU or queue time crosses a limit. Google’s target: keep serving the provisioned rate at 2× or even 10× it | Some users get fast errors | Rejecting after expensive work, or shedding the load balancer’s health pings |
| Admission by criticality | Overload hurting checkout as much as recommendations | Tag requests at the edge (Google: CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE), propagate the tag, shed the lowest first; provision for the two critical classes |
Classification work | Everything marked critical, or the tag lost downstream |
| Client-side adaptive throttling | A backend burning CPU just to say no | Each client counts requests and accepts over two minutes; past K × accepts (K = 2), it drops new requests locally with rising probability | Slower to notice recovery | Clients hammering a backend that already says “out of quota” |
| Short queues, request TTL, LIFO | Requests served after their caller gave up | Queues at most about 50% of the thread pool for steady traffic; drop expired requests at dequeue; under overload, serve newest first or use CoDel | Old requests go unanswered | Big queues “for safety”: 10 per thread at 100 ms each is a 1.1 s wait |
| Queue-based load leveling | Spiky async work flooding a slow consumer | Producers enqueue; consumers drain at a safe, steady rate, sized for the average, not the peak | Latency; a broker | Average arrivals above the drain rate |
| Backpressure | A fast producer burying a slow consumer | Bounded buffers that block or reject upstream; tighten the producer’s throttle as the backlog grows | Upstream must slow down | Unbounded in-memory buffers |
| Adaptive concurrency limit | A fixed connection limit that goes stale as the workload shifts | Raise the in-flight limit while latency is flat, cut it when latency climbs, like TCP congestion control | Noisy signals | A static limit from one old load test |
| Autoscaling | Sustained growth in load | Add capacity on CPU or queue depth; keep zone-loss headroom; pre-scale for known peaks | Minutes to react | Shedding at the CPU level the autoscaler watches, hiding the signal |
Queues are bimodal: fast while empty, slow to recover once backed up. Amazon’s arithmetic: after a one-hour outage, a queue needs double its normal capacity for another hour to catch up, and a 30-minute spike at 10× the consumers’ capacity takes 30 × 10 = 300 minutes to drain. Alarm on the age of the oldest message, not just depth.
One measurement trap: fast rejections flatter latency graphs. A service shedding 60% of its traffic can show a superb median while successful requests crawl, so track latency for successes separately.
Redundancy and recovery: have a spare, and switch safely
Redundancy gives you a second copy; these patterns decide whether it takes over cleanly. Failover covers detect, decide, promote and redirect; health checks, multi-region and consensus go deeper.
| Pattern | Failure it handles | How it works, key numbers | You pay | Classic mistake |
|---|---|---|---|---|
| N+1 or N+2 across zones | A host or a zone going away | Survivors must carry the peak: two zones need 2× peak, three need 1.5×; N+2 covers a failure during a deploy | Idle capacity | Spares too small to carry the load |
| Failover: active-passive or active-active | Losing a stateful primary, a zone or a region | Passive: a standby replays the log and is promoted, RTO of seconds to minutes. Active: every site serves; writes partitioned or reconciled | An idle standby, or about 2× infrastructure and conflicts | Unrehearsed promotion; no fencing |
| Leases and fencing | Two leaders at once (split brain) | A consensus store grants an expiring lease; each new leader gets a higher fencing token, and storage rejects older ones | A coordinator; election pauses | A lease without fencing: a paused old leader wakes and writes |
| Health checks | Traffic sent to broken instances, or wedged ones left running | Liveness restarts a stuck process; readiness takes a pod out of rotation while warming up or overloaded; startup holds both off during a slow boot. Kubernetes defaults: every 10 s, 1 s timeout, 3 failures | False positives remove capacity | Liveness checks that call the database |
| Static stability | The control plane (scaling, DNS, config) failing mid-outage | Pre-provision; the data plane runs on its last known config and credentials | Daily headroom | Recovery that needs new instances or a config push |
| Graceful degradation | A non-critical dependency failing the whole page | Drop the feature or serve something simpler: cached menu, no ETA | Rarely run code | A degraded path heavier than the normal one |
| Serve stale | An origin failing, or a stampede | Serve the last good value while revalidating or on error (stale-while-revalidate, stale-if-error) |
Old data | No alert on staleness |
| Hedged requests | Tail latency from one slow replica | No answer by the p95? Send a copy to another replica, use the first reply, cancel the other | About 5% more requests; idempotent calls only | Hedging at once (2× load) |
The fallback caveat
Fallbacks look free and often aren’t. Amazon’s Builders’ Library tells of a shipping-speed feature, around 2001, that read through a cache on each web server and fell back to the supply-chain database when the cache was unhealthy. It worked for months. Then the caches failed together, every web server hit the database, the database locked up and the website went down, along with the fulfilment centres that shared the database. A fallback turned a missing feature into a full outage.
Amazon’s alternatives: make the primary path more reliable; let the caller handle the error; push data ahead of time (EC2 instances hold IAM credentials valid for hours, so a push outage changes nothing); or run the fallback continuously, at which point it is failover. Google suggests keeping degraded modes alive by running a few servers near overload. For caches, see the safer options in hot keys and stampedes and distributed caching: serve stale, single-flight, a small gutter pool.
Hedging and fan-out
Tail latency becomes a reliability problem once you fan out. If a server takes a second 1% of the time, a request waiting on 100 such servers is slow with probability 1 − 0.99¹⁰⁰ ≈ 63%. Dean and Barroso’s fix: wait until the 95th-percentile latency, then send a second copy, which caps extra load at about 5%.
Hedging helps only when slowness is uncorrelated, like one replica’s GC pause. If the whole service is overloaded, hedges add load where it hurts, so mark them low priority and cap them with a budget.
Data safety: never lose or double-apply a write
Availability patterns keep answering; these keep the answers right.
| Pattern | Failure it handles | How it works | You pay | Classic mistake |
|---|---|---|---|---|
| Replication with majority commit | Losing acknowledged writes when a node dies | Acknowledge once a majority (2 of 3) across zones has the write | A cross-zone round trip per commit | Async replica plus auto-promotion: RPO above zero |
| Idempotent consumer | At-least-once delivery processing a message twice | Record processed IDs in the same transaction as the side effect | A dedupe table | Dedupe stored apart from the side effect |
| Transactional outbox | The row commits but the event is lost, or the reverse | Write the event to an outbox table in the same transaction; a relay or CDC publishes it | A relay; duplicates downstream | Dual writes with a crash in between |
| Saga | A multi-service workflow failing halfway | Local transactions in sequence; on failure, compensate (refund, release the seat) | Visible intermediate states | Non-idempotent compensations |
| Dead-letter queue | A poison message retried forever | Park it after N attempts; alarm on depth; redrive after the fix | A redrive process | A DLQ nobody watches |
| Checksums end to end | Silent corruption | Compute where data is created, verify where it is used, repair from a copy | CPU, a few bytes | Trusting per-hop checks |
| Backups with tested restores | Bugs and deletes that replication copies faithfully | Versioned, point-in-time backups in a separate account; scheduled restore drills | Storage, drill time | A backup never restored |
Change safety: most outages start with a change
Google’s SRE book advises checking recent changes first in a cascading failure, and considering a revert. Config, flags and firewall rules count too. Deep dive: deployment strategies.
| Pattern | Failure it handles | How it works | You pay | Classic mistake |
|---|---|---|---|---|
| Canary and progressive rollout | A bad build reaching everyone | Ship to one host or cell; compare errors and latency with the old version; widen in stages with bake time; roll back automatically | Slower releases | A canary too small or brief to show the bug |
| Feature flags and kill switches | A bad feature, or a costly one mid-incident | Deploy dark, release by flag, switch off without a deploy | Flag debt | No last-known-good cache of flags |
| Blue-green | A risky cutover with a slow rollback | Two environments; switch traffic at the router, and back in seconds | Double capacity briefly | A schema the old version can’t read |
| Staged config pushes | One bad value breaking everything at once | Config as code: review, validate, push one zone or cell at a time | Slower emergency changes | “It’s only config”, pushed everywhere |
Verification: prove it before an outage does
A pattern you have never triggered is a hope. Amazon puts it bluntly: if you haven’t load-tested to the breaking point and far beyond, assume the service fails in the least desirable way.
- Load tests past the breaking point. Goodput should plateau. A cheap trick: hold load steady and remove servers from a test fleet.
- Chaos engineering. Inject real faults (kill instances, add latency, drop packets) on a slice of production and compare with a control group.
- Game days. Rehearse failovers, restores and runbooks with the people who would do them at 3 am.
- Exercise degraded modes. Flip fallbacks and kill switches on a schedule so the paths stay alive.
Which pattern for which symptom
In interviews and incidents alike, you start from a symptom.
| Symptom | Reach for first | Then add | Avoid |
|---|---|---|---|
| One slow dependency drags every endpoint down | Timeout from its p99.9, a bulkhead pool, a circuit breaker | Fallback or serve stale for that feature | Longer timeouts, a bigger shared pool |
| Retries make the outage worse | Backoff with full jitter, retry at one layer, a 10% budget | An “overloaded, don’t retry” error code | More retry attempts |
| The outage outlives its trigger | Cut load hard, cap retries, drop expired queued work | Cheaper error paths, LIFO under overload | Waiting it out |
| Traffic spike (tatkal at 10 am, an IPL final) | Shed by criticality, per-client limits | Queue leveling for async work, pre-scaling | Unbounded queues |
| One tenant takes everyone down | Per-client quotas, a pool per tenant class | Shuffle sharding, cells | One shared queue for all tenants |
| p50 fine, p99 terrible on fan-out | Hedged or tied requests, tight timeouts | Eject slow outliers, smaller fan-out | Hedging an overloaded service |
| Bad deploy or config push | Canary with automatic rollback, staged config | Kill switches, cells | Global pushes |
| Double charges after retries | Idempotency keys | Idempotent consumers, outbox | Never retrying payments at all |
| Writes lost on failover | Majority or synchronous commit, fencing | Outbox for events | Async replica with auto-promotion |
| Pods restart in a loop under load | Local-only liveness checks, startup probes | Readiness for warm-up and overload | Liveness checks that call dependencies |
| A zone goes dark | N+1 across three zones, static stability | Cells | Scaling up after the failure |
| One poison message stalls a queue | Retry cap then dead-letter queue | Alarms on DLQ depth and message age | Infinite redelivery |
Interactions and anti-patterns
Patterns that are safe alone can be dangerous together. Interviewers probe these.
Retries multiply across layers. Google’s example: if the browser, frontend and backend each make up to 4 attempts, one tap becomes 4 × 4 × 4 = 64 queries on an overloaded database. Retry only at the layer just above the failure, and pass an “overloaded, don’t retry” error upward. (Fault tolerance shows how this sustains a metastable failure; retry storm is the anti-pattern page.)
Retries without idempotency. A timeout means “unknown”, not “failed”: the debit may have happened. Retry writes only with an idempotency key, or mark them pending and check status.
Circuit breakers and retries. Put the retry loop outside the breaker and, as Azure’s pattern guide advises, stop retrying when it reports open. Count every attempt, or the breaker trips late. Amazon notes that breakers add hard-to-test modal behaviour and often prefers a local token-bucket retry limit; many systems use both.
Timeouts longer than the caller’s. Inner timeouts plus retries must fit the outer deadline: if the gateway waits 2 s and the service tries the database three times at 1 s each, the third try serves nobody. Long deadlines also exhaust threads. In Google’s example, 1,000 threads serve 1,000 requests a second at 100 ms. If 5% hang against a 100 s deadline, they want 50 × 100 = 5,000 threads, so only 1,000 ÷ (5,000 + 95) ≈ 19.6% of requests get served. A 5% fault becomes an 80.4% error rate.
Health checks that test shared dependencies. Kubernetes warns that bad liveness probes cause cascading failures: if every pod’s check calls the database, one database blip restarts the fleet onto cold caches. Keep liveness local. Readiness may check required dependencies, but if every instance fails at once, the problem is shared and removing them all helps nobody.
Fallbacks and caches never exercised. The 2001 outage is the template. Google’s variant is the capacity cache: if the service can’t carry its load with an empty cache, the cache is a hard dependency and a restart is an outage. Provision for a cold start and warm new capacity slowly.
Protection that hides the problem
Shedding, retries, fallbacks and stale serving all mask failures; that is their job. Without metrics on retry rate, shed rate, breaker state, queue age and staleness, a system can lose its margin silently until a small fault tips it over.
Cheat sheet
| Pattern | Remember it as |
|---|---|
| Timeout | On every remote call; the callee’s p99.9 for about 0.1% false timeouts |
| Deadline propagation | One deadline at the edge; pass the time left; drop expired work |
| Retries | At most 2, capped backoff, full jitter, one layer, idempotent calls |
| Retry budget | 3 attempts per request, retries at most 10% per client: about 1.1× load, not 3× |
| Circuit breaker | Closed → open on failures → half-open trials → closed |
| Idempotency key | One key per operation, reused on every retry |
| Bulkheads | A bounded pool per dependency; threads ≈ rate × latency |
| Quotas, cells, shuffle sharding | Bound one client’s blast radius: a quota, 1/N, about 1/28 |
| Load shedding | Reject early and cheaply so goodput plateaus; shed lowest criticality first |
| Queues | Short, TTL at dequeue, LIFO under overload; alarm on age |
| Queue leveling | Consumers sized for the average; 1 hour down needs 2× for an hour |
| N+1 across zones | Three zones need 1.5× peak, two need 2× |
| Failover | Fenced: leases plus fencing tokens; rehearse it |
| Health checks | Liveness local, readiness for warm-up and overload, startup for slow boots |
| Static stability | Data plane runs on last known state; recovery creates nothing new |
| Degrade and serve stale | Alert on staleness; fallbacks must run regularly |
| Hedged requests | Second copy after the p95: about 5% extra load; 1,800 → 74 ms at p99.9 in Google’s test |
| Data safety | Majority commit, outbox, idempotent consumers, DLQ, restore drills |
| Change safety | Canary plus automatic rollback, flags, blue-green, config pushed in stages |
| Verification | Load-test past the break, chaos on a slice, game days |
How to say it in the interview
“I’d walk the request path and put a pattern where each failure would land. At the edge, per-client rate limits and shedding by criticality, so checkout survives when recommendations don’t. On every outbound call, a timeout from the callee’s p99.9 inside a propagated deadline, a pool per dependency, and a circuit breaker. Retries at one layer only: two at most, backoff with full jitter, a 10% budget, idempotency keys on writes. Async work goes through a queue with idempotent consumers and a dead-letter queue. Stateless tiers are N+1 across three zones; the database commits to a majority and fails over with fencing. Changes ship as canaries with automatic rollback, and game days prove the fallbacks work.”
Circuit breaker or retries? Both, for different timescales. Retries absorb millisecond blips; the breaker stops calls during an outage of seconds or minutes. The retry loop wraps the breaker and stops when it opens.
How do you choose a timeout? Pick the false-timeout rate you can tolerate, say 0.1%, and use the callee’s p99.9, padded if it sits near the median. Then check that attempts × timeout plus backoff fits the caller’s remaining deadline.
When would you not add a fallback? When it is heavier than the primary, or would first run during an outage. Falling back from a cache to the database can overload the database just as the cache fails. Push data ahead, serve stale or degrade, and exercise whatever you keep.
How do you stop one big customer taking everyone down? Per-client quotas at the edge, a separate pool or queue per tenant class, and shuffle sharding or cells, so a poison or noisy customer reaches only the few who share its workers or cell.
Liveness or readiness? Liveness restarts a process that is wedged, so it checks only the process itself. Readiness removes an instance from rotation while it warms up or is overloaded. A liveness check that calls the database turns one database blip into a fleet-wide restart.
Where to go next
- Fault tolerance: fault models, redundancy and metastable failures behind these patterns.
- Failover: detection, promotion, fencing, and RTO and RPO.
- Single point of failure: finding SPOFs, cells, shuffle sharding and static stability.
- Timeouts, retries and backoff with jitter: the fail-fast family in depth.
- Circuit breaker and bulkheads: containment patterns in depth.
- Backpressure and load shedding: overload protection for sync and async paths.
- Retry storm: how retries turn a blip into an outage.
- Deployment strategies: canaries, blue-green and flags.
- Reliability: SLIs, SLOs and error budgets: how to tell if any of this is working.
References and further reading
- Timeouts, retries, and backoff with jitter (Amazon Builders’ Library)aws.amazon.comTimeouts from the callee’s p99.9, retrying at one layer (243× from five layers of three tries), token-bucket retry limits in the AWS SDK, and jitter on every timer.
- Exponential Backoff And Jitter (AWS Architecture Blog, 2015)aws.amazon.comFull, equal and decorrelated jitter compared in a contention simulation; with 100 clients, jitter more than halved the calls made.
- Handling Overload (Google SRE book, chapter 21)sre.googlePer-customer quotas, client-side adaptive throttling, the four criticality levels, per-request (3 attempts) and per-client (10%) retry budgets, and retrying only at the layer above.
- Addressing Cascading Failures (Google SRE book, chapter 22)sre.googleQueue sizing, LIFO and CoDel, deadline propagation, the 4³ = 64 retry example, thread exhaustion from long deadlines, and capacity versus latency caches.
- Using load shedding to avoid overload (Amazon Builders’ Library)aws.amazon.comGoodput versus throughput, testing far past the breaking point, prioritising requests, timeout hints, bounding time in queues, and shedding that blinds autoscaling.
- Avoiding insurmountable queue backlogs (Amazon Builders’ Library)aws.amazon.comBimodal queues, backlog recovery arithmetic, Little’s law for per-workload concurrency, spillover queues, message TTLs and backpressure.
- Avoiding fallback in distributed systems (Amazon Builders’ Library)aws.amazon.comWhy rarely exercised fallbacks fail, the shipping-speed cache outage around 2001, and the alternatives: push data, convert fallback into failover, proactive retry.
- Circuit Breaker pattern (Azure Architecture Center)learn.microsoft.comClosed, open and half-open states, time-based failure counters, success thresholds, and why retry logic must stop when the breaker is open.
- Liveness, Readiness, and Startup Probes (Kubernetes documentation)kubernetes.ioWhat each probe does on failure, the default period, timeout and failure threshold, and the warning that bad liveness probes cause cascading failures.
- The Tail at Scale (Dean and Barroso, Communications of the ACM, 2013)cacm.acm.orgHow fan-out amplifies tail latency; hedged requests after the 95th percentile, the 1,800 ms to 74 ms BigTable result, and tied requests.