Backend failure modes

Backends rarely fail by switching off cleanly. They slow down, keep passing health checks while users get errors, buckle under load shifted from a dead neighbour, stay broken after the trigger has gone, or fall over on one bad request or config push. This page is a field guide: for each failure mode, what it looks like, why it is dangerous, how to detect it, and which defence to reach for.

Picture it first: a toll plaza on a holiday

A highway toll plaza has eight FASTag lanes. On a holiday weekend:

• A lane shuts, barrier down. Drivers simply pick another.
• A reader takes 40 seconds per car, light still green. Drivers keep joining its queue.
• A reader passes the supervisor’s test card but rejects one bank’s tags. Every inspection says the lane is fine.
• One lane closes in the rush. Its cars jam the others, staff close the jammed lanes, and the jam spreads.
• Rejected drivers loop round and rejoin. The jam outlasts the rush that caused it.
• One truck’s damaged tag freezes the reader in every lane it tries.
• The verification certificate expires at midnight, and every lane rejects every car.
• The cash lane opens for the first time in years, and the cashier has no change.

Map it back: lanes are replicas, the test card is a health check, looping drivers are retries, the truck is a query of death, the certificate is a forgotten dependency, and the cash lane an untested fallback. Each needs a different fix, so naming the failure is half the work.

Stop, slow or wrong

Start with what a caller sees. Almost every backend failure takes one of three shapes:

• Fail-stop. The process exits or the port refuses connections. Callers get an error in milliseconds, and redundancy handles it well.
• Fail-slow. It still answers, just late. Gunawi et al. collected 101 fail-slow hardware incidents from 12 institutions, including disks that fell to 100 KB/s because of vibration, CPUs at half speed because of a power shortfall, and network cards collapsing to kilobits per second.
• Fail-wrong. It answers quickly with the wrong thing: corrupt data, a stale entry, an empty 200. Only correctness checks catch it.

Why slow is usually worse than down

A dead dependency costs you a fast error. A slow one costs you a held thread for as long as you are willing to wait, and every new request holds another.

The Google SRE book’s example: a frontend with 1,000 threads serves 1,000 req/s at 100 ms each, so about 1,000 × 0.1 = 100 threads are busy. Then 5% of requests start hanging until a 100-second deadline. Those want 50 × 100 = 5,000 threads, the rest want 950 × 0.1 = 95, and with 1,000 threads only 1,000 ÷ 5,095 ≈ 19.6% of requests get served. That is an 80.4% error rate from a fault touching 5% of requests. Failing fast, it would have been 5%.

Slow also hides: the mean barely moves, and a slow process still passes liveness checks, so nothing replaces it.

Detect: latency percentiles (p99, p99.9) per replica and per dependency, requests in flight, thread-pool and connection-pool use.

Defend: a timeout or propagated deadline on every remote call, a few times normal latency rather than three orders of magnitude above it; bulkheads so one slow dependency holds only its own slots; circuit breakers that count slow calls as failures; outlier ejection; hedged requests for safe reads.

Gray failure: green checks, failing users

A gray failure is broken for its users but looks healthy to the system meant to notice. Huang et al. (Microsoft Research and Azure, HotOS 2017) report that gray failure, not fail-stop, is behind most of the cloud incidents they have seen. They name its trait differential observability: an app sees the system as unhealthy while the system’s own failure detector sees it as healthy. Only the detector can trigger recovery, so nothing happens until a user complains.

Three of their Azure cases:

• The probe takes another path. VMs with a network driver bug could not reach the outside world, but their heartbeats travelled through a host agent over local RPC, so they kept looking healthy.
• Recovery that kills. An undetected storage server under capacity pressure kept receiving writes, crashed, rebooted and repeated until it was removed. Removing it shrank capacity, the remaining servers degraded the same way, and the problem cascaded.
• Redundancy that hurts. A request that fans out to m back ends across n core switches touches a given switch with probability 1 − ((n − 1) ÷ n)^m. With n = 4 and m = 10, that is 1 − 0.75^10 ≈ 94%. A switch silently dropping packets has not failed outright, so nothing routes around it, and it slows nearly every request.

Detect: put caller-side error rate and latency per replica next to each replica’s health status, and alert on the gap. A replica with three times its peers’ error rate is a gray failure until proven otherwise. Probe the real path, through the same pools and disks.

Defend: health checks that exercise what the replica owns, outlier ejection driven by real traffic, and several health signals instead of one heartbeat. The details are in health checks done right.

Overload and cascading failure

The SRE book names overload as the most common cause of cascading failures. Capacity disappears; load does not.

Load moves to the survivors. In the book’s example, one of two clusters fails and the other goes from 1,000 to 1,200 req/s, beyond what it can handle, so its rate of successful requests drops well below 1,000. Load balancers react within seconds, so overload can spread across clusters in a couple of minutes.

Resources run out in turn. As requests slow down, more are in flight, and each one holds threads (until even the health-check handler starves and the host is marked unhealthy), connections, file descriptors and memory. Extra garbage collection eats CPU, which slows requests further: the GC death spiral. And logs fill disks fastest when things go wrong.

Health checks and restarts finish the job. A replica that fails its checks is ejected and its share lands on the rest. Restarted servers take the full load before their caches are warm and fall over again. In the book’s numbers, a service healthy at 10,000 req/s that tipped into a crash cascade at 11,000 will almost certainly not recover at 9,000; if only 10% of servers are healthy, load must fall to about 1,000 req/s.

Detect: saturation, not just utilisation: queue length, pool use, GC time, open file descriptors, free disk, and a healthy-host count that falls while load rises.

Defend: headroom to lose a replica or a zone (the book sizes its example at N + 2); load shedding, such as a cheap 503 once requests in flight pass a limit; short queues; bulkheads; and a cap on how many hosts the load balancer may eject. To escape a crash loop, cut traffic hard (the book suggests as little as 1%), let servers recover, then ramp back up gradually.

Metastable failures

Some outages outlive their cause. Bronson et al. (HotOS 2021), drawing on a decade of hyperscale operations, call these metastable failures: a trigger pushes the system into a bad state, and a sustaining effect, usually work amplification, keeps it there after the trigger has gone. Goodput, the rate of useful work, stays near zero until a strong push such as a big cut in load. The paper’s three states are stable (recovers once the trigger ends), vulnerable (healthy, but a big enough trigger tips it over; many systems run here because it is efficient) and metastable (held down by a feedback loop).

Their retry example: a database answers in under 100 ms below 300 queries per second, and ten times slower above. The app sends 280 req/s and retries once after a 1-second timeout. A 10-second network blip ends, the requests and retries lost during it arrive together, and the database slows. Now every query times out and is sent twice: 280 × 2 = 560 req/s. The blip is over; the outage is not. Recovery needs load below 150 req/s (150 × 2 = 300) or retries below 20 req/s (280 + 20 = 300).

Other sustaining effects from the paper:

• A cold look-aside cache. At a 90% hit rate the app can serve 3,000 req/s, since only one request in ten reaches the database. Lose the cache contents and the database sees ten times its usual load. It slows, the app times out, and because the app is what fills the cache, the cache never refills. Advertised capacity: 3,000 req/s. Hidden capacity, the load the system can recover from unaided: 300.
• Slow error handling. Error paths capture stack traces, do DNS lookups and write to disk, using up the resources that ran out.
• A link imbalance, where a connection pool’s most-recently-used policy kept steering queries onto a congested network link. It defied explanation for more than two years; the fix was one line.

The GC death spiral and swapping are the same family: the system gets less efficient just when it is overloaded.

Detect: the paper’s characteristic metric, one that the trigger knocks out of range and that recovers only when the failure ends: queueing delay, timeout rate, cache hit rate. If it stays bad after the trigger has gone, you are in a loop, not a spike.

Get out: remove the amplification, not the trigger. Cut load hard, disable or budget retries, drop queued work whose deadline has passed, then ramp back slowly so caches refill.

Prevent: retry budgets (see retry storm); retries at lower priority than first attempts; LIFO queues under overload; read-through caches that keep filling after the client gives up; cheap error paths that sample stack traces; and enough headroom that a cold cache stays under your hidden capacity.

Herds, stampedes and cold starts

A thundering herd is many clients doing the same expensive thing at once. Spread over a minute the work is harmless; packed into one second it is an outage.

• Cache stampede. A hot key expires and thousands of misses hit the database together; see hot keys, cache stampede and invalidation.
• Cold start. A restarted process has empty caches and no pooled connections. The SRE book separates a latency cache (you survive without it, slower) from a capacity cache (you cannot carry the load without it), a hard dependency that only shows itself after a restart.
• Reconnect storms. A gateway restarts and every client reconnects. As an illustration, 100,000 clients reconnecting within 1 second means 100,000 TLS handshakes that second; jittered over 30 seconds, it is about 100,000 ÷ 30 ≈ 3,300 per second.
• Synchronised timers. Midnight cron jobs, apps that sync on the hour, tokens that all expire together.

Recovery is a herd too. After Facebook’s October 2021 outage, data centres had dropped by tens of megawatts of power use, and switching everything on at once risked new crashes, so load came back gradually, drawing on years of “storm” drills.

Detect: request spikes that line up with restarts, failovers, cache flushes or clock boundaries. Defend: jitter every timer, retry and reconnect; coalesce identical requests; warm up new instances and ramp their traffic; admission control at the front door.

Query of death and bad pushes

A query of death (the SRE book’s term; a poison pill in a queue) is one input whose contents crash or wedge whatever handles it. One crash is a bug. The outage comes from retries: the load balancer resends the request to the next replica, which crashes too, and restarted replicas receive it again. A queue does the same to consumers, redelivering the poison message after every crash.

A config or rules push is the same failure with a bigger fan-out: one input reaches every replica within seconds.

Case study: Cloudflare, July 2, 2019

From Cloudflare’s own write-up (times in UTC):

• 13:42. A small WAF rule change for detecting cross-site scripting deploys globally. It is in simulate mode, which logs rather than blocks, but simulated rules still execute.
• One regular expression backtracks catastrophically. Its critical part, .*(?:.*=.*), makes the work grow super-linearly with the input: the simplified .*.*=.* takes 23 steps to match x=x and 555 with 20 x’s after the =.
• Every CPU core that handles HTTP and HTTPS traffic worldwide goes to nearly 100%, and visitors get 502 errors.
• 14:02. The team opts for a “global terminate” of the WAF, but Cloudflare’s own Access service is down, so reaching the internal control panel needs a rarely used bypass.
• 14:07. The kill switch runs. By 14:09, 27 minutes after the push, traffic and CPU are normal.

Count the layers. Quicksilver, Cloudflare’s config store, reaches every machine worldwide with a p99 of 2.29 seconds. A protection against excessive regex CPU had been removed by mistake weeks earlier. Tests did not measure CPU. Rules skipped the staged rollout that software releases get. And the recovery path depended on the broken system. Cloudflare’s fixes: restore the protection, profile every rule in tests, switch to a regex engine with run-time guarantees (re2 or Rust’s regex), stage rule rollouts, and make the dashboard reachable without Cloudflare’s own edge.

Detect: crashes or CPU spikes that correlate with a request signature, message ID or config version; restarts on many replicas at once; a filling dead-letter queue.

Defend: never retry a request that crashed the server; quarantine an input after two kills; dead-letter queues after N attempts; canary config like code (deployment strategies); per-request CPU limits; kill switches that work when your own stack is down; and cells to limit how many replicas one input can reach.

Dependencies you forget

Some dependencies appear on no diagram until they fail, usually everywhere at once and with no deploy to roll back.

DNS. On October 4, 2021, a maintenance command meant to assess Facebook’s backbone capacity disconnected all of its data centres; a bug in the audit tool failed to stop it. Facebook’s DNS servers withdraw their BGP announcements when they cannot reach the data centres, so they made themselves unreachable while still running, and internal tools broke with them. Serve last-known-good answers, and keep incident tooling independent of your own DNS.

Certificates. Expiry is a time bomb with a known date: nothing is deployed, then every handshake fails on every replica at once. Automate renewal, inventory internal and partner certificates, and alert weeks ahead.

Clocks. At midnight UTC on January 1, 2017, a leap second made time go back by one second inside Cloudflare’s DNS software. Round trips measured as time.Now().Sub(start) went negative, and the smoothed value reached Go’s rand.Int63n, which panics on negative input; at peak about 0.2% of Cloudflare’s DNS queries were affected. Measure durations with a monotonic clock, and give leases and tokens a margin for skew (clocks, ordering and conflict resolution).

import time

start = time.monotonic()        # never goes backwards
answer = resolve(name)
rtt = time.monotonic() - start  # always >= 0; time.time() can step back

Quotas and limits. API rate limits, instance quotas, file descriptors, ephemeral ports, database connection caps, a signed 32-bit ID that stops at 2,147,483,647. They bite during growth spurts and during recovery, when autoscaling asks for everything at once. Track headroom against each limit and alert well before the ceiling.

Control planes during recovery. The data plane does the daily work; the control plane makes changes. The Amazon Builders’ Library notes that control planes have more moving parts and so are more likely to be impaired, which makes “launch replacements in another zone” a recovery plan that leans on the control plane at the worst moment. Cloudflare’s engineers could not log in to their own control panel; Facebook’s could not reach their data centres remotely. Keep the recovery path’s dependencies few.

Bimodal paths and untested fallbacks

A system is bimodal when it behaves differently under failure: a fallback path, a degraded mode, a retry loop that rarely fires. That second mode runs only during incidents. As the SRE book puts it, the code path you never use is the code path that often doesn’t work.

The Amazon Builders’ Library recalls an outage from around 2001. The retail site read shipping speeds through a cache process on each web server and fell back to querying the supply-chain database when the cache was unhealthy. It worked for months, until the caches failed together, every server hit the database, the database locked up and the whole site went down. Fulfilment centres worldwide, which used the same database, stopped too.

Queues are bimodal as well. A 1-hour consumer outage at 1,000 messages per second leaves 3.6 million queued. At double capacity the net drain is 2,000 − 1,000 = 1,000 per second, so another hour passes before messages are fresh again.

Amazon’s alternatives: make the primary path more reliable; let the caller retry; push data ahead of time (EC2 instances receive IAM credentials proactively, valid for hours); turn fallback into failover by running both paths all the time; and alarm on retry rates.

Detect: fallback and retry rates; the age of the oldest queued message. Defend: paths that run every day, stale or partial answers, or failing the feature fast.

Split brain and zombie leaders

Two nodes can both believe they lead. The leader stalls in a long GC pause, its lease expires, a new leader is elected, and the old one wakes up and keeps writing. Defend with majority quorums, leases that expire before a new election can finish, and fencing tokens (a number that grows each term) that storage checks; alert when two nodes claim leadership. See consensus and leader election and distributed locking.

The field guide

Real incidents usually combine rows: Cloudflare’s 2019 outage was a bad push, an overload and a control-plane dependency at once.

Remember the detection column above all: you cannot defend against a failure you cannot see.

Designing for failure

Static stability. Keep working through a dependency failure without changing anything. The Builders’ Library example: across three Availability Zones, overprovision by 50% so each zone runs at about 66% of its tested capacity; losing a zone then needs no new instances.

Constant work. Do the same work in good times and bad. Route 53’s health checkers always send result sets sized to the maximum (10,000 entries even when 10 checks are configured), and aggregators push a fixed-size status table to the DNS servers every few seconds, so a zone failure changes values, not workload.

Blast radius. Assume any change or input can be bad: cells and shuffle sharding, canaries, zone-by-zone deploys. EC2 staggers its releases so that no two Availability Zones in a region receive the same deployment on the same day.

Test the failure. Load test until components break, then watch them recover. Blackhole a non-critical back end. Inject latency, not just crashes. Run game days, as Facebook’s storm drills take out services, data centres and whole regions.

Cheat sheet

How to say it in the interview

“I sort backend failures into stop, slow and wrong. Crashes are easy: health checks catch them and traffic moves. Slow worries me more, because a slow dependency holds threads, so a few hanging calls can exhaust a pool; every call gets a deadline and every risky dependency a bulkhead. I catch gray failures by comparing what callers see per replica with what health checks say. Overload cascades, so I keep N + 2 headroom, shed early and budget retries, because retries and cold caches turn a blip into a metastable outage. Config rolls out cell by cell like code, and recovery tooling must not depend on what it is fixing.”

What’s worse, slow or down? Slow, usually. Down returns an error in milliseconds; slow holds a thread for the whole timeout. In the SRE book’s example, 5% of requests hanging for 100 seconds want 5,095 threads against 1,000, so 80.4% of all requests fail.

How does one bad replica take out the cluster? If it crashes, its share moves to the others: four replicas of 400 req/s carrying 1,200 have zero headroom after losing one. If it is slow but looks healthy, it keeps receiving 1/N of the traffic and holds callers’ threads. If a poison request killed it, retries carry that request onward. Defend with headroom, shedding, outlier ejection and no retry after a crash.

What is a metastable failure, and how do you get out of one? A failure that persists after its trigger, because retries, a cold cache or costly error handling keep load above capacity. Break the loop: cut load hard, disable retries, drop expired work, then ramp up slowly.

Health checks are green but users see errors. What do you check? Assume gray failure. Compare caller-side error rates per replica, find the outlier, and compare what the check exercises with the real request path. Eject it, then fix the check.

Should we fall back to the database when the cache is down? Careful: that adds load exactly when the system is stressed, which is how Amazon’s retail site went down around 2001. Prefer a more reliable cache tier, stale results or failing that feature fast.

Where to go next

• Timeouts, retries and backoff with jitter: deadlines and retry budgets, the first defence against slowness.
• Circuit breaker: stop calling a dependency that keeps failing.
• Bulkheads: cap what one slow dependency can hold.
• Health checks done right: shallow versus deep checks and gray failure.
• Retry storm: the anti-pattern behind most metastable loops.
• Backpressure and load shedding: what to drop under overload.
• Hot keys, cache stampede and invalidation: herds at the cache layer.
• Cell-based architecture: limit the blast radius of any one failure.
• Observability: logs, metrics and traces: the signals behind every “detect” line here.

References and further reading

1. Addressing Cascading Failures (Google SRE book)sre.googleOverload as the main cause of cascades, resource exhaustion and the GC death spiral, the 5%-hanging-requests thread arithmetic, crash-loop recovery numbers, queries of death and cold caches.
2. Gray Failure: The Achilles’ Heel of Cloud-Scale Systems (Huang et al., HotOS 2017)microsoft.comDefines gray failure as differential observability, with Azure examples of probes on the wrong path, recovery that kills and redundancy that hurts.
3. Metastable Failures in Distributed Systems (Bronson et al., HotOS 2021)sigops.orgThe stable, vulnerable and metastable states, the retry and look-aside cache examples, hidden capacity and characteristic metrics.
4. Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems (Gunawi et al., FAST 2018)usenix.org101 fail-slow hardware incidents from 12 institutions, including disks, CPUs and network cards running far below their normal speed.
5. Avoiding fallback in distributed systems (Amazon Builders’ Library)aws.amazon.comWhy rarely exercised fallback paths make outages worse, the 2001 shipping-speed cache incident, and the alternatives Amazon prefers.
6. Static stability using Availability Zones (Amazon Builders’ Library)aws.amazon.comControl plane versus data plane, overprovisioning so that losing a zone needs no new capacity, and zone-aligned deployments.
7. Reliability, constant work, and a good cup of coffee (Amazon Builders’ Library)aws.amazon.comThe constant-work pattern, illustrated by Route 53 health checks that exchange fixed-size result tables whatever the state of the targets.
8. Details of the Cloudflare outage on July 2, 2019 (Cloudflare)blog.cloudflare.comPrimary source for the WAF regular-expression outage: timeline, the global push through Quicksilver, contributing causes and fixes.
9. How and why the leap second affected Cloudflare DNS (Cloudflare, 2017)blog.cloudflare.comA wall clock stepping backwards produced a negative duration that crashed DNS resolver selection on January 1, 2017.
10. More details about the October 4 outage (Facebook Engineering, 2021)engineering.fb.comHow a backbone maintenance command, DNS servers withdrawing their BGP routes and slow physical access combined, and why load was restored gradually.