Single point of failure (SPOF)

A single point of failure (SPOF) is any part of a system whose failure, on its own, stops a critical user journey. It can be a box on your diagram, such as a lone load balancer or primary database, or something nobody draws: a certificate, a config push, a payment provider, or the one engineer who knows the database password. Finding SPOFs, then removing them or shrinking what they can break, is most of practical reliability work.

Picture it first: the dosa stall with two tawas

A dosa stall outside a college hires a second cook for a second tawa, so one sick cook no longer closes it. Then three things go wrong in one month:

• The single gas cylinder runs out at 7 pm. Both tawas go cold together.
• The UPI QR code pays into one bank account, and that bank has a bad evening. Students carry no cash, so sales stop.
• The owner, the only person with the shutter key, goes to a family wedding. The stall stays shut for two days.

None of these was a cook. The owner removed the obvious single point of failure and kept three hidden ones: a shared dependency (the gas), a third-party provider (the bank) and a person. And if the street floods in the monsoon, two cylinders won’t help: at that scope, the street itself is the SPOF.

Map it back: the cooks are app servers, the cylinder is shared infrastructure such as a zone’s power or a config service, the bank is your payment provider, the owner is the one engineer with production access, and the street is a cloud region.

What makes something a SPOF

A component is a SPOF for a journey when two things are both true:

1. The journey hard-depends on it: no second copy, no cached answer, no fallback path.
2. Nothing takes its place within the time a user will wait.

Three consequences follow, and interviewers probe all of them.

It is per journey. If the recommendation service dies and the home page renders without its carousel, it is not a SPOF; if the page throws an exception instead, it is. A payment provider is a SPOF for checkout but not for browsing. Azure’s guidance calls this classifying dependencies as strong or weak.

It is per failure scope. Redundancy only covers failures smaller than the area your copies are spread across. “No SPOFs” means nothing until you say for which failure:

It is not only hardware. People, processes, credentials, certificates, config and the deploy pipeline all count. Azure’s failure mode analysis guide lists misconfiguration, operator error and planned maintenance right beside zone and region outages.

One number shows why a single SPOF matters so much. Two independent replicas at 99.9% each fail together only 0.001 × 0.001 of the time, so the pair is 99.9999% on paper. Put one 99.9% load balancer in front and the path drops straight back: 0.999 × 0.999999 ≈ 99.9%. The SPOF sets your ceiling. And if the two replicas share a zone or a config push, they were never independent and the six nines evaporate; the arithmetic is in availability and the nines.

How to find SPOFs

Diagrams show the request path and leave out everything the path quietly needs. A systematic walk, which Azure’s Well-Architected guidance calls failure mode analysis (FMA), finds what they hide.

1. List the critical journeys, not the services. For a food-delivery app: browse, order, pay, assign a rider, track. For an IRCTC-style tatkal rush: log in, search, book, pay.
2. Walk each journey box by box and arrow by arrow, asking: “If this disappears right now, what does the user see?” Treat reads and writes separately: a dead read replica may cost nothing; a dead primary blocks every write.
3. Draw what nobody draws: DNS, the auth service, config and feature flags, secrets and keys, TLS certificates, service discovery, NAT and egress, the CI/CD pipeline, the cloud control plane (anything that launches or scales capacity), monitoring, and the people.
4. Ask three questions of every dependency. Is it hard or soft? How many copies, sharing which failure domain? What does recovering it depend on?
5. Rank by likelihood × impact and decide: remove it, contain it, or accept it and write down why.
6. Test the answers with game days and chaos experiments: kill an instance, black-hole a zone, expire a certificate in staging. Facebook credited its “storm” drills, which take a service, a data centre or a region offline, for its careful restart in October 2021.

Count the red boxes that aren’t servers. Once teams run two of everything with a CPU, the remaining SPOFs live in shared services, external providers and people.

The usual suspects

These are the SPOFs interviewers expect you to spot in a typical design, with the standard fix, what it costs and what stays risky afterwards.

Four rows deserve a closer look.

The cache. Google’s SRE book separates a latency cache, where the service can carry full load with an empty cache, from a capacity cache, where it cannot. Only the second is a SPOF, and it often arrives by accident as traffic grows. Say 50,000 reads a second hit a 95% cache: the database sees 50,000 × 0.05 = 2,500 reads/s. Restart the cache cold and it sees all 50,000, 20× its normal load. Size the database for a cold cache, or warm the cache slowly behind request coalescing and rate limits (cache stampede).

The singleton job. “Exactly one must run” creates a SPOF. Leader election with a lease removes it, but makes the coordinator a dependency and needs fencing tokens so a paused old leader can’t act (distributed locking). Often simpler: make the job idempotent and run it on two machines.

The payment gateway. A router across two payment service providers (PSPs) that shifts traffic by recent success rate keeps checkout alive when one degrades, for the price of two integrations and daily reconciliation. It can’t route around the customer’s bank or the UPI switch, which every PSP shares (design a payment system).

The coordinator. ZooKeeper and etcd accept writes only while a majority is up, so placement matters. Three members in two zones put two in one zone, and losing that zone loses the majority; one member in each of three zones survives any zone.

Removing SPOFs with redundancy

AWS’s reliability pillar calls avoiding single points of failure, down to the physical infrastructure, a fundamental principle, and asks for at least two zones in production. Redundancy is the main tool; the details decide whether it works.

N+1, N+2 and where the spare lives

Call N the copies you need at peak. N+1 survives one failure. N+2 survives a failure while another copy is down for a deploy, which is when failures love to happen.

A spare only helps if the survivors can carry the load. Google’s SRE book describes a frontend cluster at 1,000 requests per second whose sibling cluster fails: it suddenly gets 1,200, can’t cope, and its useful throughput falls below 1,000 as servers crash. Redundancy without headroom is a slow cascading failure.

Zones change the arithmetic. Suppose checkout needs 8 app servers at peak:

• Two zones: the survivor carries everything, so each needs 8: 16 servers, 2× peak.
• Three zones: two survivors carry half each, so each needs 4: 12 servers, 1.5× peak.

This is the 50% over-provisioning in AWS’s Builders’ Library article on static stability: with three zones, each runs at about 66% of its tested load. The capacity exists before the failure, so recovery never waits on launching instances through a control plane that may be struggling too.

Stateless copies are easy; state is not

Stateless tiers just need copies behind a load balancer and health checks to remove dead ones. State is harder because copies must agree.

• Active-passive. A primary takes writes; a standby replays its log and is promoted when the primary fails. Simple and cheap, but writes pause during promotion, and an asynchronous standby can lose its last few writes (failover).
• Active-active. Every copy serves traffic, so writes need partitioning or conflict resolution (CAP and PACELC). No idle hardware or promotion step; you pay in conflicts and harder reasoning.
• Quorums. Consensus stores need a majority, so 2f + 1 members tolerate f failures: 3 tolerate 1, 5 tolerate 2. Four still tolerate only one, since a majority of 4 is 3 (data replication, quorums).

Redundancy recurses: who protects the load balancer?

Put two app servers behind a load balancer and you have moved the SPOF, not removed it. Every fix adds a component that needs its own fix, until you reach something run by someone with far more redundancy than you could build, or something cheaper to accept; write that stopping point down. There are three standard ways to make the front door redundant.

A floating IP with VRRP. Two load balancers share one virtual IP. RFC 5798’s Virtual Router Redundancy Protocol (keepalived implements it on Linux) elects a master that owns the address and advertises every second by default. A backup at the default priority of 100 declares the master dead after 3 × 1 s + (256 − 100) ÷ 256 s ≈ 3.6 seconds of silence, takes the address and broadcasts a gratuitous ARP so switches send it the traffic. You gain failover in seconds with no client changes. You pay for an idle standby, both boxes share one subnet and site, and if advertisements are lost while both are alive, both may claim the address: split brain.

DNS with several records. Publish several load balancer addresses, possibly in different sites; clients pick one, and many retry another on failure. You gain simplicity and reach across regions. You pay in control: resolvers cache answers for the TTL and some clients never retry, so a dead address keeps getting traffic until you pull it and caches expire (DNS).

Anycast. Announce the same IP prefix over BGP from many sites. Routers deliver each user to the nearest one in routing terms; a failed site withdraws its route and users flow to the next nearest. You gain survival of whole sites with no client cooperation. You pay with your own address space and BGP operations (or a provider’s), and the withdraw logic becomes a shared risk, as Facebook found.

Cloud load balancers bundle these; AWS says Elastic Load Balancing is provisioned to withstand a zone impairment without scaling up. That makes the provider your SPOF, usually the right one to accept (load balancing).

Hidden SPOFs and shared fate

Redundancy works only if copies fail independently. Shared fate is anything that takes out every copy at once:

• Same place: one rack, power feed, switch or zone.
• Same software: one malformed request (a “poison pill”) crashes every replica running that build.
• Same change: a config value or binary pushed everywhere at once.
• Same control plane: autoscaling, instance launches, DNS updates.
• Same dependency or vendor: one auth service, one secrets store, one agent on every machine.

Three outages worth knowing

Facebook, 4 October 2021. During routine maintenance, a command meant to assess available backbone capacity instead took down every backbone connection, cutting Facebook’s data centres off from each other and the internet. An audit tool should have blocked it; a bug let it through. Then a safety feature made it worse: Facebook’s DNS servers withdraw their BGP announcements when they can’t reach the data centres, a sensible health check for one sick site. With the backbone gone, every site withdrew at once, so the DNS servers were running but unreachable. Losing DNS also broke many internal tools, and with normal and out-of-band network access both down, engineers had to go to the data centres in person, through deliberately tough physical security.

AWS us-east-1, 7 December 2021. AWS runs monitoring, internal DNS, authorisation and parts of the EC2 control plane on an internal network joined to the main one by networking devices. An automated scaling activity triggered a surge of connections that overwhelmed those devices, and retries kept them congested. Running EC2 instances and existing load balancers carried on, but anything needing the control plane failed: EC2 API errors began at 7:33 AM PST and instance launches recovered only at 2:40 PM. Route 53 kept answering queries while record changes were blocked from 7:30 AM to 2:30 PM, and container tasks that died could not be replaced. Operators were partly blind, because monitoring sat behind the same congestion, and their internal deployment systems were impaired too.

CrowdStrike, July 2024. A content update to CrowdStrike’s security agent crashed Windows machines around the world. Microsoft estimated 8.5 million devices, under 1% of all Windows machines, yet the damage was broad because the agent ran on critical systems at many enterprises. Fleets that were redundant in every other way all ran the same agent and took the same update.

The pattern is the same each time: the failure did not hit a box; it hit something every box shared. (Cloudflare’s 2019 global outage from one firewall rule, covered in availability and the nines, is a fourth.)

Shrinking the blast radius

Some SPOFs can’t be removed at a sensible price: the one region your data must stay in, the global config system, the UPI switch. If you can’t make a failure impossible, make it small. Azure names both halves: build more resilience, and design for degraded operation.

Cells. Split the whole stack into independent copies, each serving a slice of customers chosen by a partition key such as customer or restaurant ID, behind a thin router. AWS’s whitepaper: with 10 cells serving 100 requests, a failure in one cell leaves 90% untouched. Cells needn’t mean more hardware; the same 30 hosts can be regrouped. You pay with a routing layer, more deployments and awkward cross-cell queries. The router is shared by every cell, so keep it as simple as possible, with no business logic (cell-based architecture).

Bulkheads. Give each dependency, or each class of customer, its own connection and thread pool, so one slow dependency can’t drain every thread (bulkheads).

Shuffle sharding. AWS’s Builders’ Library uses 8 workers. Split into 4 plain shards of 2, one bad customer takes down a shard: 25% of the service. Give each customer its own pair of workers instead, and there are 8 × 7 ÷ 2 = 28 possible pairs, so a poison customer fully takes down only the customers sharing its exact pair: about 1/28, if clients retry on the worker they still have. Route 53 uses 2,048 virtual name servers, four per domain: over 730 billion combinations.

Static stability. Let the data plane run on its last known state when the control plane fails. EC2 works this way: each host holds everything it needs to keep routing packets, which is why running instances carried on in December 2021 while launches failed. Pre-provision capacity, cache config and credentials locally, and never make recovery depend on creating something new (control plane vs data plane).

Staged rollouts, for config too. The Facebook, CrowdStrike and Cloudflare outages all began with a change. Ship every change, including config, flags and firewall rules, one cell or zone at a time, with bake time and automatic rollback (deployment strategies).

Graceful degradation. Turn hard dependencies into soft ones. Azure’s example: if the recommendation engine dies, customers still search and pay. Serve the cached menu if the catalogue is slow, hide ETAs if the maps provider is down, offer cash on delivery if every PSP fails. Add timeouts, circuit breakers and load shedding so a sick dependency fails fast. And test the fallbacks: code that only runs during outages breaks unnoticed.

When to keep a SPOF

Removing a SPOF costs hardware, engineering time and new failure modes: every failover mechanism can misfire, and split brain is worse than a short outage. AWS even lists going multi-Region when multi-AZ would do as an anti-pattern. So price both sides.

Take a food-delivery checkout averaging ₹30 lakh of orders an hour: 30,00,000 ÷ 60 = ₹50,000 a minute. The inputs below are illustrative assumptions; the method is the point.

• The primary database. With a synchronous standby in another zone, assume two failovers a year with about two minutes of failed writes each: 4 × ₹50,000 = ₹2 lakh a year at risk, less once customers retry. Multi-primary would buy almost nothing. Keep the single primary.
• The region. Assume a four-hour regional outage every two years: 2 hours a year × ₹30 lakh = ₹60 lakh a year of expected loss.
  • Active-active at an assumed ₹1.2 crore a year spends ₹1.2 crore to protect ₹60 lakh. Revenue alone doesn’t justify it.
  • A warm standby region that takes over within an hour, at an assumed ₹30 lakh a year, cuts the loss to 0.5 hours a year × ₹30 lakh = ₹15 lakh. It saves ₹45 lakh for ₹30 lakh: worth buying.

Trust, contracts, regulators and orders in flight can tip the balance, so add them when they apply. But many SPOFs are fine to keep: a nightly report that can rerun, an admin dashboard, an early startup’s single region. Keep one when its expected downtime costs less than removing it, its recovery is rehearsed, and the decision is written down with its scope (disaster recovery, cost-aware design).

Cheat sheet

How to say it in the interview

Before you finish any design, scan it for SPOFs in about a minute, left to right: the front door (DNS, CDN, load balancer), compute (stateless, N+1 across zones?), state (database, cache, broker), coordination (cron jobs, locks), the outside world (payments, SMS, maps), then operations (config, deploys, certificates, people). For each: one or many, sharing what, and what if it vanishes?

“Let me check this for single points of failure, against losing a host or a zone. DNS is an anycast provider and the load balancer is managed and multi-zone. The app tier is stateless, N+1 across three zones at about two-thirds of tested capacity, so losing a zone needs no new instances. The orders database gets a synchronous standby in another zone with automatic failover: a short write pause, no lost orders. The cache is a latency cache, so the database survives it going cold. Kafka runs replication factor 3 across zones, and the rider-assignment job holds a lease so a standby can take over. Beyond our boxes: two payment gateways behind a router, and config rolls out one zone at a time. For the region, I’d accept up to an hour of downtime with a warm standby, because active-active costs more than the downtime it saves.”

Your load balancer is now the SPOF. What do you do? Use a managed multi-zone load balancer. On your own hardware, run a VRRP pair on a floating IP (about 3.6 seconds to take over at defaults); across sites, publish several DNS records or use anycast. Each step removes one scope of failure; stop where the remaining SPOF is a provider you trust.

Is a single primary database a SPOF? For writes, yes, until it has a standby and automatic failover. With a synchronous standby in another zone, losing the primary costs a short write pause, not data. Many systems never need more; multi-primary adds conflicts for little extra availability.

How do you find SPOFs that aren’t on the diagram? Walk each critical journey and list everything it needs: DNS, auth, config, secrets, certificates, the deploy pipeline, the control plane, people. Then ask what recovery depends on, which is where Facebook and AWS got stuck. Prove it with game days.

Two replicas in different zones. Is that enough? Only if nothing else is shared. A config push, build or dependency that reaches both at once makes them one copy again. Stagger changes by zone, and make sure one survivor can carry the full load.

You can’t afford a second region. Now what? Shrink what a regional problem can do: three zones with static stability, cells, staged rollouts, and backups in another region with a tested restore. Then say plainly that a regional outage means downtime, and give the recovery time.

Where to go next

• Availability and the nines: the serial, parallel and shared-fate arithmetic.
• Failover: detection, promotion, split brain, RTO and RPO.
• Fault tolerance: keeping faults from turning into failures.
• Reliability patterns cheat sheet: timeouts, retries, circuit breakers and bulkheads.
• Cell-based architecture: cells and the thin router in front.
• Multi-region and active-active: removing the region as a SPOF.
• Consensus and leader election: quorums for stateful coordination.
• Backend failure modes: how components fail once you’ve found them.

References and further reading

1. Azure Well-Architected Framework: Architecture strategies for performing failure mode analysislearn.microsoft.comDecomposing a workload, overlaying critical flows, internal and external dependencies, the list of failure modes, strong versus weak dependencies, and degraded-mode mitigations.
2. AWS Well-Architected Reliability Pillar, REL10-BP01: Deploy the workload to multiple locationsdocs.aws.amazon.comAvoiding single points of failure down to physical infrastructure, at least two AZs for production, and the anti-pattern of going multi-Region when multi-AZ would do.
3. Static stability using Availability Zones (Amazon Builders’ Library)aws.amazon.comControl plane versus data plane, the 50% over-provisioning rule for three AZs, and active-active and active-standby patterns that need no action when a zone fails.
4. Reducing the Scope of Impact with Cell-Based Architecture (AWS whitepaper, 2023)docs.aws.amazon.comCells as bulkheads, the 10-cell, 90%-unaffected example, and why the shared cell router must stay as thin as possible.
5. Workload isolation using shuffle sharding (Amazon Builders’ Library)aws.amazon.comThe eight-worker example (25% impact with plain sharding, 1/28 with shuffle sharding) and Route 53’s 2,048 virtual name servers in shards of four.
6. Site Reliability Engineering, chapter 22: Addressing Cascading Failures (Google)sre.googleHow losing one cluster overloads the rest (1,000 to 1,200 QPS), and latency caches versus capacity caches that quietly become hard dependencies.
7. RFC 5798: Virtual Router Redundancy Protocol (VRRP) Version 3rfc-editor.orgMaster and backup election for a shared virtual IP, the 1-second default advertisement interval, Master_Down_Interval and the gratuitous ARP on takeover.
8. More details about the October 4 outage (Engineering at Meta, 2021)engineering.fb.comA maintenance command disconnected the global backbone, DNS servers withdrew their BGP routes by design, and internal tools that needed DNS failed with it.
9. Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, December 7, 2021aws.amazon.comCongestion on an internal network that hosted monitoring, internal DNS, authorisation and parts of the EC2 control plane; running workloads survived while control-plane actions failed.
10. Helping our customers through the CrowdStrike outage (Microsoft, July 2024)blogs.microsoft.comMicrosoft’s estimate that one security-agent update affected 8.5 million Windows devices, under 1% of all Windows machines.