Failover

Failover is switching work from a failed, or failing, component to a standby or a peer, automatically or by hand. Redundancy gives you a second copy; failover is the mechanism that turns that copy into availability. Every failover runs four steps: detect, decide, promote, redirect. How long they take is your recovery time, and whatever the standby is missing when it takes over is your data loss.

Picture it first: the housing society’s generator

Many Indian housing societies have a diesel generator and an automatic changeover panel. Watch one through a power cut and you see every part of failover.

• Detect. The mains drops, but the panel waits a few seconds, because most dips are flickers. Too fast and you crank the engine for nothing; too slow and the lifts sit dark.
• Promote. The generator cranks and stabilises. One that hasn’t run for months may not start.
• Redirect. The panel switches the building’s feed. The seconds without power are your recovery time.
• Fence. An interlock ensures generator and mains are never connected together, because back-feeding a “dead” street line can kill the lineman repairing it. Two live sources are worse than none: that is split brain.
• Failback and test. When the mains returns, the panel switches back, a second changeover. And the caretaker runs the generator every Sunday; societies that skip it find out during the monsoon.

A flat’s battery inverter, which bridges the gap so the fridge never notices, is a hot standby: near-zero recovery time, paid for every day.

Map it back: mains is your primary, the generator your standby, the panel your failover controller, the flicker delay your health-check threshold, the interlock your fencing.

The four steps of a failover

Every failover, from a load balancer dropping a dead app server to evacuating a region, runs the same four steps:

1. Detect. Missed heartbeats, failed health checks or timeouts suggest the component has stopped. It is never certain: dead and slow look the same.
2. Decide. Something with authority declares it dead: a quorum of observers, an orchestrator or a person.
3. Promote and reconfigure. Pick the most up-to-date replica, fence the old primary, let the replica replay the log it has, and make it writable. Stateless components skip this.
4. Redirect. Clients find the new primary: an IP moves, a DNS record changes, a proxy’s health checks flip, or a registry publishes the address.

Here is the arithmetic for a common self-managed stack: PostgreSQL with Patroni and etcd, with applications reaching the primary through HAProxy. Patroni’s primary holds a leader key in etcd that expires after ttl (30 seconds by default) and renews it on every HA loop (loop_wait, 10 seconds by default; see Patroni’s settings).

• Detect. The key outlives a dead primary by 20–30 seconds, depending on when it last renewed. Worst case: 30 s.
• Decide. Replicas notice within one HA loop, so allow up to 10 s; the freshest eligible replica wins an atomic race to create the key.
• Promote. It replays the WAL it already has and switches to a new timeline. Assume about 2 s.
• Redirect. HAProxy polls each node’s Patroni endpoint GET /primary, which answers 200 only on the leader. Checking every 2 s and requiring 2 passes takes about 4 s.

Worst case: 30 + 10 + 2 + 4 = 46 seconds of failed writes. Best case: 20 + 0 + 2 + 4 = 26 seconds. Detection is about two-thirds of it, and it is a timer you chose. For comparison, AWS says an RDS Multi-AZ DB instance typically fails over in 60–120 seconds, and a Multi-AZ DB cluster in under 35 seconds.

RTO and RPO, precisely

RTO (recovery time objective) is the longest you accept being unable to serve, from when the failure starts hurting users to when they are served again. RPO (recovery point objective) is the most data you accept losing, measured in time: an RPO of 5 seconds means writes from the last 5 seconds may be gone. Both are targets; a drill measures what you actually achieve.

The four steps are most of the RTO for a component failure; add whatever follows, such as caches warming or a backlog draining. RPO comes from replication, because a standby takes over with only what reached it (data replication covers the mechanics).

• Asynchronous. The primary acknowledges a commit, then ships it. Anything not yet shipped dies with the primary, so RPO ≈ replication lag at the moment of failure.
• Synchronous or semi-synchronous. The primary acknowledges only once a standby has the change. Promote that standby and nothing acknowledged is lost: RPO = 0. You pay a round trip on every commit, and a slow or missing standby can stall commits. RDS Multi-AZ DB instances replicate synchronously; Multi-AZ DB clusters use semi-sync, needing one reader’s acknowledgement.

Take a UPI-style payments ledger at a peak of 3,000 writes per second (illustrative), replicating asynchronously.

• On a normal day, lag is about 0.1 s: 3,000 × 0.1 = 300 acknowledged payments exist only on the primary at any moment.
• Primaries rarely die on a calm day. A saturating disk or a lossy network slows replication first. If lag has climbed to 4 s when the primary crashes, 3,000 × 4 = 12,000 customers were told “payment successful” and the record is on a dead machine.

So state RPO as the lag you will tolerate at the worst moment, and enforce it:

• Refuse to promote a stale replica. Patroni ignores followers more than maximum_lag_on_failover bytes behind; Redis Sentinel skips replicas disconnected from the master for too long. The price: if every replica is too stale, nobody is promoted until the primary returns or a human decides.
• Stop accepting writes nobody is copying. Redis’s min-replicas-to-write 1 with min-replicas-max-lag 10 makes a master refuse writes once no replica has acknowledged for 10 seconds. That bounds the loss, and costs write availability when replicas are gone.
• Go synchronous for money. For the ledger, semi-sync to a standby in another zone makes RPO zero; the round trip is the price.

Detect: telling dead from slow

You can’t observe “dead”, only “silent for a while”, so every detector is a timeout: time to detect ≈ check interval × consecutive failures required, plus the check’s own timeout.

Route 53 health checks put numbers on it. Checkers probe every 30 s, or every 10 s at extra cost, and an endpoint changes state after a run of consecutive results set by the failure threshold, which defaults to 3. That is 30 × 3 = 90 s at the defaults, or 10 × 3 = 30 s on the fast interval, before DNS caches even begin to expire. An HTTP check fails if connecting takes over 4 s, or the response over 2 s after that.

Faster detection buys RTO and costs false alarms, and a GC pause or a saturated link can look like death for a few seconds. A needless failover can discard the async window, break every connection, leave the new primary with cold caches and, across regions, add latency to every request.

Three ways to detect better, not just faster:

• Use several observers in different places. Route 53 runs checkers in many locations and calls an endpoint healthy if more than 18% of them say so, so trouble near a few checkers can’t fail it. One observer can’t tell “the primary died” from “my link to it died”.
• Check what users need. A process that accepts TCP with a full disk isn’t healthy. But a check that queries a shared database fails every app server when the database blips, and the load balancer removes the whole fleet (health checks done right).
• Adapt to the network. A φ-accrual detector, used by Cassandra and Akka, learns the distribution of heartbeat gaps and outputs a suspicion level, φ = −log10(chance that a heartbeat this late is merely late). Cassandra convicts at φ = 8: a live node’s heartbeat would be that late only about once in 100 million, if the network behaves like its recent past. On a jittery network, conviction automatically comes later.

Decide: who pulls the trigger

Detection says “I can’t reach it.” Deciding says “it is dead, and I am replacing it.” Get the second half wrong and you get two replacements.

Google’s SRE book calls leader election by heartbeats and timeouts “conceptually unsound”: in its case study, file-server pairs that shut each other down (STONITH) when heartbeats stopped ended up both active or both down once the network got slow. Choosing a leader is a consensus problem, so lean on a consensus system such as etcd or ZooKeeper.

A quorum in practice: Redis Sentinel

Sentinel separates noticing from acting. A Sentinel that gets no valid reply for down-after-milliseconds marks the master SDOWN (subjectively down); once quorum Sentinels agree, it is ODOWN (objectively down). That only detects. To act, one Sentinel must win votes from a majority of all Sentinels, which gives it a new configuration epoch no other Sentinel can reuse. It promotes the replica with the best replica-priority, then the largest replication offset, and as the new configuration spreads, the higher epoch wins. Clients ask Sentinel for the master’s address.

With five Sentinels and a quorum of 2, two can declare ODOWN but three must authorise, so there is no failover on the minority side of a partition. Run at least three Sentinels, in three zones.

People for the big moves

AWS’s disaster recovery whitepaper says automatic failover on health checks or alarms “should be used with caution”, because a false alarm still costs real downtime and data. So regional failover is often started by a person, with every later step automated into one button press that only touches the data plane, such as flipping a health check’s state. The SRE book gives the counterweight: human-gated failover costs availability and scales badly, and the engineers are often already overloaded, because primaries and secondaries lose touch most often during bigger incidents.

Promote and fence: never two primaries

PostgreSQL’s documentation is blunt: “PostgreSQL does not provide the system software required to identify a failure on the primary and notify the standby database server.” It gives you the promotion (pg_ctl promote) and a new timeline marking a new history; detection, decision and fencing are the job of tools like Patroni.

Promotion should pick the freshest safe candidate. In Kafka, each partition tracks an in-sync replica set (ISR) of followers caught up with the leader, and new leaders come from it. If every in-sync replica is gone, unclean.leader.election.enable chooses between waiting for one to return (the default since Kafka 0.11.0.0) and promoting whichever replica returns first, which can lose committed messages. RPO against RTO, as one flag.

Why are two primaries worse than none? With none, writes fail and clients retry. With two, both accept writes, histories fork, and someone’s acknowledged writes must later be discarded or merged by hand. An outage ends when the node returns; a fork does not. Layer the defences:

• Majority decisions. Only a side holding a majority can elect.
• Leases with a safety margin. The old primary must stop before anyone else starts. Patroni stops its PostgreSQL when it can’t renew the key; because a hung process can’t do even that, Patroni can also arm a watchdog that resets the machine, by default 5 seconds before the key expires.
• Epochs. Every promotion gets a strictly higher number: a Sentinel configuration epoch, a Raft term, a PostgreSQL timeline.
• Fencing tokens. Send the epoch with every write downstream and have the receiver refuse lower numbers. A paused process can’t know it was paused, so only the receiver can enforce this (distributed locking).
• STONITH (“shoot the other node in the head”). Cut the old primary’s power, port or storage before promoting; PostgreSQL’s docs call such a mechanism necessary. Use it alongside the others, since the kill command crosses the same sick network.

Redirect: getting clients to the new primary

A promoted primary that nobody can find is still an outage. Four ways to point clients at it:

• Floating IP. VRRP or keepalived moves a virtual IP to the new machine and announces it with a gratuitous ARP, in about 3.6 s at protocol defaults (single point of failure derives it). Clients change nothing, but it works only within one subnet.
• DNS. Update the record; RDS fails over by repointing the instance’s DNS name at the standby. Resolvers and clients cache the old answer for the TTL, some for longer: AWS recommends a JVM DNS cache TTL of at most 60 seconds, warning that some Java configurations never refresh until restart (DNS).
• A health-checking proxy. HAProxy, a cloud load balancer or a database proxy routes to whichever node says it is primary. Everyone switches within seconds behind one stable address, but it adds a hop that needs its own redundancy (load balancing).
• Client-side discovery. Clients ask a registry: Sentinel, an etcd or ZooKeeper watch, Redis Cluster’s MOVED redirects, Kafka metadata refreshes. No extra hop, but every client library must get it right (service discovery).

The moments after the switch are where users feel it:

• In-flight work fails. Open transactions on the old primary are gone and clients retry, so make writes idempotent: a payment retried after failover must not charge twice.
• Pools hold dead sockets. Evict connections on error and cap their lifetime (connection pooling).
• Retries arrive together. Thousands of clients reconnecting at once can flatten a cold new primary; use backoff with jitter.
• Planned switchovers can drain. For maintenance, stop new work, let in-flight work finish, then switch: zero errors instead of a few seconds of them.

Cold, warm, hot and active-active

How ready the standby is sets your daily bill and your takeover time. Cold is backups plus code to rebuild; warm runs but is smaller or out of the traffic path; hot is full-size, in sync and ready in seconds. Active-active has no standby: every copy serves, and AWS notes there is then “no such thing as failover”, only routing away from a failed site, if the others can absorb its load.

Aurora shows the gap inside one product: with a replica to promote, service is typically restored in under 60 seconds, often under 30; with none, Aurora recreates the primary, typically in under 10 minutes.

AWS’s reliability pillar lists the four disaster recovery strategies “in increasing order of cost and complexity, and decreasing order of RTO and RPO”:

Inside a region the same spectrum appears: a stateless N+1 pool behind a load balancer is active-active, and a database with a synchronous standby in another zone is hot active-passive. Mind any scale-up step: AWS treats auto scaling as a control-plane action and calls depending on the control plane during recovery an anti-pattern, because it may be struggling in the same event.

Going deeper: when failover makes things worse

Flapping

A half-alive node passes and fails checks in turn, and every flip moves traffic. Add hysteresis: require several consecutive results to change state in either direction (Route 53’s failure threshold works both ways), hold off after any failover, and never fail back automatically.

A standby that can’t carry the load

A warm standby at a quarter of production size receives all of production’s traffic and falls over. A new primary with a cold buffer cache serves everything from disk. In a pool, survivors absorb a dead neighbour’s share, slow down, fail their own health checks and are removed in turn: a cascading failover. Size standbys for full load, keep them warm, and shed load while the new primary catches up.

A blip that triggers a big move: GitHub, October 2018

At 22:52 UTC on 21 October 2018, replacing failing optical equipment cut the link between GitHub’s US East Coast network hub and its primary East Coast data centre for 43 seconds. Orchestrator, GitHub’s Raft-based MySQL failover tool, formed a quorum from nodes in the West Coast data centre and an East Coast public cloud and promoted West Coast primaries, exactly as configured. The East Coast primaries held a few seconds of writes the West never received (954 on one of the busiest clusters), and the West soon held writes the East lacked, so failing back was unsafe. GitHub chose data integrity and ran degraded for 24 hours and 11 minutes (data replication tells the replication side).

Three failover lessons:

• Match detection time to the size of the move. 43 seconds of silence moved primaries across the country; a regional move deserves a longer, surer outage than a node move.
• Allowed topologies must match what the application tolerates. GitHub’s first technical follow-up was to stop Orchestrator promoting primaries across regions: East Coast applications couldn’t absorb a cross-country round trip on most database calls.
• Async plus automatic cross-region promotion can fork history, and then there is no failback, only repair.

Failback is a second failover

After failover you run in what PostgreSQL’s docs call a degenerate state: one server, no standby. Rebuild redundancy first, resynchronising the old primary as a standby (pg_rewind speeds this up) or building a fresh one. Move back only if you must, as a planned, drained switchover at a quiet hour; never automatically, and never while the old node holds writes the new one lacks.

Test it, or it doesn’t exist

A failover you haven’t tested is a hope. Standbys rot quietly: configuration drifts, a credential expires, a replica stopped replicating last month, the recovery region’s quota is too low to scale up.

• Switch over on a schedule. PostgreSQL’s docs recommend regular switching between primary and standby: it allows maintenance on each machine and tests the failover mechanism.
• Force real failovers. RDS offers “reboot with failover”, Sentinel SENTINEL FAILOVER, Patroni patronictl failover. Run them in production at a quiet hour with on-call watching.
• Run game days and chaos experiments. Kill the primary, black-hole its network, fill its disk, isolate a zone; inject partial failures too, such as a slow disk or packet loss.
• Measure what users saw. Time each of the four steps, count acknowledged writes missing afterwards, and watch client errors until they return to normal. Those are your real RTO and RPO.

GitHub’s post-incident commitments included exactly this: validating failure scenarios with fault injection and chaos engineering before they reach users.

Cheat sheet

How to say it in the interview

Walk the tiers of your design and give each one a mechanism and two numbers.

“The app tier is stateless, N+1 across three zones; an instance failing two health checks five seconds apart leaves rotation in about ten seconds, with no data at risk. Redis has three Sentinels in three zones, so a replica is promoted within seconds; replication is async, so we may lose a few cache writes, which is fine because the database is the source of truth. The payments database has a semi-synchronous standby in another zone, promotion decided by a quorum through Patroni and etcd, and the old primary fenced by epoch: RPO zero, RTO under a minute, and clients retry idempotently. Kafka runs replication factor 3, acks=all, min in-sync replicas 2, no unclean leader election. Across regions we keep a warm standby with async replication, RPO of seconds and RTO of minutes, started by a person pressing a scripted button. And we fail over on purpose every quarter.”

How do you prevent split brain? Only a majority can elect, the old primary’s lease expires before a new one is granted, and every promotion carries a higher epoch that downstream storage checks. STONITH is a backstop, not the only defence.

Why not detect failures as fast as possible? Short timeouts fire on GC pauses and network blips, and a needless failover breaks connections, may discard async writes and can move you somewhere slower. Use several observers, require consecutive failures, and make bigger moves wait longer.

The async replica is 5 seconds behind when the primary dies. What now? Promote it and lose 5 seconds of acknowledged writes, or wait and stay down. Decide in advance with a maximum promotion lag, keep the old primary’s log for reconciliation, as GitHub did, and use semi-sync where losing writes is unacceptable.

Should regional failover be automatic? Usually not: it is rare and expensive, and a false trigger can fork your data. Automate every step after the decision, prefer data-plane actions, and let a person press the button. Automate node- and zone-level failover.

How do you know your failover works? Run it: scheduled switchovers, forced failovers in production at quiet hours and game days, timing each step and counting lost writes.

Where to go next

• Single point of failure: finding what needs a standby, and VRRP and anycast front doors.
• Fault tolerance: keeping faults from becoming failures at all.
• Availability and the nines: how MTTR, which failover shrinks, turns into nines.
• Data replication: sync, async and semi-sync, and lost writes on failover.
• Consensus and leader election: how Raft and Paxos make “who is primary” safe.
• Distributed locking: leases and fencing tokens in depth.
• Health checks done right: detection that doesn’t take down the whole fleet.
• Disaster recovery: backups, RTO and RPO when a whole region is lost.
• Multi-region and active-active: running without a standby at all.

References and further reading

1. Failing over a Multi-AZ DB instance for Amazon RDSdocs.aws.amazon.comTypical failover of 60–120 seconds, failover by repointing the DNS record, and the advice to keep the JVM’s DNS cache TTL at 60 seconds or less.
2. AWS Well-Architected Reliability Pillar, REL13-BP02: Use defined recovery strategies to meet the recovery objectivesdocs.aws.amazon.comThe four DR strategies with AWS’s RPO and RTO ranges, hot standby as a fully scaled warm standby, and control-plane dependence as an anti-pattern.
3. Disaster Recovery of Workloads on AWS: Disaster recovery options in the clouddocs.aws.amazon.comWhy automatic failover on health checks should be used with caution, button-press manual failover, data-plane-only recovery, and “no such thing as failover” in active-active.
4. Amazon Route 53: How Route 53 determines whether a health check is healthydocs.aws.amazon.com10- or 30-second check intervals, the 4-second connect and 2-second response limits, and the 18% rule across health checkers in many locations.
5. Redis Sentinel documentationredis.ioSDOWN and ODOWN, quorum for detection versus majority for authorisation, configuration epochs, replica selection, and min-replicas-to-write.
6. PostgreSQL documentation: Failover (log-shipping standby servers)postgresql.orgPostgreSQL leaves failure detection to external tools, requires STONITH-style fencing, and recommends regular switchovers as a test of failover.
7. Patroni documentation: Replication modespatroni.readthedocs.ioWhat asynchronous failover can lose, maximum_lag_on_failover, and synchronous_mode versus synchronous_mode_strict.
8. How to do distributed locking (Martin Kleppmann, 2016)martin.kleppmann.comProcess pauses that outlive a lease, and fencing tokens (33 versus 34) enforced by the storage service.
9. October 21 post-incident analysis (GitHub, 2018)github.blogA 43-second partition, an Orchestrator quorum that promoted West Coast primaries, 954 unreplicated writes on one cluster, and 24 hours 11 minutes of degradation.
10. Site Reliability Engineering, chapter 23: Managing Critical State (Google)sre.googleCase studies of heartbeat-and-STONITH split brain and of human-gated failover, and why leader election needs real consensus.