Heartbeats

A heartbeat is a small message that one process sends on a fixed schedule to say “I’m still here”. A monitor that stops hearing it for long enough declares the sender failed and acts: it fails over to a replica, hands the work to someone else, or takes the node out of rotation. The hard part is not sending the beat; it is choosing how long silence must last before you act.

Picture it first: the guards’ check-in call

A security agency guards 200 ATM kiosks overnight. Every guard must phone the control room every 15 minutes and say “all well”, and the supervisor notes the time of each guard’s last call.

When a call is late, the supervisor must choose. Send a patrol van after one missed call and the vans chase guards whose phones lost signal all night. Wait for four missed calls and a robbery has an hour’s head start. Calls every 5 minutes surface problems sooner, at triple the switchboard load.

From silence alone, the supervisor can never tell whether a guard is hurt, asleep, or in a basement with no signal. A late call is evidence, not proof.

Map it back: guards are nodes, the control room is the monitor, “all well” is the heartbeat, the schedule is the interval, “four missed calls” is the timeout, and the van is failover. When the supervisor phones a guard instead, that is a pull probe.

What a heartbeat is and what it is for

There are two ways to collect evidence of life:

• Push (heartbeat). The monitored node sends a beat every interval and the monitor records arrival times. Cheap for the monitor, and a node too broken to send stops being trusted automatically.
• Pull (probe or ping). The monitor asks “are you there?” and waits for a reply. It controls the rate and learns the round-trip time, but must know whom to ask. Load balancer health checks and SWIM-style pings work this way.

Many systems use both; Amazon describes servers self-reporting to central monitoring, plus active checks for servers too broken to report.

A beat should carry a sequence number (to tell a late beat from a lost one) and an incarnation number that changes on restart, so a rebooted node is never confused with its old self. Load or progress counters turn it into a status report too.

What happens when beats stop depends on the node’s job: fail over a leader or primary (failover, leader election); reassign its partitions, shards or jobs; release its locks, leases and ephemeral registrations; stop sending it traffic via the load balancer or service discovery; and re-replicate its data.

Slow looks exactly like dead

On a real network there is no upper bound on how long a message can take, or how long a process can stall before sending one. A beat 4 seconds late might mean a crash, a garbage-collection freeze, a congested switch or a dropped packet. To the monitor all four look the same: silence.

So every practical failure detector is unreliable: sometimes it suspects a node that is alive. Chandra and Toueg made this precise in the 1990s with two properties:

• Completeness: every node that really crashes is eventually suspected, for good.
• Accuracy: nodes that are alive are not suspected.

Completeness is easy: suspect anyone who goes quiet. Perfect accuracy needs timing guarantees the network doesn’t give, the same wall FLP puts in front of consensus. So real detectors guarantee completeness and tune accuracy, measured as detection time (crash to permanent suspicion) and mistake rate (wrong suspicions per second). Every knob in this article moves one against the other.

The timeout arithmetic

A fixed-timeout detector has two knobs: the interval between beats and the number of beats it will miss, k, before suspecting. Its worst-case detection time is roughly:

detection time ≈ k × interval + network delay

Compare two settings for one monitored node:

The second setting detects a crash about 3× faster for 10× the messages. The column people miss is the last: pause tolerance comes from the total timeout, not the number of beats. Any GC pause, VM migration or disk stall longer than the timeout looks like a crash.

Beats per timeout buy tolerance to lost packets. With independent 1% loss, three beats in a row vanish with probability 0.01³ = 10⁻⁶. Negligible? Across a fleet: 1,000 nodes × 86,400 beats a day × 10⁻⁶ ≈ 86 false suspicions a day. Five misses gives 10⁻¹⁰, about 0.009 a day. Real losses are worse than independent: in the phi accrual paper’s week-long Japan–Switzerland experiment, about 0.4% of beats were lost, mostly in bursts, the longest 1,093 beats (roughly 2 minutes).

What real systems choose

Real defaults span three orders of magnitude, and the spread is not random: each system sizes its timeout from the cost of the action it triggers.

The pattern: cheap, reversible actions get short timeouts; expensive ones get long timeouts. A Raft election costs about a second of paused writes and loses no committed data, so etcd acts after 1 s. A dead DataNode means copying every block it held, so HDFS waits 2 × 300 s (recheck interval) + 10 × 3 s (heartbeat) = 630 s. Kubernetes uses two stages: stop scheduling onto a silent node after 50 s, evict its pods 5 minutes later.

Kafka re-tuned this trade-off in public. Its default session timeout rose from 10 s to 45 s in Kafka 3.0 (KIP-735): in cloud environments “transient network/load failures are much more common than genuine client failures”, and each false alarm costs two rebalances, one when the member is dropped and one when it rejoins.

Choosing the timeout: fixed, jittered, adaptive

Fixed, sized from the round trip

Start from measurements, not round numbers. The etcd tuning guide is a good template: heartbeat interval about 0.5–1.5× the round-trip time between members, election timeout at least 10× the round-trip time. Its 100 ms and 1,000 ms defaults suit one data centre; across continents it allows up to 50 s. The Raft paper’s version is broadcastTime ≪ electionTimeout ≪ MTBF. etcd also warns that a slow disk alone can make a leader miss heartbeats.

Jitter

When 5,000 nodes restart together they beat in lockstep, and the monitor sees a spike every interval instead of a smooth stream. Randomise each node’s first beat and jitter every interval a little. Raft randomises its timeouts for a related reason: followers timing out together would split the vote. See timeouts, retries and backoff with jitter.

Adaptive timeouts and phi accrual

A fixed timeout is right for one network on one day. Adaptive detectors keep a sliding window of recent inter-arrival times and derive the timeout from it, stretching when the network gets noisy and shrinking when it calms down.

The phi accrual failure detector (Hayashibara, Défago, Yared and Katayama, 2004) goes further: instead of a yes/no verdict it outputs a suspicion level, φ, that rises while the silence lasts:

• Keep the last ~1,000 inter-arrival times and fit a normal distribution.
• Take the time since the last beat and ask what fraction of past gaps were at least this long: the chance the beat is merely late.
• φ = −log10 of that chance. Suspecting at φ = 1 is wrong about 10% of the time, at φ = 2 about 1%, at φ = 3 about 0.1%, and at φ = 8 about once in a hundred million.

Each application picks its own threshold. Cassandra’s phi_convict_threshold defaults to 8. So does Akka’s cluster detector, with 1 s beats and a 3 s “acceptable heartbeat pause” for GC; its docs suggest 12 on cloud networks that hiccup more.

Worked example, mean gap 1.0 s: φ crosses 8 at mean + 5.6 standard deviations. On a steady link (σ = 0.1 s) that is 1.56 s; on a jittery one (σ = 0.4 s), 3.24 s. Same threshold, different timeouts, chosen by the data.

A number also lets one detector drive graded responses. The paper’s own example is a master with workers: at a low φ, stop sending the worker new jobs; at a moderate φ, resubmit its unfinished jobs elsewhere; at a high φ, remove it and free its resources. HDFS does a two-step version with fixed timers (stale at 30 s, dead at 10.5 min), and Kubernetes another (taint at 50 s, evict about 5 minutes later).

Who watches whom: topologies at scale

The arithmetic changes completely at a thousand nodes. Assume N = 1,000, each beating once a second.

Central monitor. Every node beats to one coordinator: 1,000 messages a second, trivial for one machine, with one consistent view of who is alive. But the monitor is a single point of failure and a hot spot, so it is usually replicated or backed by a consensus store, as with Kafka’s group coordinator, the HDFS NameNode and Kubernetes. Kubernetes keeps each heartbeat tiny, an update to a small Lease object, and posts the bulky node status only on change or every 5 minutes. With 5,000 nodes renewing every 10 s, that is 5,000 ÷ 10 = 500 small writes a second.

All-to-all. Every node beats to every other: 1,000 × 999 ≈ 1 million messages a second, 999 in and 999 out per node. Nothing central can fail, but load grows with N² and nodes can disagree about who is alive. The SWIM paper names this quadratic load as the reason classic heartbeating struggles beyond a few dozen members. A middle ground is to have each node watched by a few peers: Akka Cluster uses 9 by default.

Randomised probing (SWIM). Each period, every node pings one random peer. If no ack arrives, it asks k other members to ping that peer for it, which separates “the peer is down” from “my link to it is bad”. Only then is the peer suspected, and it gets a chance to refute that before being declared failed. Verdicts ride on the pings themselves. Per-node load stays at about one ping and one ack per period whatever the cluster size, and the expected time to first detection stays near e ÷ (e − 1) ≈ 1.6 periods. Cassandra uses the gossip form: once a second each node exchanges state with a random peer and runs its own phi detector. See the gossip protocol.

Heartbeats and leases

A lease is a heartbeat with a promise attached. The holder renews it periodically (a heartbeat) and promises to stop acting once it expires unrenewed; only then may anyone else take over. A plain heartbeat changes only the monitor’s mind; a lease makes the silent node stand down too.

etcd lease-bound keys, ZooKeeper sessions (ephemeral nodes vanish on expiry) and Kubernetes leader-election Leases all work this way. Each side times the lease on its own clock, so clocks must tick at similar rates, not agree; leave a margin for drift.

Leases do not stop zombies. A node frozen in a 40-second GC pause cannot notice its lease expired; it wakes and carries on. The fix is a fencing token, a number that grows with each grant and is checked by the storage it writes to; see distributed locking.

Going deeper: failure modes

GC pauses and flapping

A stop-the-world pause longer than the timeout produces a perfect false positive: the node is declared dead, its work moves, then it resumes and rejoins. Repeat that and the node flaps, each flap a rebalance. Remedies, best first: make the timeout longer than the pauses you actually see; use an adaptive detector with a pause margin; require several good beats before trusting a node again (hysteresis); and fix the pause itself.

Asymmetric and partial partitions

Networks fail in shapes: A reaches B, B can’t reach C, C reaches A, and every all-to-all monitor holds a different list of the living. SWIM’s indirect probes cut verdicts caused by one bad link; for decisions that must be unique, such as who leads, let a consensus majority decide.

Gray failure: the heartbeat is fine, the work is not

The most dangerous case is a node that heartbeats perfectly while failing every request. Microsoft Research’s gray-failure paper calls this differential observability: the heartbeat module runs, request handling is stuck, so the detector sees a healthy node and clients see a broken one.

Kafka met this head-on. Consumers once heartbeated from the processing thread, so a slow batch looked like a crash. KIP-62 moved heartbeats to a background thread, which created the opposite risk, so Kafka added a second clock, max.poll.interval.ms.

HDFS made the mirror-image choice: an optional lifeline message lets a DataNode too busy to process heartbeats promptly still prove it is alive. The lesson from both: decide whether your heartbeat proves the process is alive or the work is moving, and add a second signal for the other, such as “last offset committed”, “jobs finished this minute”, or a check through the real request path. Client-side error rates catch what neither does.

Failover storms after a network blip

When the monitor loses the network for 30 seconds, every node misses its beats at once. A naive monitor declares the fleet dead and starts a mass failover far worse than the blip, then a reconnection storm. Kubernetes guards against this: by default it evicts pods from at most 0.1 nodes a second, slows down when 55% or more of a zone looks unhealthy, and if every zone looks unhealthy it blames its own connectivity and evicts nothing. AWS load balancers fail open in the same spirit, sending traffic to all targets when all fail their checks. Rate-limit failovers and jitter reconnects (retry storm).

Split brain

A false positive on a leader produces two leaders: the old one, alive behind a slow link, and the new one. Heartbeats cannot prevent this; the design around them must. Elect through a majority quorum so only one side can win, make leaders that lose the majority step down, and fence everything the old leader writes. Some high-availability clusters power off the suspect machine before promoting a replacement.

Piggybacking: any message is a heartbeat

Explicit beats only need to fill silence. Raft’s AppendEntries calls double as heartbeats, a ZooKeeper client sends a PING only when its session is otherwise idle, and SWIM carries membership changes on its pings. On a busy link, real traffic proves liveness for free.

Heartbeats, health checks and probes

These terms get mixed up in interviews. They differ in who asks, what is proven and what happens next.

Amazon’s guidance makes a good default: load balancers act automatically only on local checks (disk, process, memory); deeper dependency checks go to central systems that react slowly and with rate limits, so one failing shared dependency can’t fail every server at once. More in health checks done right.

Three designs, worked through

A delivery rider app: location pings as presence

A Swiggy-style delivery platform needs rider locations anyway, so the location ping is the heartbeat. 300,000 riders online, each pinging every 5 seconds, is 300,000 ÷ 5 = 60,000 messages a second; at about 200 bytes each, 12 MB/s. Store last_seen per rider with a short expiry, which acts as a lease.

Grade the response, because riders lose signal in lifts and basements: after 15 s of silence (three missed pings), stop offering new orders; after 2 minutes, flag the active order to operations and call the rider before reassigning it. Adapt the interval too: with 100,000 riders on a delivery at 5 s and 200,000 idle at 30 s, load drops to 20,000 + 6,667 ≈ 26,700 messages a second, 56% less. See design food delivery.

Chat presence: online and last seen

In a WhatsApp-style app, presence rides on the persistent connection (WebSockets and friends). TCP keepalive won’t do: on Linux it waits 2 hours (7,200 s) of idleness before probing. Send an application ping every 30 s or so and mark the user offline when the socket closes or two pings go missing; “last seen” is the last beat. At 50 million connected users that is about 1.7 million pings a second, so gateways write to the presence store only when state changes, and push presence only to users with that chat open. See design WhatsApp.

A job scheduler reassigning work

Workers heartbeat every 5 s with their running job IDs and a progress counter. The scheduler declares a worker dead after 30 s of silence, or stuck after 10 minutes without progress (the Kafka lesson). It reassigns jobs with an incremented attempt number that acts as a fencing token, so a zombie’s late result is rejected, and side effects carry idempotency keys because some jobs will run twice. See design a job scheduler and long-running jobs.

Cheat sheet

How to say it in the interview

“Each worker heartbeats to the coordinator every 3 seconds, and I declare it dead after about 10 seconds of silence: three missed beats plus delay. Shorter would fail over faster but turn GC pauses into false failovers, each costing a partition rebalance. The heartbeat runs on its own thread, so I also track progress: no committed offset in 5 minutes means stuck. Reassigned work carries a new epoch and storage rejects old epochs, so a worker that was only paused can’t corrupt anything. At thousands of nodes I’d move to gossip-style probing with a phi accrual detector.”

Why not just make the timeout really short? You can’t tell slow from dead. Any pause or loss burst longer than the timeout becomes a false failover, and each costs a rebalance, an election or a pointless data copy.

How would you detect failures in a 5,000-node cluster? Not all-to-all: about 25 million messages a second at one beat each. Either a replicated central service where each node renews a small lease (500 writes a second at 10 s renewals), or SWIM-style random probing with gossip, which keeps per-node load constant.

The heartbeats are fine but requests are timing out. What’s going on? Gray failure: the liveness signal and the work fail differently. Make liveness reflect progress, run health checks through the real request path, and watch per-node client error rates and latency.

What happens if a node you declared dead is actually alive? Split brain: two owners of the same work. Use majority quorums for decisions, leases that make the old owner stand down, and fencing tokens so storage rejects its writes; make it rejoin with a new incarnation number.

What is a phi accrual detector? It learns the distribution of gaps between beats and outputs φ, minus the log of the chance that the current silence is just a late beat. Pick a threshold such as 8; on a jittery network the effective timeout stretches by itself.

Where to go next

• Consensus and leader election: what happens after a leader’s heartbeats stop.
• Distributed locking: leases, GC pauses and fencing tokens.
• Gossip protocol: spreading membership and failure verdicts at scale.
• Service discovery: registrations that expire when heartbeats stop.
• Health checks done right: liveness, readiness and deep checks.
• Failover: turning a detection into a safe switchover.
• ZooKeeper and etcd: sessions, leases and watches in practice.
• Kafka: consumer groups, rebalances and their timeouts.

References and further reading

1. The φ Accrual Failure Detector (Hayashibara, Défago, Yared and Katayama, JAIST research report, 2004)dspace.jaist.ac.jpSuspicion as a number, φ = −log10 of the chance a late beat still arrives; the detection-time versus mistake-rate trade-off; a week of 100 ms heartbeats between Japan and Switzerland.
2. SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol (Das, Gupta and Motivala, DSN 2002)cs.cornell.eduWhy central and all-to-all heartbeating do not scale, and randomised direct plus indirect probing with constant per-member load.
3. Gray Failure: The Achilles’ Heel of Cloud-Scale Systems (Huang et al., HotOS 2017)microsoft.comDifferential observability: a heartbeat module that is healthy while request handling is stuck.
4. Apache Kafka documentation: consumer configskafka.apache.orgheartbeat.interval.ms (3 s), session.timeout.ms (45 s) and max.poll.interval.ms (5 min), and what happens when each expires.
5. etcd tuning guideetcd.io100 ms heartbeat and 1,000 ms election timeout defaults, sizing both from round-trip time, and slow disks causing missed heartbeats.
6. Apache ZooKeeper Programmer’s Guide: sessionszookeeper.apache.orgSession timeouts negotiated between 2 and 20 ticks, PINGs only when the session is idle, and ephemeral nodes deleted on expiry.
7. Kubernetes documentation: Nodes (heartbeats and the node controller)kubernetes.ioNode Lease heartbeats, the 5-minute eviction delay, eviction rate limits and the rule that stops evictions when every zone looks unhealthy.
8. Apache Hadoop hdfs-default.xmlhadoop.apache.orgdfs.heartbeat.interval (3 s), the 30 s stale-DataNode interval, the 5-minute recheck interval and the lifeline protocol.
9. Apache Cassandra documentation: Dynamo architecture (membership and failure detection)cassandra.apache.orgGossip once a second to a random peer, a phi accrual detector on every node, and UP/DOWN as local decisions.
10. Implementing health checks (Amazon Builders’ Library)aws.amazon.comLiveness, local and dependency checks, fail-open load balancers, cautious automation and zombie servers.