The clock starts at the last beat heard, not the crash
Beats every second, dead after 3 missed: a crash at 3.3 s is declared at 6.1 s. Detection ≈ missed beats × interval + network delay.
A beat is 4 seconds late. A crash, a GC freeze, or a dropped packet?
The catch
It can’t tell. All three are silence. A late beat is evidence, not proof, so speed costs false alarms.
What makes a detector patient?
Bad: More beats
10 beats × 100 ms = a 1 s timeout. A 1.8 s GC pause trips it.
Good: A longer timeout
Pause tolerance = the total timeout. Extra beats only absorb packet loss.
630 s
before HDFS declares a DataNode dead
Timeouts follow the action’s cost. An etcd election pauses writes about a second: act after 1 s. A dead DataNode means copying every block.
Phi accrual: one threshold, two timeouts
It learns past gaps and scores the silence: φ = −log10(chance the beat is just late). Threshold 8 means 1.56 s on a steady link, 3.24 s on a jittery one.
Kafka’s heartbeats were perfect. The consumer had been stuck for minutes.
A background thread sends the beats, so the session timer never fires. A progress clock does, after 5 minutes without poll().
“Dead” is a decision. When the node wakes, treat it as dead.
- Fence its writes: storage rejects old tokens
- Reject its old leases
- Make it rejoin with a new incarnation number
The line to say
A detector can be fast or right, not both. Size the timeout by a false alarm’s cost; fence anything declared dead.