One component’s failure is the next one’s fault.
A rounding bug makes payments send ₹99.99, not ₹100. The ledger accepts it, and a user sees a wrong balance.
Three copies. Fault-tolerant now?
Bad: Count the copies
Copies cover only assumed faults. A slow disk walks straight past.
Good: Name the fault model
Crash, slow or lying? Say which, then size the copies.
The worse the fault you assume, the more replicas you pay.
To survive one: 2 if you can always tell dead from slow, 3 with majority quorums, 4 if a node may lie.
A driver bug breaks a VM’s networking, but its heartbeats still arrive over local calls.
Gray failure
The detector sees healthy; customers see it broken. Recovery trusts the detector, so nothing starts until they complain.
Prefer a dead node to a limping one
- Timeouts on every call turn slow into failed
- Health checks walk the path users take
- Callers eject back-ends far slower than peers
- Self-checks exit on impossible state
3k/s
sent to a database that handles 2,000
Illustrative tatkal rush: 1,500 requests a second, each retried once. A 15-second stall ends; the retries keep goodput near zero. Cap retries at 10%: 1,650 fits.
ECC fixes more bit errors each week. A quorum answers with a replica dead. Users see nothing.
The slow leak
Your margin is shrinking silently. Alert on lost redundancy, not just on user-visible errors.
Remember
Name the fault model, then add redundancy, detect end to end, contain behind boundaries, recover with idempotent retries, and inject faults. Turn slow and gray into clean crashes.