Every failover runs four steps: detect, decide, promote, redirect.
How long they take is your recovery time (RTO). Whatever the standby lacks when it takes over is your data loss (RPO).
46 s
of failed writes, with Patroni’s default timers
Detect 30 s + decide 10 + promote 2 (assumed) + redirect 4, worst case. Detection is two-thirds of it, and it’s a timer you chose.
Fail over at the first silence?
Bad: Tightest timeouts
GC pauses look like death. Needless failovers break connections and may discard async writes.
Good: Detect better, not faster
Several observers, consecutive failures; bigger moves wait longer.
12,000
payments acknowledged, then lost with the primary
Illustrative: a UPI-style ledger at 3,000 writes/s, async. Typical lag of 0.1 s risks 300. Dying primaries slow first; lag can hit 4 s at the crash.
The old primary wakes up still believing it leads.
Each promotion gets a higher epoch. Storage refuses any lower one, because a paused process can’t know it was paused.
2018. GitHub loses a link for 43 seconds. Orchestrator promotes West Coast primaries, exactly as configured.
The cost
Both coasts now held writes the other lacked. No failback, only repair: 24 hours 11 minutes degraded.
Test it, or it doesn’t exist
- Switch over on a schedule
- Force real failovers at a quiet hour
- Game days: kill the primary, isolate a zone
- Time each step; count missing writes
Remember
Detect, decide by quorum, promote and fence, redirect. RTO is their sum; RPO, the replication lag at the worst moment. Never two primaries; test it on a schedule.