Two numbers: how much you lose, how long you’re down
RPO caps the recent data you can lose, measured in time. RTO caps the downtime. The business sets both; a drill measures what you get.
2.2M
orders lost with only a nightly backup
50 orders a second, disaster at 14:00, last backup at 02:00. Your real RPO is the age of your newest surviving copy.
The replica promotes in 2 minutes. Why is downtime still 44 minutes?
The sum
Downtime is detect + decide + recover + verify: 5 + 15 + 14 + 10. The largest term is a human declaring the disaster.
Four strategies: how much already runs in the DR region
Each row down turns work done mid-disaster into capacity you pay for daily. Pick the cheapest that meets your RPO and RTO.
All regions live. Still need backups?
Bad: No, every region is a copy
A bad migration reaches every region within seconds.
Good: Yes, locked and point-in-time
Only going back in time undoes a bad write.
Bad DELETE at 10:42? Go back to 10:41.
- Restore the 02:00 base backup
- Replay the archived WAL in order
- Stop at 10:41, pause and check
- Salvage later good writes from the damaged database
GitLab, 2017: the ways back failed one by one
Nobody owned the backups, so nobody tested them regularly. The best copy left was six hours old and took about 18 hours to copy back.
The interview line
Agree an RPO and RTO, pick the cheapest strategy that meets them, keep locked point-in-time backups, and prove it with drills.