Disaster recovery

Disaster recovery (DR) is how you get a system back after an event that high availability cannot absorb: losing a whole region, or losing the data itself to a bad migration, a mistaken delete or an attacker. You plan it around two numbers, how much recent data you can lose (the RPO) and how long you can be down (the RTO), and you only know you have it once a drill has proved it.

Picture it first: a ledger, a carbon copy and a photo

A cloth wholesaler sells on credit, so his ledger is the business. Every entry goes through carbon paper, leaving a copy in a tray at the back of the shop, and every night his son photographs the day’s pages into the family’s cloud folder. Three bad days show three different jobs:

• The clerk falls ill. Another clerk picks up the pen within minutes and nothing is lost. That is high availability.
• A fire takes the whole market. The ledger and its carbon copy burn together; the photos survive. He is trading from a new shop in three days, but every sale since last night is gone. That is disaster recovery, described by how much was lost and how long trade stopped.
• An assistant writes wrong totals across ten pages. The carbon copy faithfully records the same wrong totals; only last night’s photos show the truth. That is why backups exist even when you have replicas.

Map it back: the clerk is a server or a zone, the carbon copy is replication, the photos are offsite point-in-time backups, the market is a region, “since last night” is the RPO and “three days” is the RTO.

HA, DR and backups do different jobs

High availability (HA) handles component failures inside a site, automatically, in seconds; you measure it in the nines. Disaster recovery handles losing a site, a region or the data; in the AWS whitepaper’s words, availability focuses on components of a workload, DR on discrete copies of the entire workload. Backups are point-in-time copies kept apart from the live system, the only protection against a delete, a corrupting bug or an attacker, which replication copies everywhere within seconds.

Multi-zone HA already covers many “disasters”: AWS places its Availability Zones far enough apart that most common disasters, such as a fire or a flood, should hit only one. Region-level DR is for rarer events, or for regulators. Backups belong in every design.

RPO and RTO, precisely

The AWS whitepaper defines both:

• Recovery point objective (RPO): the maximum acceptable time since the last data recovery point. In plain words, how much recent data you can lose, measured in time.
• Recovery time objective (RTO): the maximum acceptable delay between the interruption of service and its restoration. How long you can be down.

Both are objectives the business sets; what you achieve in a drill is a separate measurement.

What sets your RPO

Your actual data loss is the age of the newest surviving copy. Take an order service writing 50 orders a second, hit by a disaster at 14:00:

• Nightly backup at 02:00 only. You lose 12 hours: 12 × 3,600 s × 50 = 2,160,000 orders, about 2.2 million.
• Nightly base backup plus the database log archived at least once a minute. You lose at most a minute: 60 s × 50 = 3,000 orders.
• An asynchronous replica in another region, about 1 s behind. Roughly 50 orders acknowledged but never shipped.
• A synchronous replica. Nothing acknowledged is lost, but every commit waits for a round trip; light in fibre covers about 200,000 km a second, so each 1,000 km adds about 10 ms to every write.

RPO is bought with replication lag and backup frequency, and paid for in bandwidth, storage and, for synchronous replication, write latency.

What sets your RTO

RTO is not “how fast the database promotes”. It is a sum:

RTO ≥ detect + decide + recover + verify

The whitepaper says the same: with a one-hour RTO you must detect, notify, escalate, evaluate, declare a disaster and recover inside the hour. For the order service with a warm standby, a plausible budget:

• Detect, 5 min. Three failed one-minute health checks, then two minutes to page and acknowledge.
• Decide, 15 min. The incident lead confirms the whole region is impaired and declares a disaster.
• Recover, 14 min. Fence the old primary (1), promote the replica (2), scale the app tier from a quarter to full size (8), flip routing and wait out a 60-second DNS TTL (3).
• Verify, 10 min. Place and refund a test order, spot-check recent data.

Total: 5 + 15 + 14 + 10 = 44 minutes, inside a 60-minute RTO. The human decision is the largest term; the promotion is one of the smallest.

With backups only, rebuilding the stack from code takes, say, 30 minutes, and restoring a 2 TB base backup at an assumed 200 MB/s takes 2,000,000 MB ÷ 200 MB/s = 10,000 s ≈ 2.8 hours. Then every log segment since 02:00 must be replayed, which the PostgreSQL docs warn can take a while: four to six hours in all.

How the targets drive cost

Each step down in RPO or RTO moves money from storage into running capacity: live replicas for RPO, servers already running for RTO, since anything created mid-disaster is slow. Set targets from a business impact analysis: what an hour of downtime or lost data costs, and how likely each disaster is. AWS is blunt: if recovery costs more than the loss it prevents, don’t build it unless a regulator requires it.

What counts as a disaster

AWS groups disasters into natural events, technical failures and human actions such as misconfiguration. For design, ask a sharper question: does a second live copy save you, or do you need to go back in time?

Google’s SRE book calls offering replication in place of a backup a classic but flawed answer: synced replicas guarantee that a corrupt row or an errant delete reaches every copy. In a study of 19 data recovery efforts at Google, the most common user-visible losses came from software bugs deleting data or breaking references, and the hardest were low-grade corruptions found weeks to months later.

Four strategies, from cheap to instant

AWS describes four strategies in increasing order of cost and decreasing RTO and RPO. What separates them is how much of the workload is already running in the recovery region before anything goes wrong.

• Backup and restore. Only backups live there: data, plus the infrastructure-as-code templates, images and config to rebuild. At failover you build, deploy and restore; AWS warns that without infrastructure as code this may push you past your RTO.
• Pilot light. Data replicates continuously into live databases and object stores, and core infrastructure exists, but app servers are off or not deployed. It cannot serve a single request until you start and scale them.
• Warm standby. A scaled-down but fully working copy runs all the time and can take traffic at once; at failover you scale up and switch. Scaled all the way up it is a hot standby, and nothing needs launching.
• Multi-site active/active. Every region serves users, and losing one means routing its users elsewhere. It costs the most because writes in two regions can conflict, the subject of multi-region and active-active.

The same strategies also work between Availability Zones. Choose per workload, not per company: a payment ledger might get a hot standby, a catalogue a pilot light (its search index can be rebuilt), an analytics warehouse plain backup and restore.

Backups done right

Full, incremental and continuous

A full backup copies everything: simple to restore, heavy to take. An incremental copies only changes since the last backup: cheap, but a restore needs the whole chain, and Google’s SRE book notes that a three-year-old full plus daily incrementals is over 1,000 interdependent backups, each another chance to fail. A continuous backup ships the change log as it is written. And only a physical copy plus that log gives point-in-time recovery; a logical dump such as pg_dump cannot.

Point-in-time recovery

Every PostgreSQL change is first written to the write-ahead log (WAL), stored as 16 MB segment files by default. Continuous archiving copies each finished segment somewhere safe. To recover, you restore a base backup and replay archived WAL on top, stopping at any moment after the base backup finished. MySQL does the same with a full backup plus its binary log.

# On the primary (postgresql.conf)
wal_level = replica
archive_mode = on
archive_command = '/opt/dr/ship-wal %p %f'   # copy the segment off the host; exit 0 only if it really landed
archive_timeout = 60                         # close a segment at least once a minute

# On a fresh server, next to an empty recovery.signal file
restore_command = '/opt/dr/fetch-wal %f %p'
recovery_target_time = '2026-10-27 10:41:00+05:30'
recovery_target_action = 'pause'             # inspect, then run SELECT pg_wal_replay_resume();

Details from the PostgreSQL docs that matter:

• The archive command must tell the truth: PostgreSQL recycles a segment once it exits with zero.
• archive_timeout bounds RPO on quiet systems, but forced segments are still 16 MB each: up to 1,440 × 16 MB ≈ 23 GB a day at one a minute, before compression, which is why the docs suggest about a minute.
• Pause before you promote, so you can check the data and move the target if it is wrong.
• Recovery starts a new timeline, so you can retry a different point without overwriting history.

The 3-2-1 rule, locks and isolation

The classic rule, as CISA puts it: keep 3 copies of important data (the original and two backups), on 2 types of media, with 1 offsite. In the cloud, read “media” as a different storage service and “offsite” as another region and, crucially, another account.

Isolation matters because the disaster is often an attacker: CISA’s ransomware guide warns that many ransomware variants hunt for reachable backups to delete or encrypt them. Three defences:

• Immutable storage. S3 Object Lock keeps object versions write-once-read-many (versioning required). In compliance mode no user, not even the account’s root user, can delete a locked version or shorten its retention; governance mode lets specially permitted users bypass it, which is handy for testing.
• A separate account. AWS notes that cross-account copies help against insider threats and account compromise.
• Don’t replicate deletes. By default S3 replication does not copy delete markers, so a deletion in the source leaves the DR copy intact.

For retention, Google’s SRE book reports keeping 30 to 90 days for many services, because low-grade bugs are sometimes noticed months later.

Keys, config and the things you forget

A backup is also only as available as its key: deleting an AWS KMS key makes everything encrypted under it unrecoverable after a 7 to 30 day wait. Keep DR copies’ keys in the recovery region or account, and back up infrastructure as code, images, secrets and config too.

A backup you have not restored is not a backup

Google’s SRE book quotes an old saying: nobody wants backups; what people want are restores. Test full recovery continuously, and alert when a recovery stops reporting success, because you only know you can recover if you actually do.

A nightly restore drill can be small:

def nightly_restore_drill():
    started = now()
    backup = catalog.latest("orders-db")
    db = restore_into_isolated_account(backup)          # never into production
    ok = (
        db.scalar("SELECT count(*) FROM orders") > 0     # an empty backup is the classic failure
        and db.scalar("SELECT max(created_at) FROM orders") > backup.taken_at - minutes(5)
        and smoke_test(db)                               # place, read and refund a test order
    )
    metrics.gauge("dr.restore_minutes", (now() - started).minutes)               # real restore time
    metrics.gauge("dr.restorable_age_minutes", (now() - backup.taken_at).minutes)  # real RPO
    if ok:
        metrics.heartbeat("dr.restore_ok")   # page if no heartbeat for 36 hours
    db.destroy()

The alert fires on a missing success, not on an error message, which matters in GitLab’s story below.

Failing over a whole region

Failover covers promotion and fencing; this is the runbook around them.

1. Detect. Health checks that reflect what users experience page a human.
2. Declare. A named role decides against agreed criteria, such as “region impaired for 15 minutes, no provider estimate”. AWS urges caution with automatic failover, since a false alarm costs real data and downtime, and suggests manual initiation of fully automated steps: one button.
3. Fence. Stop the old primary accepting writes (revoke its credentials, cut off its app tier) so a half-alive region cannot build a second history.
4. Promote. Per the whitepaper, an RDS read replica takes a few minutes, including a reboot; an Aurora global database can promote a secondary region in under a minute.
5. Shift traffic. Clients follow within about one DNS TTL; Route 53’s docs call 60 or 120 seconds a common choice for failover records. Pooled connections ignore DNS until they reconnect.
6. Verify and reconcile. Run smoke tests, then deal with the tail that never replicated.

Fail over with the data plane, not the control plane

AWS splits services into a control plane, which creates and changes resources, and a data plane, the running resources doing their job. Control planes have more moving parts and lower availability targets, and are likelier to misbehave when the wider system is impaired, exactly when you need them. So AWS advises failing over with data-plane operations only: health checks and routing controls, not console edits; running capacity, not new instances (even Auto Scaling is control plane).

The Amazon Builders’ Library calls the goal static stability: keep working through an impairment without changing anything. Its zone example: with three Availability Zones, overprovision by 50%, so each zone carries 1/3 of the load with capacity for 1/2 and runs at (1/3) ÷ (1/2) ≈ 67%; losing one leaves two at exactly 100%. An active/active pair that must survive losing a region runs each at no more than 50%. Raise service quotas in the DR region ahead of time, too.

The un-replicated tail

With asynchronous replication, the old primary acknowledged some writes the replica never saw, so never simply reattach it when it returns. Instead:

• Extract the tail from its WAL, binary log or application logs and compare it with the new primary.
• Replay what is safe through idempotent paths and compensate for conflicts; for money, reconcile against the gateway’s or bank’s settlement records.
• Prevent ID reuse: after restoring an older copy, GitLab bumped every database sequence by 100,000.

Failback

Going home is a second, planned migration: rebuild the old region as a replica of the new primary, let it catch up, then switch in a quiet hour. Many teams simply stay put.

Testing: drills, game days and DiRT

A DR plan that has never run is a hypothesis. In the AWS whitepaper’s words, the only error recovery that works is the path you test frequently, so keep recovery paths few: untested ones rot as load grows, quotas lag and images age. Build up in steps:

• Tabletop exercises: walk a scenario on paper with engineers, leads and communications staff; cheap, and good at exposing missing owners.
• Automated restore tests: the nightly drill above.
• Single-service failovers: move one service to the DR region at a quiet hour, then back.
• Game days: a planned exercise that takes out a zone or a region and runs the real runbook with the real on-call team.

Google’s DiRT is an annual, company-wide, multi-day disaster recovery exercise mixing theoretical and practical drills, in which SREs cause real outages in production to check that systems react as expected and to find unexpected weaknesses. Google also regularly cycles power to entire data centre buildings to see whether users notice. Practice pays: when Gmail lost user data in February 2011 and restored it from offline tape, the team could estimate the restore time up front and recovered over 99% of the data before the estimate ran out, because similar restores had been simulated many times.

Chaos engineering is the continuous, smaller version: define steady state from output metrics, inject real faults such as crashed servers or severed links, and try to disprove that the steady state holds (Principles of Chaos Engineering).

Measure, don’t just pass. Record each drill’s actual RTO (fault to verified service) and RPO (age of the newest surviving write), phase by phase. AWS adds a telling signal: leaders declining to invoke DR while the RTO is at risk suggests the plan is inadequate or nobody trusts it.

Going deeper: GitLab’s lost six hours

GitLab’s postmortem of its 31 January 2017 outage is the best-known lesson in why backups count only once restored. Times are UTC, from GitLab’s own write-up.

GitLab.com ran one PostgreSQL primary and one hot-standby secondary for failover. Backups came in three forms: a daily pg_dump uploaded to S3, daily LVM snapshots used to refresh staging, and Azure disk snapshots, enabled for other servers but not the database hosts.

• About 17:20. An engineer takes a manual LVM snapshot of production for staging.
• Evening. Spam, plus a background job deleting a wrongly flagged account, push database load up.
• About 23:00. Replication breaks: WAL segments the secondary needed were already gone from the primary, and with no WAL archiving there was nowhere else to fetch them. Rebuilding means wiping the secondary’s data directory and running pg_basebackup, which seems to hang.
• About 23:30. An engineer, believing they are on the secondary, wipes the data directory on the primary. They stop within a second or two; about 300 GB are already gone.

Then the recovery options failed one by one. The secondary was wiped. The S3 bucket for pg_dump backups was empty: the job ran pg_dump 9.2 against a 9.6 database and errored, and its failure emails were rejected by the receiving server for lacking DMARC signing, so nobody knew. No disk snapshots existed for the database. The best copy left was the 17:20 snapshot.

Copying it off staging disks throttled to around 60 Mbps took about 18 hours; the database was back on 1 February around 17:00 to 18:00. Changes from 17:20 to the deletion at 23:30, roughly six hours, were lost: an estimated 5,000 projects, 5,000 comments and 700 new user accounts. Git repositories and wikis, stored separately, were unaffected. GitLab’s root-cause analysis ends at ownership: backups were not tested regularly because nobody was responsible for them.

Cheat sheet

How to say it in the interview

“HA handles a server or zone failing, automatically. DR handles losing a region or the data itself. Backups are the only undo for a bad write, because replication copies mistakes in seconds. For each workload I agree an RPO and RTO with the business; for orders, say one minute and one hour. Async replication to a second region gives seconds of RPO, and WAL archived to a locked bucket in another account gives point-in-time recovery. RTO is detect plus decide plus recover plus verify, so I keep a warm standby, fail over with capacity that already exists, and a human declares the disaster. Nightly restore tests and regular failover drills prove the numbers.”

What RPO and RTO would you target for a payments system, and why? For the ledger, effectively zero RPO inside a region: commit only once a synchronous replica in another zone has the write. Across regions replication is asynchronous, so I reconcile the few-second tail against the gateway’s and bank’s records using idempotency keys. RTO of 15 to 30 minutes with a hot standby, and a human declares, because a split-brain double debit is far worse than 20 minutes of “payment pending”.

Why isn’t replication a backup? Replication copies every change, including the bad ones: a DELETE without a WHERE, a corrupting migration or an attacker’s wipe reaches every replica within seconds. A backup is a point-in-time copy you can go back to, ideally locked and in another account.

How do you know your DR works? Only by doing it. Restores run automatically every night with checks and a success heartbeat; failovers run as scheduled drills that record actual RTO and RPO phase by phase. Numbers drifting past the objectives are bugs.

Active-active or warm standby? Active-active gives near-zero RTO, but you pay for full capacity everywhere and must handle conflicting writes. Warm standby is cheaper, keeps one writer and takes minutes to promote and scale. I’d pick it unless the business needs near-zero downtime or several regions for latency anyway. Both still need backups.

Should failover be automatic? Inside a region, yes: losing a zone is an HA event. Across regions, usually not, because a false alarm costs real data and downtime. I automate every step but start with a human decision against agreed criteria.

Where to go next

• Multi-region and active-active: routing users, homing writes and resolving conflicts across regions.
• Failover: promotion, fencing and split brain in detail.
• Availability and the nines: what HA buys, in minutes of downtime a year.
• Data replication: synchronous versus asynchronous replication, and where lag comes from.
• Write-ahead logging: the log behind crash recovery, replication and point-in-time recovery.
• Blob storage (S3): versioning, replication and object locks for backup storage.
• Online schema and data migrations: avoiding the bad migration in the first place.

References and further reading

1. Disaster Recovery of Workloads on AWS: Disaster recovery options in the clouddocs.aws.amazon.comThe four strategies, failing over with data-plane operations only, the case for manually initiated failover, promotion times, and why replication alone does not protect against corruption.
2. Disaster Recovery of Workloads on AWS: Business Continuity Plan (BCP)docs.aws.amazon.comThe RTO and RPO definitions, business impact analysis, and the rule that recovery should not cost more than the loss it prevents.
3. AWS Well-Architected Reliability Pillar: REL13-BP02 Use defined recovery strategies to meet the recovery objectivesdocs.aws.amazon.comThe RPO and RTO ranges AWS gives for backup and restore, pilot light, warm standby and active-active, and the pilot-light versus warm-standby distinction.
4. Static stability using Availability Zones (Amazon Builders’ Library)aws.amazon.comControl plane versus data plane, static stability, and overprovisioning three zones by 50% so losing one needs no new capacity.
5. Best practices for Amazon Route 53 DNSdocs.aws.amazon.comUsing data-plane health checks and routing controls for failover, and 60 to 120 second TTLs on records that must change quickly.
6. Data Integrity: What You Read Is What You Wrote (Google SRE book, chapter 26)sre.googleWhy replication is not recoverability, tiered backups and retention, continuous restore testing, and the Gmail restore from tape.
7. Lessons Learned from Other Industries (Google SRE book, chapter 33)sre.googleGoogle’s annual DiRT exercises, in which SREs cause real outages to check how systems and people respond.
8. Postmortem of database outage of January 31 (GitLab, 2017)about.gitlab.comGitLab’s own account of the accidental deletion, the backup procedures that failed, the 18-hour restore and the lost data.
9. PostgreSQL documentation: Continuous Archiving and Point-in-Time Recoverypostgresql.orgBase backups plus archived WAL, archive_timeout, recovery targets, timelines, and why pg_dump output cannot be replayed.
10. Amazon S3 User Guide: Locking objects with Object Lockdocs.aws.amazon.comWrite-once-read-many storage, retention periods, and compliance versus governance mode.