Multi-region and active-active

Multi-region means running your system in two or more cloud regions, hundreds or thousands of kilometres apart, so that neither distance to users nor the failure of one region decides your latency and uptime. Active-active means more than one region serves live traffic at once. The servers are the easy part. The hard part is the data: which region may write which records, how fast changes cross the gap, and what happens to the writes that had not crossed it when a region dies.

Picture it first: a bank with three branches

A cooperative bank has branches in Mumbai, Delhi and Chennai. Each keeps a copy of the account book, and a courier carries changes between them every few minutes. The bank can run the book three ways:

• One head office. Only Mumbai changes balances. The others show balances from their slightly old copy and send deposits on to Mumbai.
• A home branch per account. Any branch shows your balance; only the branch where you opened the account can change it.
• Every branch writes. If you withdraw ₹5,000 in Delhi and your spouse withdraws ₹5,000 in Chennai before the courier arrives, both slips are real, and someone must decide the balance.

Then Mumbai floods. Delhi and Chennai must serve Mumbai’s customers too, so they need spare counters on a normal day, and whatever is still in the courier’s bag is missing until the water goes down.

Map it back: the branches are regions, the courier is asynchronous replication, its trip time is replication lag, the bag is the un-replicated tail you lose on an unplanned failover, and the spare counters are capacity headroom.

Why go multi-region, and why not

A region’s availability zones (AZs) are separate sites in one metro area, close enough for synchronous replication, so spreading across them survives a building-level failure. A second region guards against events that take out a whole metro or a cloud service’s regional deployment, and serves needs unrelated to failure. AWS’s multi-Region guidance names three reasons:

• Bounded recovery time. A 99.99% target allows 52.6 minutes of downtime a year. AWS notes that detection can take 5 minutes, engaging an operator 10 more, and recovering from one issue 30 to 45 minutes: most of the year’s budget in one incident. A second region lets you fail over in bounded time and debug later.
• Latency. Fibre costs about 10 ms of round trip per 1,000 km (the physics is in the CAP article). Mumbai to Northern Virginia is about 12,900 km, so at least 129 ms per round trip. A new HTTPS connection takes about three round trips (TCP, TLS 1.3, the request), so an Indian user of a US-only app waits about 390 ms before your server starts work.
• Data residency. The Reserve Bank of India, for example, directed in 2018 that payment-system data be stored only in India, though the foreign leg of a cross-border payment may also be kept abroad. Rules like this can force an in-country region and limit where it may replicate.

Against it: the same guidance says most workloads meet their goals with several AZs in one region, a multi-region architecture typically costs about twice as much, and a badly built one can lower availability, because replication, failover tooling and global routing are new things that fail. Multi-region is the last rung of the ladder, after multi-AZ, backups and a practised disaster recovery plan.

Four topologies

Every multi-region design answers one question: which region may write a given record right now?

Active-passive

One primary region serves all traffic; the standby receives replicated data and waits. A hot standby runs at full size. A warm one runs scaled down and must grow during failover, adding minutes and assuming capacity is available while everyone else fails over too. (Pilot light and backup-and-restore belong to disaster recovery.) You gain simplicity: one writer, no conflicts. You pay for an idle region, far users pay the full distance on every request, and the standby is the region you exercise least, so it is the one most likely to surprise you.

Read local, write global

Every region serves reads from a local replica; all writes go to one primary region, sent there by the application or forwarded by the database. It suits read-heavy products. You gain local reads with a single writer. You pay a cross-region round trip on far writes, and local reads are stale by the lag. The classic bug: a Singapore user renames their profile, the write goes to Mumbai, the page reloads from Singapore and shows the old name. Fix it with read-your-writes: read from the primary for a few seconds after a write, or return a version with the write and let the local replica wait until it has caught up.

Home region per user, tenant or shard

Each user, tenant or shard has a home region that owns its writes; other regions hold replicas for reads and failover. AWS calls this write partitioned and DynamoDB write to your Region; CockroachDB builds it in as REGIONAL BY ROW tables, where a hidden crdb_region column names each row’s home. You gain local writes for almost everyone and no conflicts, because each record still has one writer, and each region becomes a blast-radius boundary, the multi-region form of a cell. You pay with routing that must know every user’s home, awkward cross-home operations (a transfer from a Mumbai user to a Frankfurt user touches two owners), and a migration when a user moves for good.

Multi-writer

Any region writes any record, in one of two ways. Asynchronously, resolving later: writes commit locally, replicate in the background, and a rule settles collisions; DynamoDB global tables work this way by default. Synchronously, agreeing first: a write commits only after a quorum of regions agrees, as in Spanner, CockroachDB set to survive a region failure, and DynamoDB’s strong mode. The first gives local writes and pays in conflicts; the second gives one truth and pays a cross-region round trip per write.

Moving data between regions

Asynchronous replication: the lag is your RPO

Usually the writer commits locally and ships a change log (a write-ahead log, binlog or change stream) to the other regions. Lag is normally under a second to a few seconds, and it grows with write bursts, big transactions and network trouble.

On an unplanned failover, whatever the old primary committed but had not shipped is the un-replicated tail. At 3,000 writes/s and 0.8 s of lag, that is 3,000 × 0.8 ≈ 2,400 acknowledged writes the new primary has never seen. They may survive on the old primary’s disks, or, with DynamoDB MREC, replicate when the region returns, only to lose to any newer write made meanwhile. Reconciling them is a business decision, not a database feature: replay from an upstream log, quarantine the affected records, or accept the loss.

So alarm on lag. Aurora publishes AuroraGlobalDBRPOLag and DynamoDB global tables publish ReplicationLatency per region pair; Aurora PostgreSQL can even cap lag with rds.global_db_rpo, at the risk of holding up transactions on the primary.

Managed global databases

Aurora Global Database: read local, write global. One primary region writes; up to 10 read-only secondary regions receive changes through the storage layer, with lag typically under a second. A planned switchover waits for the target to catch up, so RPO is zero, and on recent versions it typically takes under 30 seconds. An unplanned failover does not wait: RPO is typically seconds, RTO minutes. Fencing writes on the old primary is best effort, so AWS warns of split brain; Aurora tries to snapshot the old volume so you can recover the tail, and AWS suggests a DNS cache TTL as low as 5 seconds.

DynamoDB global tables: multi-writer, with a mode fixed at creation. The default multi-Region eventual consistency (MREC) mode accepts writes in every region and replicates them typically within a second. Concurrent writes to one item are settled by last writer wins on an internal timestamp, and the overwritten write is not logged in CloudWatch or CloudTrail. Conditional writes check only the local copy, and transactions are atomic only in their own region. Multi-Region strong consistency (MRSC) replicates each write to at least one other region before acknowledging it, needs exactly three regions (three replicas, or two replicas plus a witness), gives zero RPO, and rejects a write to an item being modified elsewhere with a retryable ReplicatedWriteConflictException. It has no transactions or TTL, and writes pay a cross-region round trip. Global tables carry a 99.999% SLA, against 99.99% for one region.

Spanner multi-region: synchronous quorum. Two read-write regions hold two replicas each, and a third region holds a witness. A write commits when a replica in the leader region and any two of the other four voters agree; Google keeps voting regions within about a thousand miles to bound that round trip. Nearby replicas serve reads. Dual-region configurations inside Australia, Germany, India and Japan meet residency rules with zero RPO.

CockroachDB: synchronous quorum, chosen per table. SURVIVE REGION FAILURE needs at least three regions and adds at least the round trip to the nearest other region to every write; with three regions it raises each range from 3 to 5 replicas (2 + 2 + 1). Table localities choose REGIONAL BY TABLE, REGIONAL BY ROW, or GLOBAL for reference data that every region reads fast and writes slowly.

Conflicts: two regions, one record

With asynchronous multi-writer replication, a conflict happens whenever one record is written in two regions within the replication window. A family sharing one cart on two phones is enough.

The options, from cheapest to most careful:

• Last writer wins (LWW). Keep the later timestamp. It is free and deterministic, and it silently drops the other write. It is safe when each write sets the whole state (a watch position, a “delivered” flag) and fatal for read-modify-write: two regions each running balance = balance − 500 on ₹1,000 both write ₹500 after ₹1,000 was spent. And “later” is only as good as clock sync (clocks and ordering).
• CRDTs. Data types whose merge converges in any order, such as a counter kept per region and summed, or an add-wins set for carts. You pay with a limited set of types and growing metadata for removed elements.
• Application merge. Keep both versions and let code or the user decide, as Amazon’s Dynamo did for carts. The most flexible, and the most code.
• Avoid the conflict. One writer per record (a home region) or consensus. For money and stock, this is usually the answer.

For counted resources, escrow splits the quantity in advance. Of 1,000 seats, 600 sell in Mumbai and 400 in Singapore with local, strongly consistent writes, and a background job rebalances when one side runs low. AWS’s DynamoDB guidance splits a credit balance the same way, so if one region vanishes, the other keeps spending its half.

Global uniqueness: usernames, seats and IDs

Async multi-writer cannot enforce “only one”: both regions check, both see no owner, both commit. Instead:

• One owner per claim. A small strongly consistent registry, or hash each name to an owning region. Signup pays one cross-region round trip; nothing else does.
• Consensus. A unique index in Spanner or CockroachDB, or a conditional put on a DynamoDB MRSC table.
• Accept and repair. Rename the loser later. Fine for display names, not for login handles.

Seats and stock follow the same rule: home each show in one region or escrow its quantity (contention and double booking). For IDs, never share one auto-increment counter; use random UUIDs, or time-ordered IDs that embed a region or worker number (unique ID generator).

Routing users to regions

DNS routing. Route 53, for example, offers latency-based routing (the lowest measured latency from the user’s network, which can shift week to week), geolocation routing (by country or continent, stable), and weighted and failover policies. It locates users by the EDNS client-subnet hint when the resolver sends one and by the resolver’s address otherwise, so users of a distant resolver can be misplaced. Always set a default record. Netflix’s 2013 active-active setup used directional geo routing because latency routing “could cause unpredictable traffic migration effects”.

Health-checked failover. Health checks drop an unhealthy region from DNS answers, but not instantly: checks need consecutive failures, resolvers cache answers for the TTL (some clients for longer), and open connections stay where they are. AWS suggests 60–120 second TTLs on failover records and cutting load-balancer keep-alive (an Application Load Balancer defaults to 3,600 seconds) towards your recovery target, say 300 seconds.

Anycast. One IP address is announced from many edge sites, and internet routing delivers each packet to the nearest, so failover happens inside the provider’s network instead of in DNS caches. AWS Global Accelerator, for example, gives you static anycast IPs and steers new connections to healthy regional endpoints; global HTTP load balancers and CDNs work similarly.

Sticky home-region routing. The nearest region is not always the right one. A Mumbai-homed user in Berlin lands on the Frankfurt edge, which reads their home from the session token or a directory and forwards the request over the provider’s backbone. Netflix’s Zuul gateways did this for “mis-routed” requests so that a device session never spanned regions, either proxying them or redirecting the client.

Evacuation needs a switch outside the region

The switch that moves traffic, and write ownership, away from a region must not depend on that region.

• Use data planes, not control planes. Route 53’s DNS answers and health checks run on a distributed data plane that AWS designs for 100% availability; its API and console are a more centralised control plane, hosted in us-east-1. A runbook that edits records through the API may fail exactly when you need it. Prefer data-plane switches such as Route 53 Application Recovery Controller (ARC) routing controls: on-off switches served from endpoints in five regions, with safety rules such as “only one of active and standby may be on”.
• Check your own tools. If the deploy pipeline, feature flags, dashboards or operators’ sign-on live in the failing region, so does your ability to leave it.
• Drain before handing over writes. Stop writes in the old region, wait for them to arrive, then let the new one write. When the old region is healthy, DynamoDB’s guidance suggests waiting about 10 times the recent maximum ReplicationLatency.

AWS advises automating the failover action fully but leaving the decision to start it with a few named people, who weigh the likely data loss against the outage. An automatic trigger that flaps can turn a blip into an outage.

Capacity: surviving the loss of a region

If N regions share the load evenly and one dies, each survivor’s share rises from 1/N to 1/(N − 1) of the total. Every region must be able to carry 1/(N − 1), so on a normal day it runs at most (1/N) ÷ (1/(N − 1)) = (N − 1) ÷ N:

• 2 regions: 50% each, and the fleet is 2× the load.
• 3 regions: 66.7% each, fleet 1.5×.
• 4 regions: 75% each, fleet 1.33×.

A worked example: 90,000 requests/s across 3 regions is 30,000 each, and 45,000 each after a loss. Build every region for 45,000 and it runs at 30,000 ÷ 45,000 = 66.7% normally. At 500 requests/s per server, that is 90 servers per region, 270 in all, against 180 with no headroom.

Three things push the ceiling lower:

• Survivors need headroom too. To stay at or below 80% after a loss, three regions run at 0.8 × 66.7% ≈ 53% normally.
• Failover traffic goes to the nearest neighbour, not evenly. With a 50/30/20 split, if the 50% region fails over entirely to the 30% one, that region carries 80% of global load. Size for the worst case, or spread failover with weights.
• It arrives cold. Caches are empty and autoscaling lags. Netflix shifted traffic gradually in its 2013 drill so services could scale and caches warm, and its gateways could cap and shed traffic. Pre-scale, keep service quotas equal in every region, and shed load rather than collapse. Google advises keeping multi-region Spanner under 45% high-priority CPU per region.

Every region also stores a full copy and applies every write (DynamoDB’s replicated writes consume write capacity in each replica), over paid links: 5,000 writes/s of 2 KB is 10 MB/s, or about 864 GB a day to each other region, at your provider’s inter-region price.

Going deeper: how multi-region goes wrong

Split brain between regions. After an unplanned failover the old primary may still accept writes from clients that have not moved: stale DNS caches, open connections, a batch job with a hard-coded endpoint. Fence at more than one layer (database read-only, a region flag the application checks, a routing control), send writes through one global endpoint, and never let a recovered region write until it has rejoined as a replica.

Replication lag spikes. Lag is your RPO and your staleness, so a spike quietly raises both. A backfill can push lag from milliseconds to minutes; failing over then loses minutes of writes. Alarm on lag against the RPO and throttle bulk writers.

A “global” dependency that is really single-region. The most common failure. The app runs in three regions, but login calls an identity service in one, or every region reads flags and secrets from one place. AWS’s guidance: no cross-region dependencies on the request path, keys and secrets per region, and staggered certificate expiry so one renewal mistake cannot hit every region at once. Find these by cutting a region off in a test.

Cross-region chatty calls. A service calling a peer in another region pays the distance every time: 20 sequential calls from Mumbai to Frankfurt, at a physics floor of about 66 ms each, add 20 × 66 ≈ 1.3 s. This often appears after a partial failover, which is why AWS advises failing over a whole user journey together and Netflix ruled out cross-region calls on the user’s call path.

The failover nobody tested. In AWS’s words, “Having an untested recovery approach is equal to not having a recovery approach.” Untested failovers fail on boring things: a quota never raised, an expired certificate, configuration drift.

Testing and operating it

• Evacuate regions on purpose. Netflix’s 2013 Chaos Kong exercise moved most of its US traffic to us-west-2, held it for over 24 hours, then returned to a 50/50 split; its Split-brain exercise cut connectivity between regions to prove each kept serving while replication queued. Some regulated firms rotate their primary region for months at a time, and follow-the-sun designs switch daily.
• One runbook for drills and real events, automated end to end.
• Deploy one region at a time. Netflix waited many hours between regional deploys (deployment strategies).
• Watch each region from outside it, with synthetic checks from the other regions and alarms in every region.
• Watch the bill: inter-region transfer, replicated writes, idle standbys and drill time (cost-aware design).

Cheat sheet

How to say it in the interview

“First I’d ask why we need more than one region: far users, an RTO one region can’t meet, or residency. If it’s only uptime, multi-AZ plus a standby may do. For a global consumer app I’d home each user in the region nearest their signup. Their writes go there, the edge forwards travellers home, and each region replicates asynchronously to a partner, so a region loss costs at most the lag, which I alarm on. Usernames and the ledger go through a consensus store and pay a cross-region round trip. With three regions each runs at no more than two-thirds of capacity, and we drill evacuations regularly.”

How do you handle a user who writes in two regions? Prevent it: route every write for a user to their home region, so two devices in two countries still reach one writer. Where a region must accept them anyway, make writes safe to merge: whole-state writes under LWW, CRDTs for counters and sets, or version checks that reject stale updates. Never let LWW settle a read-modify-write such as a balance.

How do you guarantee a username is unique globally? Give each claim one owner: a strongly consistent registry (Spanner, CockroachDB, a conditional put on a DynamoDB MRSC table) or a region chosen by hashing the name. Signup pays one cross-region round trip. Async multi-writer cannot do it, because both regions’ checks pass inside the lag window.

What happens to in-flight writes when a region dies? With async replication, writes acknowledged but not shipped are missing from the survivors: 3,000 writes/s × 0.8 s of lag is about 2,400. They may return with the region, but anything rewritten meanwhile clashes, so I make writes idempotent and replayable from an upstream log and decide per data type whether to replay, quarantine or accept the loss. With synchronous replication, acknowledged writes are already in another region.

Active-active or active-passive for a payments ledger? Active-active for stateless services and reads, but exactly one writer per account at any moment. I’d keep each ledger in a home region with a synchronous replica in another region in the same country, say Mumbai and Hyderabad, about 620 km apart and so at least 6 ms of round trip, or use a consensus database spread over in-country regions. Either way RPO is zero and the data stays in the country. Async multi-writer with LWW is out: it can double-spend or drop entries.

Where to go next

• CAP theorem and PACELC: why cross-region consistency costs latency.
• Disaster recovery: RPO, RTO, backups and the four DR strategies.
• Cell-based architecture: cells, stamps and geodes.
• Data replication: leader-based and leaderless replication, and where lag comes from.
• Clocks, ordering and conflict resolution: last writer wins, version vectors and CRDTs.
• Domain Name System: resolvers, TTLs and caching.
• Contention and double booking: selling the last seat safely.

References and further reading

1. AWS Multi-Region Fundamentals (AWS Prescriptive Guidance)docs.aws.amazon.comThe three reasons to go multi-region, the 99.99% downtime arithmetic, the roughly 2× cost, cross-region dependencies, the us-east-1 Route 53 control plane, quota parity, one-region-at-a-time deploys and failover testing.
2. Amazon DynamoDB Developer Guide: How global tables workdocs.aws.amazon.comMREC versus MRSC: replication typically within a second, per-item last writer wins, local-only conditional writes and transactions, the three-Region MRSC rule, RPO and the 99.999% SLA.
3. Amazon DynamoDB Developer Guide: Using DynamoDB global tables (design guidance)docs.aws.amazon.comWrite-to-any, write-to-one and write-to-your-Region modes, evacuating a live or offline Region, the 10× ReplicationLatency wait, suppressed late writes and the split-balance example.
4. Amazon Aurora User Guide: Using switchover or failover in Aurora Global Databasedocs.aws.amazon.comSwitchover with zero RPO versus failover with an RPO of seconds, best-effort write fencing and split brain, the snapshot of the old primary, and the 5-second DNS TTL advice.
5. Google Cloud Spanner: Instance configurationscloud.google.comMulti-region and dual-region replica layouts, the write quorum, voting regions within about a thousand miles, zero-RPO dual-region configurations and the 45% CPU guidance.
6. CockroachDB documentation: Multi-Region Survival Goalscockroachlabs.comSurviving a region needs at least three regions, raises replication from 3 to 5, and adds at least one round trip to the nearest region to every write.
7. Amazon Route 53 Developer Guide: Best practices for Route 53 DNSdocs.aws.amazon.comData-plane health checks for failover, 60–120 second TTLs on failover records, and always setting a default for geo and latency routing.
8. Amazon Application Recovery Controller: Best practices for routing controldocs.aws.amazon.comRouting controls served from five regional endpoints, lower TTLs, and cutting the Application Load Balancer’s 3,600-second default keep-alive so clients actually move.
9. Active-Active for Multi-Regional Resiliency (Netflix Technology Blog, 2013)netflixtechblog.comNetflix’s us-east-1 and us-west-2 design: stateless services, no cross-region calls on the user path, Zuul re-routing and load shedding, staged deploys, and the Split-brain and Chaos Kong exercises.
10. Reserve Bank of India: Storage of Payment System Data (April 2018)rbi.org.inA real data-localisation rule: payment-system data must be stored only in India, with the foreign leg of a transaction also allowed abroad.