Who takes writes? Usually one leader, streaming its log.
One log means one order, so copies differ only in how far they’ve got. Let more nodes write and conflicts appear.
Assume 40,000 reads/s and 2,000 writes/s. Add replicas?
Bad: They add write capacity
Every replica applies all 2,000 writes/s.
Good: They add read capacity
At 10,000 per server, reads need four. Shard for writes.
One commit, three moments to say “OK”
Async replies after the local flush. Semi-sync waits for one follower’s disk. remote_apply waits until queries there see it.
≈ 1,000
acknowledged writes lost in one async failover
Clients heard “OK”, but the promoted follower has only what reached it. 2,000 writes/s × 0.5 s of lag.
Reading everything from the leader wastes your replicas. How does she see Flat 4B?
The trick
Carry the commit’s log position (LSN). The replica answers once it has replayed that far, or the leader does.
Leader dead? Each failover step has a trap.
- Detect: a short timeout mistakes a GC pause for death
- Elect: promote only by majority, or two leaders appear
- Fence: storage rejects the old leader’s older epoch
Run DROP TABLE. Every follower applies it in milliseconds.
Replication copies mistakes too. Keep point-in-time backups; a replica delayed an hour gives you time to stop replay.
Remember
Single leader; commits wait for any one of two followers; replicas serve reads. Read a user’s own data from the leader, or wait for their LSN. Fence on failover. Keep backups.