2.9M/s
replica reads and writes at peak
A million gets each ask two replicas; 300k puts go to all three. At 40k per node, surviving a zone outage takes 120 nodes.
Where does each key live?
Bad: hash(key) mod N
Add node 121 and 99.2% moves: 342 TB.
Good: 4,096 fixed partitions
Node 121 takes 0.8%, 2.9 TB, as whole partition files from many donors.
Two of three for writes, two for reads
With N = 3, W = 2 and R = 2, every read set overlaps every write set. The read sees v2 and repairs the stale copy.
Phone and laptop edit one cart at once, via different nodes. Last-write-wins would silently drop one.
The trick
Version vectors show neither came first. Keep both as siblings; the client merges them and writes back.
33×
rewrites of each byte by leveled compaction
Each node takes 15 MB/s of new data at peak. Compaction turns that into ≈ 500 MB/s of disk writes, so you pick the strategy per keyspace.
Two replicas delete a key. The third was down.
It returns. Repair sees one value and two blanks, and copies the value back. The deleted key rises: a zombie.
The fix: mark deletes, repair in time
- A delete writes a timestamped tombstone
- Keep tombstones through a 10-day grace period
- Merkle-repair every replica at least weekly
- Drop one only in a compaction with every older SSTable
Remember
Hash keys into 4,096 partitions, three replicas in three zones. Quorums of two overlap, version vectors keep concurrent writes, hints and Merkle repair heal failures, and each node is an LSM tree.