300/s
ops on the busiest document
The system peaks near 1M ops a second, spread over 15M open documents. 300 ops cost 15 ms of CPU a second: one server per document is plenty.
Send state or intent?
Bad: Diff and merge
Asha bolds “at Wankhede” as Ravi replaces “Wankhede”. Intent is lost.
Good: Small ops, in order
Insert, delete, apply style. The document is the replay of its ops.
Ravi deletes “IPL ”. Where does Asha’s “2026” go?
Her insert at 4 was typed before his delete. Shift it over the delete, 4 − 4 = 0, and both screens read “2026 FINAL”.
Thirty people type at once. Every op can cross every other.
The trick
The server puts every op in one order, and each client keeps one op in flight. 30 editors become 30 two-person problems.
4×
more presence traffic than edits
Suppose 2M people share a document. Their cursors move twice a second, each move reaching three others: 12M messages a second. So cursors skip the log.
The owner freezes, then wakes up still thinking it’s in charge.
Its 10 s lease runs out and S2 takes over with epoch 8. The log refuses S1’s late epoch-7 write, so history never forks.
Failover without losing a keystroke
- Ack only after a three-zone log append
- The new owner loads snapshot plus log tail
- Clients resend unacked ops, same client_seq
- Duplicates get re-acked, never applied twice
Remember
A document is a log of small ops with one owner, fenced by an epoch. Transform late ops, append before you ack, snapshot every 1,000 ops, keep cursors off the log.