600M
devices connected at the evening peak
Assume 30% of 2 billion users. At a million sockets a gateway, that’s 600 gateways: connections, not messages, size the fleet.
History store or post office?
Bad: Keep every message
100B × 300 B is 30 TB a day, about 11 PB a year.
Good: Delete after delivery
Only undelivered messages wait, up to 30 days: about 4 TB.
Three ticks, three different machines
One tick: the server stored it on replicas, before any push. Two: Bob’s phone saved it. Blue: Bob opened the chat.
Phones retry, so a message can arrive twice. Phone clocks drift, so they can’t order a chat.
The trick
One partition owns each chat. It drops repeated client message IDs and stamps seq = last + 1.
1 KB
uploaded per group message, not 1.5 MB
A full group is about 1,500 devices. With a sender key, Priya uploads one ciphertext and the server copies it into every inbox.
Bob walks out of his flat. Wi-Fi becomes 4G.
His old socket on gateway 7 goes quiet, not closed. He reconnects to gateway 12, then gateway 7’s late “disconnected” deletes the new route.
The fix: epochs, and the inbox as truth
- Each login gets a higher epoch
- Route updates are compare-and-set on epoch
- Delete inbox entries on device ack, not push
- A new session closes the old one
Remember
Thin gateways hold the sockets. One partition per chat dedupes, sequences and writes every recipient device’s inbox before the first tick. Delete on delivery; the server sees only ciphertext.