Design WhatsApp

WhatsApp turns a phone number into a mailbox that follows you around. A message typed in a Bengaluru auto reaches a cousin in Toronto before the auto has moved a metre; the family group lights up with three hundred Diwali wishes at midnight; a voice note sent to a phone that is switched off waits until the phone comes back. The product is tiny — send, deliver, show two ticks — but it runs at a size few systems reach: more than two billion users, over 100 billion messages a day, and hundreds of millions of phones holding a connection open at once. Its twist is that the server is built to know as little as possible. Messages are end-to-end encrypted, kept only until they are delivered, and then deleted. In this problem you design that system, from the sockets up.

Understand the problem

A chat app sounds like create-read-update-delete on messages. It is not. The hard part is a delivery system over flaky mobile networks that promises “never lost, never shown twice, always in order”, at a scale where one percent of users is twenty million phones. And unlike Slack or Discord, WhatsApp is not a history store: the phone is the archive. The server is a post office that holds a letter only until it is collected.

Functional requirements

1. Users should be able to send a text message to another user and see it arrive within a fraction of a second when both are online.
2. Users should be able to message a group of up to 1,024 members, the limit WhatsApp has used since late 2022.
3. Users should be able to receive messages sent while their device was offline as soon as it reconnects. Following WhatsApp’s published policy, the server keeps an undelivered message for up to 30 days and then deletes it.
4. Users should be able to see whether a message was sent, delivered and read — one grey tick, two grey ticks, two blue ticks.
5. Users should be able to share photos, videos, voice notes and documents.

Non-functional requirements

• Scale. More than 2 billion users and over 100 billion messages a day. About 600 million devices connected at the evening peak (estimated below).
• Low latency. For an online recipient in the same region: p50 under 100 ms and p99 under 300 ms from tapping send to the message appearing. Across continents, add the network round trip of 100–250 ms.
• Durability until delivery. Once the sender sees the first tick, the message must survive any single server, rack or zone failure until every recipient device has acknowledged it or 30 days pass. After delivery, deleting the server copy is the design.
• No duplicates on screen, even though mobile retries are constant.
• Per-chat ordering. Messages in one chat appear in the same order on every device. There is no promise of order across different chats.
• Availability. 99.99% for sending and receiving, which is about 52 minutes of downtime a year. Secondary features such as last seen and typing indicators may degrade first.
• Privacy. End-to-end encrypted by default. The server stores only ciphertext, and only briefly.
• Kind to phones. One connection per device, compact binary frames and few wake-ups: battery and data matter on a ₹10,000 phone with a 1.5 GB-a-day plan.

Out of scope

• Voice and video calls. A different media path (SRTP, relays, jitter buffers) with its own design; see Design Zoom.
• Status, Channels and Communities announcements. These are one-to-very-many broadcasts, closer to a feed or to Slack and Discord than to private chat.
• Payments. A regulated system with a ledger of its own; see Design a payment system.
• Encrypted backups and contact discovery. Both important, both separable from message delivery.
• Spam and abuse detection. It matters, but with end-to-end encryption it works on metadata, rate limits and user reports rather than message content.

The set-up

Plan the approach

Split the system in two out loud: a connection plane of gateways that hold hundreds of millions of sockets and do nothing clever, and a message plane that routes, stores until delivery, and orders. Build each requirement on that skeleton, then spend the time on the deep dives.

Core entities

• User. A user ID, a phone number, a profile and privacy settings (last seen, read receipts).
• Device. One primary phone and up to four linked companions (laptop, browser, tablet), each with an ID such as 4471.0 for the phone or 4471.3 for a laptop. Each device has its own public identity key, signed prekey and stock of one-time prekeys, and a push token.
• Chat. A 1:1 chat ID derived from the two user IDs, or a random group ID with a member list. Each chat has a counter for its next sequence number.
• Message envelope. Chat ID, sender device, client message ID, server sequence number, server timestamp and an opaque ciphertext — one per recipient device in a 1:1 chat, one shared sender-key ciphertext in a group.
• Inbox entry. One per recipient device per envelope. It exists only until that device acknowledges it.
• Receipt. “Delivered” or “read”, for a chat, up to a sequence number.
• Session. Which gateway a device is connected to right now, and the connection’s epoch.
• Media object. An encrypted blob with an ID, size and SHA-256 hash. The decryption key is not stored with it; it travels inside the encrypted message.

As storage, that is five small tables, none of them a message history:

inbox     key (device_id, inbox_seq)   chat_id, chat_seq, sender_device, cid, ts, ciphertext
sessions  key device_id                gateway_id, epoch, connected_at      (memory, TTL)
chats     key chat_id                  last_seq, owner_epoch, members (groups only)
dedupe    key (sender_device, cid)     chat_id, chat_seq                     (TTL 7 days)
prekeys   key device_id                identity_key, signed_prekey, one_time_prekeys[]

API

There are two surfaces: frames on the persistent connection for everything time-sensitive, and a few HTTPS calls.

The persistent connection. WhatsApp’s own protocol is a compact binary encoding of XMPP-style stanzas inside a TCP connection encrypted with the Noise protocol framework. In an interview, a WebSocket carrying binary frames is a fine stand-in. The frames are shown as JSON for readability.

Alice’s phone sends a 1:1 message with one ciphertext per device — Bob’s two and her own laptop:

{
  "type": "msg",
  "cid": "3EB0A1F2C4D9",
  "to": "user:4471",
  "payloads": [
    { "device": "4471.0", "kind": "pkmsg", "ciphertext": "MwohBf9..." },
    { "device": "4471.3", "kind": "msg",   "ciphertext": "MwogQx2..." },
    { "device": "1029.2", "kind": "msg",   "ciphertext": "Mwoh9aK..." }
  ]
}

cid is the client message ID, generated once when the message is composed and reused on every retry. pkmsg marks a first message that also sets up an encryption session. The server answers with an ack that earns the first tick:

{ "type": "ack", "cid": "3EB0A1F2C4D9", "chat": "c:1029-4471", "seq": 812, "ts": 1793080800123 }

Bob’s phone receives its own envelope, and acknowledges its inbox position once the message is safely in its local database:

{ "type": "msg", "chat": "c:1029-4471", "seq": 812, "from": "1029.0", "cid": "3EB0A1F2C4D9", "box": 99120, "kind": "msg", "ciphertext": "MwogQx2..." }
{ "type": "ack", "box": 99120 }

Receipts travel the same pipe, and one frame covers every message up to a sequence number:

{ "type": "receipt", "kind": "read", "chat": "c:1029-4471", "up_to_seq": 812 }

HTTPS calls for slower or bulkier work:

GET  /v1/keys?user=4471            identity key, signed prekey and one one-time prekey per device
POST /v1/keys/prekeys              top up this device's one-time prekeys
POST /v1/media                     encrypted bytes in; media_id and download URL out
POST /v1/groups                    create a group; returns group_id
POST /v1/groups/{id}/members       add or remove members (admins only)

Back-of-envelope estimates

Show the arithmetic; these numbers decide the shape of the fleet. (For the method, see back-of-envelope estimation.)

Messages. 100 billion a day ÷ 86,400 seconds ≈ 1.16M messages/s on average. Evenings are busier than nights and festival midnights spike harder still, so plan for 3× at peak: about 3.5M messages/s.

Deliveries. One message becomes several deliveries: one per group member, and one per linked device. In 2014 WhatsApp’s servers took in 19 billion messages a day and sent out 40 billion, about 2.1 out for every one in. Use 2: 1.16M × 2 ≈ 2.3M deliveries/s on average, around 7M/s at peak. Receipts flow back at a similar rate once “read up to N” batching is counted, so the gateways move roughly 1.16M + 2.3M + 2.3M ≈ 6M frames/s on average, before acks and heartbeats.

Connections. In 2014, 147 million of 465 million monthly users were connected at the daily peak: 147 ÷ 465 ≈ 32%. Assume 30% of 2 billion: 600M concurrent connections, a little more with companion devices. WhatsApp showed 2.28 million connections on one box back in 2012; plan for 1 million per gateway to leave room for encryption, bursts and failover. 600M ÷ 1M = 600 gateways, about 800 with a spare zone’s worth.

Memory per connection. That 2012 box used about 35 GB of active plus 18 GB of wired memory for 2,277,845 sockets: 53 GB ÷ 2.28M ≈ 23 KB per connection, kernel buffers included. A million connections fit in about 23 GB.

Inbox storage. An inbox entry is about 300 bytes: around 100 bytes of encrypted text plus routing headers. Most entries live for seconds. Say 20% of deliveries wait for an offline device, for 8 hours on average (phones off overnight, mobile data switched off): 2.3M × 0.2 × 28,800 s ≈ 13 billion entries × 300 B ≈ 4 TB, or 12 TB with three replicas. Compare a design that keeps everything forever: 100B × 300 B = 30 TB a day, about 11 PB a year before replication. Deleting after delivery is what lets WhatsApp skip a message-history database altogether.

Inbox operations. Every delivery is one write and one delete: 2.3M × 2 ≈ 4.6M operations/s on average, around 14M at peak. That, not the size, is what sizes the inbox tier.

Media. Assume 5% of messages carry media, at about 200 KB after the app compresses it: 100B × 5% = 5B items × 200 KB = 1 PB uploaded a day. Each item is downloaded about twice: 2 PB ÷ 86,400 s ≈ 23 GB/s ≈ 185 Gbit/s of average egress. As a sanity check, WhatsApp reported 146 Gbit/s out on Christmas Eve 2013, with about a quarter of today’s users.

What the numbers decide:

• Connections, not messages, set the fleet size. A specialised, thin gateway tier sized by sockets and memory.
• Millions of short-lived inbox writes a second. Memory-first, partitioned, replicated storage; disk only for messages that have to wait.
• Terabytes, not petabytes. No history database — the biggest structural difference from Slack and Discord.
• Media is nearly all the bytes. It must never flow through chat servers: blob store plus CDN.

High-level design

Build the design one requirement at a time. By the end it works in one region at moderate traffic; the deep dives then scale it.

Users should be able to send a text message to another user

Start with five components:

• Clients. Each device keeps one persistent, encrypted connection open and a local database that is the real message archive.
• Chat gateways. They terminate connections, authenticate devices, decode frames and pass them on. They know nothing about chats.
• Chat service. The brain. It is partitioned by chat ID, so one partition owns each conversation. It de-duplicates, assigns sequence numbers, writes inboxes and routes.
• Inbox store. One queue per recipient device, on replicated storage. An entry lives until that device acknowledges it.
• Session registry. Which gateway each online device is connected to.

1. Alice’s phone sends the envelope over its open connection to gateway 7.
2. The gateway forwards it to the chat-service partition that owns chat c:1029-4471. The partition checks the client message ID against recent ones and assigns sequence number 812.
3. It writes one inbox entry per recipient device — Bob’s phone, Bob’s laptop and Alice’s own laptop — to replicated storage, and only then acks Alice: one tick.
4. It asks the session registry where device 4471.0 is connected: gateway 12.
5. It pushes the envelope to gateway 12, which writes it to Bob’s socket.
6. Bob’s phone saves the message locally and acks. The chat service deletes that inbox entry and sends a “delivered” receipt to Alice: two ticks.

Two choices are deliberate. Store before you push: the first tick promises the message will not be lost, and only a replicated write can keep that promise. One inbox per device: Bob’s phone reads instantly while his laptop sits closed for a week, so each device acks on its own.

Users should be able to receive messages sent while offline

If Bob’s phone is switched off, steps 1–3 happen exactly as before; step 4 finds no session. Two things then take over: a push notification to wake the phone, and an inbox drain when it reconnects.

• The push carries no message. It says only “you have something”, or carries a payload the app decrypts on the device. Apple and Google relay it, so it must not contain plaintext. It exists to wake the app, which then connects and pulls over its own encrypted channel.
• The drain is ordered and paged. The phone says “I have everything up to inbox position 99,119”; the server streams later entries in pages of, say, 100, and deletes each page once it is acked. A phone that was off for a week drains thousands of messages in a few seconds.
• There is one code path. Delivering to an online device is the same “send everything after the cursor” call, triggered by a new message instead of a reconnect. One path means one set of bugs.
• Expiry is a sweeper. Entries older than 30 days are deleted. The sender never gets a second tick; the message simply stays at one.

Push itself is a classic notification system: per-platform token management, collapsing many messages into one alert, and respecting the user’s mute settings.

Users should be able to message a group

Add a group service that owns each group’s name, admins and member list, and a cache of “member → devices” inside the chat service. The partition that owns the group assigns its sequence numbers, expands the member list into devices, writes one inbox entry per device and pushes to the ones that are online.

Membership changes — “Ravi added Asha”, “Dad left” — are written into the group’s own sequence like any other message, so every device applies them in the same order relative to the chat. Without that, a phone could show a message from Asha before the line saying she was added. With end-to-end encryption, Priya does not encrypt the message six or nine times: groups use sender keys, so she uploads one ciphertext that every member device can decrypt. That is a deep dive of its own.

Users should be able to see sent, delivered and read

The three ticks map onto three server events:

• ✓ sent: the server has durably stored the message and acked the sender.
• ✓✓ delivered: a recipient device has acked its inbox entry. The server sends a “delivered” receipt to the sender’s devices.
• Blue ✓✓ read: the recipient opened the chat. Their device sends “read up to sequence N”, which the server relays.

Receipts are tiny messages that go through inboxes too — Alice may be offline when Bob reads — and they are batched: opening a chat with 30 unread messages sends one “read up to 841”. With several devices, pick a rule: two ticks once any of Bob’s devices has it, while each device still acks its own copy. In a group, receipts go only to the sender, whose phone counts them, so a 1,024-member group never generates a million receipt messages. If Bob turns read receipts off, his phone simply never sends them. The full sequence is drawn in the delivery deep dive.

Users should be able to share photos, videos, voice notes and documents

Media must not go through the chat path: one 2.4 MB video would cost as much gateway bandwidth as 8,000 text messages. Following the scheme WhatsApp documents, the sender’s phone generates a fresh random key, encrypts the file, uploads the ciphertext to a blob store, and then sends an ordinary encrypted message carrying the URL, the key and a SHA-256 hash of the blob.

1. Alice’s phone compresses the video, encrypts it with a random 32-byte key and uploads the ciphertext in resumable chunks (see large file uploads), getting back a URL.
2. She sends a normal encrypted message with the URL, the key, the ciphertext’s SHA-256 hash and a tiny thumbnail.
3. Bob’s phone receives it through the usual path and shows the thumbnail.
4. When Bob taps it, his phone downloads the ciphertext from the CDN, checks the hash and decrypts.

Forwarding is cheap: a forwarded video reuses the same blob, URL and key, so a clip forwarded a million times during an IPL final is stored once — WhatsApp’s privacy policy mentions keeping forwarded media temporarily for exactly this. Unreferenced blobs are deleted.

You now have a working messenger: 1:1 and group chat, offline delivery, receipts and media. The deep dives are about what breaks at two billion users.

Deep dives

How do we hold 600 million open connections?

Polling every few seconds would flatten the battery and still add delay, so the server must push over a connection that stays open for hours. APNs and FCM exist to wake an app, not to carry chat traffic. At 600 million devices, the cost of one idle socket decides the size of the fleet.

Bad: a thread per connection. Blocking code is easy to write: read a frame, handle it, write the reply. But every thread needs a stack and scheduler state, and almost all of them are asleep at any moment. Even a modest 256 KB stack makes 2 million connections cost 512 GB of memory before a single message moves.

Good: an event-driven server. A thread per core watches hundreds of thousands of sockets with epoll or kqueue and runs code only when bytes arrive, so an idle connection costs a small structure plus its kernel buffers. WhatsApp did this with Erlang on FreeBSD, one lightweight Erlang process per connection. In 2012 a single 24-core box held 2,277,845 TCP connections with 41.9% of its CPU idle — after raising kernel limits such as kern.ipc.maxsockets to 2.4 million and kern.maxfiles to 3 million.

The catch appears when that same process also runs the chat logic. Every deploy restarts a million connections: a million handshakes, authentications and inbox syncs at once. Deploy the fleet twice a day and you have built your own retry storm.

Great: a thin gateway tier, separate from chat logic. Gateways do only a connection’s work: terminate TCP and the Noise encryption, authenticate the device, decode frames, enforce per-connection rate limits, track heartbeats, and forward frames to the chat service over a few multiplexed internal connections. They hold no message state, so a crash loses sockets, not messages. Business logic lives in the chat service, which you can deploy ten times a day without dropping a single connection. In 2014 WhatsApp ran about 150 chat servers at roughly a million phones each.

Three operational details make it work:

• Heartbeats sized for mobile networks. Carrier NAT boxes forget idle TCP mappings after a few minutes, while TCP’s own keepalive defaults to two hours. The app sends a tiny ping every few minutes when idle, and the gateway declares a connection dead after two missed pings. See heartbeats.
• Graceful drain. Before upgrading a gateway, tell its clients “reconnect elsewhere within a random 0–60 seconds”. A million reconnects become about 17,000 a second instead of one spike.
• Connection-level balancing. An L4 load balancer or a DNS list of gateway addresses; a client that fails tries another address with backoff and jitter.

The recommendation: event-driven gateways holding about a million connections each and deployed rarely, with all chat logic behind them. Size them on memory and handshakes per second, not on messages.

How do we find the gateway a device is connected to?

The chat service has an envelope for device 4471.0. Which of 800 gateways holds its socket?

Bad: broadcast to every gateway. Each gateway checks whether it has the device. That turns 2.3M deliveries a second into 2.3M × 800 ≈ 1.8 billion internal messages a second, 799 out of every 800 wasted.

Good: a session registry. When a device connects, its gateway writes device → (gateway, epoch) into a replicated in-memory store partitioned by user ID. On disconnect it deletes the entry, and a TTL refreshed by heartbeats cleans up after crashed gateways. It is small — 700M entries × about 100 bytes ≈ 70 GB — and its write rate is the login rate, a few hundred thousand a second at peak. The chat service caches reads.

The difficulty is staleness. When Bob walks out of his flat, his phone moves from Wi-Fi to 4G. The old TCP connection is not closed; it just goes quiet, and gateway 7 believes Bob is still there until heartbeats time out minutes later. Meanwhile Bob reconnects to gateway 12. Two things go wrong: messages pushed to gateway 7 vanish into a dead socket, and gateway 7’s eventual “Bob disconnected” deletes the newer route.

Great: connection epochs, with the inbox as the source of truth.

• Every connection gets an epoch, a per-device number that rises with each login. Registry writes are compare-and-set: a new route is accepted only with a higher epoch, and a delete only if its epoch still matches. Gateway 7’s late delete for epoch 41 is ignored because the entry already says 42.
• A push is a hint, not a hand-off. The inbox entry is deleted when the device acks, never when a gateway accepts it. Whatever disappeared into the dead socket is redelivered when Bob’s phone syncs. The route may be wrong; the inbox may not.
• One live session per device. When epoch 42 registers, the registry tells gateway 7 to close epoch 41, so a device never receives two copies of the stream.

A further step drops the separate store: partition the chat service by user ID with consistent hashing and let each user’s home partition hold their sessions in memory, rebuilt from reconnects when a partition moves. WhatsApp’s 2014 talk describes partitioned Erlang services along these lines.

How do we make every message appear exactly once?

Networks drop packets, phones die mid-send and servers crash. Exactly-once delivery over an unreliable network is impossible; exactly-once display is not. You get it from at-least-once delivery plus idempotent processing at both ends.

A message can be lost or duplicated in three places: between the sender and the server (the ack is lost, so the phone resends), inside the server (a crash before the write is replicated), and between the server and the recipient (the device’s ack is lost, so the server redelivers).

Bad: fire and forget. The phone shows a tick as soon as the bytes leave. Every lift ride and every metro tunnel silently loses messages.

Good: acks and retries at every hop. The phone keeps the message in a local outbox until the server acks; the server keeps the inbox entry until the device acks. Nothing is lost — but duplicates appear. Alice’s message was stored, the ack died in a tunnel, the phone resent it, and Bob sees “Running late” twice.

Great: at-least-once delivery, de-duplicated at both ends.

• The client message ID is born once. The phone generates cid when the message is composed and stores it in the outbox; every retry carries the same one.
• The server de-duplicates on (sender device, cid) with an insert-if-absent into a table that keeps entries for a few days. A repeat gets the original ack — the same sequence number — and nothing else happens.
• The recipient de-duplicates on (chat, seq). A redelivered entry is acked again but not shown again.

Where the first tick is earned matters just as much: ack only after the envelope is on two of three replicas in different failure domains. WhatsApp’s 2014 talk adds a trick for the write load — most messages are picked up very quickly, so its offline store used a write-back cache that synced to disk after a delay. Most deliveries then never touch a disk. You pay with a narrow window in which losing both in-memory replicas at once loses ticked messages, so put them in different zones. The vocabulary of guarantees is in delivery guarantees.

How do we keep messages in order within a chat?

Two messages in one chat can take different gateways, different threads and different retries, so arrival order is not send order. Every device must still show the same order.

Bad: sort by the sender’s phone clock. Phone clocks are wrong all the time: set by hand, drifting, or pushed forward to beat a game timer. If Alice’s phone runs two minutes fast, Bob’s reply sorts above her question.

Good: the server’s receive timestamp. Better, but “the server” is hundreds of machines whose clocks differ by milliseconds, and a retried message gets a later time than one sent after it — wrong in exactly the quick back-and-forth users notice.

Great: a per-chat sequence number from a single owner. Chats are partitioned by chat ID, so exactly one chat-service partition owns c:1029-4471 at a time. It assigns seq = last + 1 as part of the inbox write, and devices display by sequence. You gain one order per chat on every device, cheap gap detection (814 arrived but 813 did not — wait briefly, then ask for the range), and trivial sync (“everything after 811”). You pay by funnelling each chat through one owner. That is fine — even a busy 1,024-member group sends a few dozen messages a second — but it would be a hotspot for a million-member channel.

Two details make it robust:

• Ownership moves. After a failover the new owner may not know the exact last number. Make the sequence a pair, (owner epoch, counter): the new owner bumps the epoch, so its numbers sort after everything the old owner issued.
• Your own message comes first. When Alice and Bob type at the same moment, each phone shows its own message instantly, before the server has ordered anything. Either re-sort when the ack brings the real sequence number (the bubble jumps), or accept that two messages sent within the same second may sit in different orders on the two phones. Most messengers accept the second; say which you choose.

Encryption does not force an order: the Double Ratchet keeps keys for skipped messages, so out-of-order ciphertexts still decrypt. Ordering is purely a display problem. More in message ordering and clocks, ordering and conflict resolution.

How do we fan out group messages under end-to-end encryption?

A full group of 1,024 members, at about 1.5 devices each, is roughly 1,500 device inboxes. Encryption decides who pays for the copies.

Bad: pairwise encryption for every message. Priya’s phone encrypts each message separately for each of the roughly 1,500 other devices and uploads every copy: for a 1 KB message, about 1.5 MB per message over a patchy 4G connection. Fine for a 1:1 chat with a handful of devices; hopeless for a big group.

Good: sender keys, fanned out by the server on write. The first time Priya writes to the group, her phone creates a sender key — a random 32-byte chain key and a signing key pair — and sends it to every member device over the existing pairwise sessions. That is the expensive step, done once. After that, each message is encrypted once with a key derived from the chain, which ratchets forward, then signed and uploaded as a single ciphertext; the server copies it into each member device’s inbox. Priya uploads 1 KB and the server does about 1,500 inbox writes. This is the scheme WhatsApp documents for groups, and server-side fan-out like this is largely why its servers sent about two messages out for every one in.

Alternative: one log per group, read through cursors. Store each message once in a per-group log; each device keeps a cursor and pulls “everything after mine”. Writes drop from about 1,500 to one. But nothing can be deleted until the slowest cursor passes it — one member offline for 29 days pins a month of the group’s messages — and a reconnecting phone must check each of its groups instead of reading one inbox. Slack and Discord work this way because they keep history anyway; see Design Slack / Discord.

The hidden cost is a change of membership. When someone leaves, every remaining member discards their sender key and starts over, because the leaver holds the old ones. Each remaining sender then redistributes a new key pairwise to about 1,500 devices the next time they speak: up to 1,023 × 1,500 ≈ 1.5 million pairwise messages after a single departure. That cost grows with the square of the group size. It is one good reason private encrypted groups stay capped near a thousand, while broadcast features use a different design.

The recommendation: sender keys with fan-out on write for groups, pairwise for 1:1 chats. A busy group in the last over of an IPL final — 20 messages a second × 1,500 devices = 30,000 inbox writes a second — spreads across inbox partitions keyed by device, so no single partition melts.

How does end-to-end encryption work, and what can the server still see?

The server plays two parts: a directory of public keys, and a postman for envelopes it cannot open. WhatsApp builds on the Signal protocol:

1. Registration. Each device generates a long-term identity key pair, a signed prekey it rotates periodically, and a batch of one-time prekeys, and uploads only the public halves.
2. Starting a session (X3DH). To write to Bob, Alice’s phone fetches a prekey bundle for each of his devices: identity key, signed prekey and one one-time prekey, which the server hands out once and deletes. Three or four Diffie-Hellman calculations give a shared secret without Bob being online — the asynchronous start the protocol was designed for. Devices top up one-time prekeys when they run low.
3. Every message (Double Ratchet). Each message is encrypted with a fresh key from a chain that ratchets forward, and every round trip mixes in a new Diffie-Hellman exchange. Stealing today’s keys reveals neither yesterday’s messages nor, after the next round trip, tomorrow’s.
4. Several devices. Each device has its own identity key. The server keeps the list of devices per account, signed by the primary phone so that a compromised server cannot quietly add one. The sender encrypts separately to every device on both sides — client fan-out — and the server deletes each copy once its device has it.
5. Transport. Client-to-server traffic is wrapped again in Noise Pipes, hiding metadata from network observers.

You give things up, and an interviewer will ask which: no server-side search, no server-side history for a new device, no content-based spam filtering and no server-generated link previews. Metadata is still sensitive: who talks to whom, and when, deserves the same care as content. Keys and their storage are covered in encryption and key management.

The final design

Put the pieces together. The hot path is gateway, chat service, inbox and back out through a gateway; everything else either wakes a phone, hands out keys or moves bytes for media.

1. Connect. A device opens one Noise-encrypted TCP connection to a chat gateway picked by an L4 load balancer, authenticates with its key, and registers its session with a new epoch.
2. Send. The gateway forwards each frame to the chat-service partition that owns the chat. It de-duplicates on (device, client message ID) and assigns the next per-chat sequence number.
3. Persist. It writes one inbox entry per recipient device to the replicated inbox store, then acks the sender: one tick.
4. Route. It looks up each recipient device in the session registry and pushes through that device’s gateway. The device acks, the entry is deleted, and a “delivered” receipt flows back: two ticks. “Read up to N” receipts follow the same path.
5. Wake. A device with no session gets a content-free push through APNs or FCM; on reconnect it drains its inbox in order. Entries expire after 30 days.
6. Groups and keys. The directory supplies group members, their devices and public prekey bundles. A group message arrives as one sender-key ciphertext, and the chat service copies it into every member device’s inbox.
7. Media. Phones encrypt media, upload it through the media service to the blob store, and send the URL, key and hash in a normal message; recipients download the ciphertext from the CDN, check the hash and decrypt.

Check it against the requirements. About 600 gateways at a million sockets each hold 600 million connections at roughly 23 KB apiece. An online delivery is two gateway hops, one partition and one replicated in-memory write, which keeps the in-region p99 well under 300 ms. The first tick waits for a replicated write, so a ticked message survives a server, rack or zone failure; client message IDs and sequence numbers hide every retry; a per-chat sequence gives every device the same order. Deleting on delivery keeps the inbox to a few terabytes. And the server never holds a key that can read a message or a photo.

What interviewers look for by level

Follow-up questions

What if we add “online” and “last seen”?

Presence is the most-read and least-important data in the system, so make it cheap. Only a device with a chat open needs someone’s presence: when Bob opens his chat with Alice, his phone subscribes to her presence, and unsubscribes when he leaves. A separate partitioned presence service (WhatsApp’s 2014 talk shows one) keeps “online” and “last seen” in memory, fed by connect and disconnect events from the gateways, and pushes changes only to current subscribers. Debounce flapping so a phone switching networks does not blink offline, coalesce updates to one every few seconds, never store typing indicators, and check privacy settings at subscribe time. If presence fails, chats keep working and “last seen” goes blank.

What if a user links a new laptop — how does it get the history?

The server has no history to give. Following WhatsApp’s multi-device design, the phone encrypts a bundle of recent chats, uploads it as an encrypted blob and sends the key to the laptop over their end-to-end encrypted session; the laptop downloads it, decrypts it, stores the messages locally and deletes the key. Changing settings — archived chats, starred messages — sync through a server-side copy encrypted with keys only the user’s devices hold. New messages are then fanned out to the laptop like any device.

What if a whole data centre goes down?

Clients reconnect to gateways in another region with the same jittered backoff that tames reconnect storms. Inbox entries must survive, so replicate the inbox synchronously across zones within a region and asynchronously to a second region. A failover may redeliver some messages, which de-duplication hides, and at worst lose the last second of ticked messages not yet copied. Session routes need no replication: reconnects rebuild them. Chat partitions move to the surviving region and bump their owner epoch, so sequence numbers stay ordered. More in multi-region and active-active.

What if we need a broadcast to a million followers?

That is a different product — WhatsApp calls it Channels. Fan-out on write would mean a million inbox writes per post, and sender keys a rekeying storm on every unfollow. Store each post once in a per-channel log, let followers read from a cursor when they open the channel, push only a lightweight notification, and drop pairwise end-to-end encryption, since a public broadcast has no secret audience to protect. That design is much closer to Design Slack / Discord and to a news feed.

What if a phone stays offline for more than 30 days?

The sweeper deletes its inbox entries, the senders’ messages stay on one tick for ever, and the phone never sees them. Its stock of one-time prekeys may also have run out, so new senders start sessions without one, which X3DH allows with slightly weaker protection against replay. If the user reinstalled the app rather than just going quiet, their identity key has changed: contacts get a “security code changed” notice and their phones build fresh sessions before sending.

How to say it in the interview

“I’ll design a WhatsApp-style messenger: two billion users, about a hundred billion messages a day, end-to-end encrypted, with a server that keeps messages only until they’re delivered.

Connections dominate: 30% of users online at peak is 600 million sockets, about 600 gateways at a million each. Messages average 1.2 million a second, deliveries twice that, and because we delete after delivery the inbox is terabytes, not petabytes.

So I split a thin gateway tier — event-driven, a million connections a box, rarely deployed — from a chat service partitioned by chat. When Alice sends, the owning partition de-duplicates on her client message ID, assigns a per-chat sequence number, writes one entry per recipient device into a replicated inbox, and only then acks: one tick. It looks up Bob’s device in a session registry, pushes through his gateway, and when his phone acks, deletes the entry and sends ‘delivered’: two ticks. Read receipts are batched as ‘read up to N’.

Offline devices get a content-free push and drain their inbox on reconnect. Routes go stale when phones switch networks, so connections carry epochs, the registry uses compare-and-set, and the inbox — not the route — is the source of truth.

Order comes from the per-chat sequence, never phone clocks. Groups of up to 1,024 use Signal sender keys, so the sender uploads one ciphertext and the server copies it to every member device; the price is rekeying when someone leaves.

Encryption is the Signal protocol: public prekeys on the server let sessions start while the recipient is offline, and every device has its own keys. The server sees metadata but never content, so there is no server-side search or history. Media is encrypted on the phone and served from a blob store behind a CDN; the message carries only the URL, key and hash.”

Concepts used in this design

• Real-time connections: long polling, SSE and WebSockets — why the server needs an open connection to push.
• Load balancing: L4 vs L7 — spreading long-lived connections across gateways.
• Heartbeats — finding dead mobile connections before NAT boxes forget them.
• Consistent hashing — assigning chats and users to partitions.
• Delivery guarantees — the at-least-once delivery under the ticks.
• Effectively-once: idempotent consumers and the inbox pattern — de-duplication on client message IDs.
• Message ordering — one order per chat, not a global one.
• Clocks, ordering and conflict resolution — why phone clocks cannot order messages.
• Push vs pull architecture — fan-out on write for groups, pull from a log for channels.
• Encryption and key management — the key directory and per-device keys.
• Blob storage and CDN — the separate path for encrypted media.
• Design a notification system — waking offline phones.
• Design Slack / Discord — the sibling problem, where the server keeps searchable history.

References and further reading

1. 1 million is so 2011 (WhatsApp blog, 2012)blog.whatsapp.comOne FreeBSD + Erlang server holding 2,277,845 TCP connections with 41.9% CPU idle, plus the kernel limits that made it possible.
2. That’s ‘Billion’ with a ‘B’: Scaling to the next level at WhatsApp (Rick Reed, Erlang Factory SF 2014, slides)erlang-factory.com465M monthly users, 19B messages in and 40B out per day, 147M concurrent connections, about 1M phones per chat server, 230K peak logins/s, and a write-back cache for offline storage.
3. Making messaging interoperability with third parties safe for users in Europe (Meta Engineering, 2024)engineering.fb.comInfrastructure handling over 100 billion messages a day; an XMPP-based protocol over a Noise-encrypted persistent connection that the server pushes on.
4. WhatsApp Encryption Overview: technical white paperwhatsapp.comIdentity keys and prekeys, client fan-out to every device, sender keys with server-side group fan-out, encrypted media in a blob store, and Noise Pipes transport.
5. How WhatsApp enables multi-device capability (Meta Engineering, 2021)engineering.fb.comA phone plus up to four companion devices, an identity key per device, client-side fan-out, and messages not stored after delivery.
6. The X3DH Key Agreement Protocol (Marlinspike and Perrin, Signal)signal.orgPrekey bundles published to a server so a session can start while the recipient is offline; one-time prekeys deleted after use.
7. The Double Ratchet Algorithm (Perrin and Marlinspike, Signal)signal.orgA new key for every message, a Diffie-Hellman step every round trip, and support for lost and out-of-order messages.
8. WhatsApp Privacy Policywhatsapp.comMessages are deleted from servers once delivered; undelivered messages are kept encrypted for up to 30 days, then deleted.
9. Communities now available (WhatsApp blog, 2022)blog.whatsapp.comGroups of up to 1,024 users.
10. Two billion users (WhatsApp blog, 2020)blog.whatsapp.comMore than two billion users, with end-to-end encryption on every private message by default.