Real-time updates

Real-time updates is the playbook for any feature where the server learns something first and the user should see it within about a second: a chat message, a driver’s car creeping across the map, a wicket in a live match, the next word of an AI answer. It has two halves. The last mile gets an event from your servers onto the user’s screen, by polling, long polling, Server-Sent Events or WebSockets. Server-side fan-out gets the event from whichever server produced it to whichever server holds the recipient’s connection. Weak answers solve the first half and forget the second.

Picture it first: news of a late train

Your train is late and you are waiting at a big junction. You can find out when it will arrive in several ways.

1. Ask at the enquiry counter every five minutes. Usually there is no news, and real news reaches you two and a half minutes late on average.
2. Wait at the counter until the clerk has news. You hear it at once, then queue again; anything announced meanwhile is lost unless the clerk wrote it down.
3. Listen to the loudspeaker. News reaches you as it is announced, one way only.
4. Carry a walkie-talkie to the station master. Either side can talk, but the station must keep a handset charged for every passenger.
5. The control room hears the news, but you are on platform 6: someone must route the announcement to your platform’s speaker.
6. A numbered logbook lets a passenger back from the tea stall ask what came after announcement 117.
7. On a festival weekend the departure board refreshes every few seconds for 40,000 people.

Map it back: counter, waiting, loudspeaker and walkie-talkie are short polling, long polling, Server-Sent Events and WebSockets; the control room is server-side fan-out (a registry plus pub/sub); the logbook is sequence numbers and resume; the board is batched broadcast.

Spotting the pattern

Listen for these cues:

• “Live”, “as it happens”, “in real time”: chat and typing indicators, online dots, live comments and scores, a driver on a map, collaborators’ cursors, order tracking, auction bids, stock ticks.
• The server learns first, from another user, a matching service or an exchange feed, so the client cannot know when to ask.
• A freshness target of about a second.
• Many recipients per event: a group chat, a channel, a match with a million viewers. That is the fan-out half.
• Frequent client input: game moves, edits, cursors. That decides SSE versus WebSockets.

It does not apply when:

• Minutes of staleness are fine. A dashboard refreshed every five minutes wants a cached GET on a timer (scaling reads).
• The user is offline. No socket reaches a phone in a pocket; send a mobile push (APNs, FCM) through a notification system.
• The receiver is another company’s server. That is a webhook.
• One user waits for one rare result, such as a report. Polling is fine (long-running jobs).

The running example is a Slack-style team-chat app sold to large Indian companies. At the 11 am peak, 1,000,000 users are online and send 5,000 messages per second, each reaching on average 4 online members: 20,000 deliveries per second, so a user receives a message every 50 seconds on average. Targets: delivered in under a second, nothing lost, in order within a conversation.

Levels 0 and 1: polling and long polling

Level 0: short polling

The client asks on a timer, GET /messages?after=… every N seconds. Each request is ordinary stateless HTTP: any server can answer, and a lost response is fixed by the next poll.

Two formulas set the cost: requests per second = users ÷ N and average delay ≈ N ÷ 2. At N = 5 s the running example sends 1,000,000 ÷ 5 = 200,000 requests per second. A user gets a message every 50 seconds, so about nine polls in ten come back empty: 180,000 wasted requests per second, about 1.6 Gbit/s if headers and cookies cost 1 KB per exchange (an assumption). The 2.5-second average delay misses the target; polling every second meets it with 1,000,000 requests per second, 49 in 50 empty. Uber once found that 80% of the requests reaching its mobile API gateway were polls, draining batteries on patchy networks.

You gain stateless servers and no new infrastructure. You pay in empty requests and half an interval of delay. It breaks when the target drops below a few seconds for hundreds of thousands of users. Say: “Polling costs users ÷ interval and adds half the interval, so a one-second target means a million requests a second, nearly all empty.”

Level 1: long polling

The server holds each request until it has news or a timeout fires, and the client immediately asks again. RFC 6202 puts the average delay near one network transit, the worst case at over three transits (response, next request, next response), and suggests about 30 seconds as a safe hold because proxies cut longer requests.

Requests per second are at most users ÷ timeout plus deliveries: 1,000,000 ÷ 30 + 20,000 ≈ 53,000, a quarter of Level 0, with near-instant delivery. But every event costs a full HTTP exchange, so a user receiving typing indicators or a location every two seconds is back to polling. An event that lands between a response and the next request must wait in a buffer, so the client sends a cursor (?after=118) or loses it. And you hold a million open requests: the state of a persistent connection without its efficiency.

You gain near-instant delivery over plain HTTP. You pay a full exchange per event plus cursor bookkeeping. It breaks at several events per user per minute. Say: “Long polling is my fallback where streaming is blocked, not my default.”

Level 2: Server-Sent Events

With SSE the client makes one GET and the server never finishes the response: it writes text/event-stream events as they happen, and the browser’s EventSource hands each one to your code. Protocol details are in real-time connections; three features drive the design:

• Reconnect is built in. If the stream drops, the browser retries after its reconnection time (the server can set it with retry:); a 204 response tells it to stop.
• Resume is built in. Each event can carry an id:. On reconnect the browser sends the last one as Last-Event-ID, and the server replays from there.
• It is plain HTTP, through the same load balancers, auth and HTTP/2 multiplexing as your API. Send a comment line about every 15 seconds so proxies don’t close a quiet stream.

GET /stream HTTP/1.1
Accept: text/event-stream
Last-Event-ID: 42:118

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-store

retry: 3000

id: 42:119
data: {"conv":42,"seq":119,"text":"Standup moved to 11:30"}

: keep-alive

SSE is one-way; the client sends with ordinary POSTs. That covers a surprising share of “real-time” features: notifications, feeds, live scores, order tracking, and streaming an LLM’s answer token by token. It scales, too: LinkedIn held 100,000 SSE connections per node for messaging, and Uber’s SSE-based push platform reached 1.5 million concurrent connections and 250,000 messages per second.

The limits: payloads are UTF-8 text, so binary needs base64; over HTTP/1.1 browsers allow only 6 connections per domain across all tabs (HTTP/2 multiplexes, 100 streams by default); and each client message is a separate request. Uber moved to bidirectional gRPC streams in 2019 for instant acknowledgements and binary payloads.

You gain push latency with HTTP tooling and free resume. You pay with one-way, text-only traffic. It breaks when clients send often or need binary frames. Say: “Updates only flow down, so SSE is enough: sends are POSTs, and Last-Event-ID gives me resume for free.”

Level 3: WebSockets

A WebSocket starts as a GET with Upgrade: websocket; the server answers 101 Switching Protocols, and the TCP connection then carries framed text or binary messages both ways (RFC 6455), with a few bytes of framing each and Ping and Pong frames as keepalives. Use it when the client talks as much as it listens: chat, games, collaborative editors. The protocol is the easy part; interviewers probe the consequences:

• A dedicated gateway tier. Sockets terminate on servers that only hold connections and relay messages. They are stateful: restarting one drops everyone on it, so keep them thin and deploy them carefully. Slack runs such gateways in several regions so clients connect nearby.
• Load balancers and idle timeouts. An L4 or L7 load balancer must pass the upgrade through; the socket then stays pinned to one gateway until it reconnects. Every hop has an idle timeout: an AWS Application Load Balancer closes a connection after 60 seconds without data by default. Heartbeat every 20 to 30 seconds and treat two misses as dead (heartbeats).
• Per-connection costs. Each socket costs a file descriptor, memory and a load-balancer slot. LinkedIn hit the limits in order: a 30,000 open-file limit, then a 4 GB heap exhausted near 60,000 connections (about 20 KB per TLS connection), before settling at 100,000 per node on 16 GB. Discord ran up to 500,000 sessions per Erlang VM.

Sizing the running example. Plan 50,000 connections per gateway, half LinkedIn’s figure, leaving room for a failed neighbour’s users: 1,000,000 ÷ 50,000 = 20 gateways plus spares, each with about 1 GB of TLS state (50,000 × 20 KB). Heartbeats every 25 seconds are 1,000,000 ÷ 25 = 40,000 tiny frames per second in total, against Level 0’s 200,000 full HTTP requests.

You gain two-way, low-latency messages with tiny overhead. You pay with a stateful tier: connection limits, heartbeats, careful deploys, and no free resume. It breaks at the second gateway: Asha is on gateway A, Ravi on gateway B, and A has no idea where Ravi is. Say: “About 20 gateways at 50,000 connections each, with heartbeats every 25 seconds under the load balancer’s 60-second idle timeout.”

Choosing the last mile

Two newer options come up in follow-ups: WebTransport, over HTTP/3, with several streams plus unreliable datagrams for game state that is worthless when late; and WebRTC data channels, peer to peer after a signalling exchange that usually runs over a WebSocket. Neither changes the server-side half.

Level 4: fan-out between servers

Now answer “how does server B know?” with two pieces.

A connection registry. On connect, the gateway writes user → (gateway, connection) to a fast store such as Redis, with a TTL of two heartbeat intervals that each heartbeat refreshes. A clean disconnect deletes the entry; a crashed gateway’s entries expire. A phone and a laptop are two entries. The registry answers “is Dev online, and where?”: online dots, direct routing, and whether to send a mobile push.

A pub/sub bus. Each gateway subscribes to a topic per conversation, such as conv:42, while any of its users belongs to it. The message service publishes each message once; only subscribed gateways receive it and write it to their sockets (publish-subscribe). Slack’s design has this shape: a channel server, placed by consistent hashing on the channel ID, sends each message to every gateway subscribed to that channel.

The steps: (1) Asha sends; (2) the service numbers and stores the message; (3) it publishes once to conv:42; (4) subscribed gateways B and C deliver it, and D gets nothing; (5) Dev has no socket, so APNs or FCM wakes his phone. Step 5 matters because phones drop sockets whenever an app goes to the background.

Per-room or per-user topics? Per-user topics are simple and suit notifications, but a 10,000-member channel costs 10,000 publishes per message. Per-room topics cost one publish and at most one delivery per gateway. Many systems use both.

Which bus? Redis Pub/Sub is fast and simple, but its documentation is blunt: delivery is at-most-once, and a message missed during a disconnect is lost for good. (In a cluster, sharded Pub/Sub keeps each message on one shard.) NATS core behaves similarly. Kafka is durable, but its unit is a partition, not a room; it belongs behind the message service as the log, not as the per-room doorbell.

The throughput is modest: 4 online recipients spread over 20 gateways reach on average 20 × (1 − 0.95⁴) ≈ 3.7 gateways, so 5,000 messages per second become about 18,500 bus deliveries per second. The hard part is correctness.

You gain delivery from any producer to any connection. You pay with a registry and a bus to run, and the rule that the store, not the bus, is the truth. It breaks whenever a gateway restarts or the network blips: Redis drops those messages and nothing replays them. Say: “Store first with a sequence number, then publish once to the conversation’s topic. Pub/sub is a doorbell, not a mailbox.”

Level 5: never lose, never reorder

Sequence numbers and resume

Give every conversation a sequence number assigned by a single owner when the message is stored (route by conversation ID, as Slack’s channel servers do), so numbers are ordered and gap-free. Order only matters within a conversation (message ordering).

The client remembers the last number it applied. On reconnect it asks to “resume conversation 42 after 117”, and the gateway subscribes first so nothing new slips past, replays everything after 117 from the store, then streams live. A message can now arrive twice, so the client drops sequences it has already applied and fetches the gap if it sees a jump. The at-most-once bus becomes at-least-once delivery with dedupe, which users experience as exactly once (delivery guarantees). Sends mirror it: a client-generated message ID makes a retried send safe (idempotency).

def on_resume(conn, cursors):                 # cursors: {conv_id: last_seq}
    for conv in cursors:
        bus.subscribe(f"conv:{conv}", conn)    # 1. live first, so nothing slips past
    for conv, last in cursors.items():
        missed = store.read(conv, after=last, limit=500)   # 2. then replay
        if len(missed) == 500:
            conn.send({"type": "resync", "conv": conv})    # too far behind: client reloads
        else:
            conn.send_all(missed)

def on_message(m):                             # on the client
    last = cursors.get(m.conv, 0)
    if m.seq <= last:
        return                                 # duplicate: already applied
    if m.seq > last + 1:
        fetch_gap(m.conv, after=last, before=m.seq)
    apply(m)
    cursors[m.conv] = m.seq

A user in 300 conversations would send 300 cursors, so messaging apps often keep a per-user inbox with its own sequence, resumed with one number at the cost of one write per recipient (the inbox pattern). And not every event needs this rung: Slack does not persist typing indicators, because the next one replaces a lost one.

Uber’s SSE platform ran this loop in production: a client reconnecting with “last seen 2” told the server that 3 never arrived; a one-byte heartbeat went out every 4 seconds, with a reconnect after 7 silent seconds; and messages expired after a TTL of up to 30 minutes.

Reconnect storms

Resume makes reconnects safe; now make them cheap. When a gateway with 50,000 sockets dies, every client reconnects with a TLS handshake, auth, a registry write, subscriptions and a replay query. Suppose the resume path handles 5,000 per second (an assumption). If everyone retries at once and then every second, the backlog takes 10 seconds and 275,000 attempts to clear (50,000 + 45,000 + … + 5,000). A careless deploy does the same thing on purpose.

The defences:

• Jittered, capped exponential backoff. Spread retries over a window: 50,000 ÷ 30 s ≈ 1,700 per second (backoff with jitter, retry storms).
• Admission control. Gateways accept connections at a fixed rate and tell the rest when to retry (load shedding).
• Drain before deploying. Stop routing new connections to a gateway, then move its clients in batches: 50,000 over 10 minutes is about 83 per second (deployment strategies). Slack drains a failed region to the nearest healthy one the same way.
• Cheap reconnect work. After a session server restarted, Discord’s hash-ring lookups (500,000 sessions × 5 guilds × 12 µs) took about 30 seconds until a shared read-only copy of the ring cut them to 750 ms; Uber put Redis in front of Cassandra to absorb deploy and failover herds.
• Cap the replay. A client hundreds of messages behind gets a fresh snapshot.

You gain ordered delivery that survives crashes, deploys and flaky phones. You pay with a durable write per message, cursors, dedupe and backoff discipline. It breaks when one room has a million members online. Say: “Messages are stored with a per-conversation sequence; clients subscribe, replay after their cursor and drop duplicates; reconnects use full jitter, and deploys drain gradually.”

Level 6: broadcast to a million viewers

The top rung is one room with a huge audience: a cricket final with 1,000,000 viewers in one live-comments room and 2,000 comments per second at a wicket (illustrative). Sending everything is 2,000 × 1,000,000 = 2 billion sends per second, and nobody can read 2,000 comments a second. Change what you send before you change how.

In the order you would add them:

1. Sample or rank. Show each viewer about 10 comments per second (an assumption about what a screen can usefully show): 10 million sends per second.
2. Batch. One frame every 500 ms with up to five comments: 2 × 1,000,000 = 2 million frames per second, 100,000 per gateway across 20.
3. Coalesce state. Score, viewer count and leaderboard are state, not events: send only the latest value, as Uber collapsed repeats of a message type.
4. Fan out in tiers. Discord measured 30 to 70 µs per send; one event to a 30,000-member guild took up to 2.1 seconds, and a million sends at 50 µs would take 50 seconds. Its fix grouped recipients by node, one send per node. Generalise it: publisher → relays → gateways → sockets, so the publisher sends 2 batches × 4 relays = 8 messages per second.
5. Protect slow consumers. Bound each socket’s send queue; when a viewer lags, drop old comments or send a snapshot.
6. Degrade gracefully. Under load, cut frames from 2 per second to 1, then 0.2; drop reactions before comments, comments before the score.

When everyone sees identical data and a few seconds is fine, as with the score card, the cheapest broadcast is a JSON file behind a CDN with max-age=2 that clients poll: the origin sees about one request per edge location every 2 seconds (scaling reads). Polling is wasteful only when every answer differs.

You gain cost that grows with gateways and frame rate, not audience × events. You pay with sampled, lossy delivery, extra hops and relays to run. Say: “Nobody reads 2,000 comments a second, so I sample to about 10, send two frames a second, coalesce the score, and fan out through relays so the publisher sends 8 messages a second.”

Which classic problems use it

Most designs need a last-mile rung plus Level 4; the question is how far up to climb.

Common mistakes

• WebSockets for everything. Notifications, scores and token streams flow one way; SSE does them with plain HTTP and built-in resume.
• Forgetting the server-side fan-out. One WebSocket server on the whiteboard invites “what if the recipient is on another server?”
• Pub/sub as the source of truth. Redis Pub/Sub is at-most-once. Store first, then publish.
• No resume after reconnect. Every crash, deploy and train tunnel silently loses messages.
• Connection state only in gateway memory. Without a registry you cannot show who is online, route a direct message, or decide to push.
• Pinned connections with no draining. A restart drops 50,000 users at once and starts a reconnect storm.
• Ignoring idle timeouts. A quiet socket behind a 60-second idle timeout dies at second 60.
• A synchronous database write per location ping. At 125,000 pings per second (scaling writes), publish the latest position to the trip’s topic and persist in batches.

Cheat sheet

• Polling: requests per second = users ÷ interval; average delay ≈ interval ÷ 2.
• Long polling: requests per second ≤ users ÷ hold timeout + deliveries.
• Gateways = peak connections ÷ connections per gateway, plus spares.
• Heartbeat under half the shortest idle timeout on the path (60 s on an ALB by default).
• Reconnects per second after a crash ≈ dead gateway’s connections ÷ jitter window.
• Broadcast sends = viewers × frames per second, not viewers × events.

How to say it in the interview

“Users send and receive messages within a second, so I’ll terminate WebSockets on about 20 gateways at 50,000 connections each, with heartbeats every 25 seconds under the load balancer’s 60-second idle timeout. Gateways register users in Redis with a TTL and subscribe to a topic per conversation. The message service assigns the next sequence number, stores the message, then publishes once. Redis Pub/Sub is at-most-once, so a reconnecting client sends its last sequence number, the gateway subscribes then replays, and the client drops duplicates. Offline members get a mobile push. Reconnects use full jitter, deploys drain gradually, and a million-viewer room gets sampling, batching and relays.”

Why not SSE instead? For one-way updates I would use it. Chat with typing indicators sends constantly, so a two-way socket saves a full HTTP request per event.

How does a message reach a user on another server? The message service stores it and publishes to the conversation’s topic; every gateway holding a member is subscribed and forwards it. The registry tells us who has no socket at all.

What if Redis drops a pub/sub message? Nothing is lost: the message is already stored with a sequence number, so the client sees the gap, or resumes after reconnecting, and fetches it.

A gateway with 50,000 connections crashes. What happens? Its registry entries expire within two heartbeats. Clients reconnect with jitter over about 30 seconds, roughly 1,700 per second, resubscribe elsewhere and replay from their cursors. Users see a pause, not lost messages.

A cricket final puts a million viewers in one room. Now what? Sample to about 10 comments a second, batch into two frames a second, keep only the latest score, fan out through relays, bound each socket’s queue, and lower the frame rate under load.

Where to go next

• Real-time connections: the protocols behind the last mile.
• Publish-subscribe: topics and delivery semantics for the fan-out bus.
• Push vs pull architecture: the trade-off behind polling versus streaming.
• Stateful vs stateless design: why gateways are special.
• Load balancing: L4 vs L7: upgrades and long-lived connections.
• Heartbeats: detecting dead connections.
• Timeouts, retries and backoff with jitter: reconnecting without a stampede.
• Design a notification system: reaching users with no open connection.
• Scaling writes: the companion playbook for location pings and message storage.

References and further reading

1. RFC 6455: The WebSocket Protocolrfc-editor.orgThe HTTP Upgrade handshake answered with 101, two-way framing over one TCP connection, and Ping/Pong frames as keepalives and heartbeats.
2. RFC 6202: Known Issues and Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTPrfc-editor.orgLong polling’s header overhead, average one-transit and worst-case three-transit latency, held-request resources, and 30 seconds as a safe hold timeout.
3. WHATWG HTML Living Standard: Server-sent eventshtml.spec.whatwg.orgThe text/event-stream format, id and retry fields, automatic reconnection with Last-Event-ID, 204 to stop reconnecting, and a comment line every 15 seconds for proxies.
4. MDN Web Docs: EventSourcedeveloper.mozilla.orgThe limit of 6 SSE connections per browser and domain over HTTP/1.1, and negotiated stream limits (default 100) over HTTP/2.
5. Redis documentation: Pub/Subredis.ioAt-most-once delivery (a message missed during a disconnect is lost for good), Streams for stronger guarantees, and sharded Pub/Sub from Redis 7.0.
6. Elastic Load Balancing: Application Load Balancer attributesdocs.aws.amazon.comThe connection idle timeout: 60 seconds by default, configurable from 1 to 4,000 seconds; send data before it expires.
7. Real-time Messaging (Slack Engineering, 2023)slack.engineeringStateful gateway servers holding WebSockets, channel servers placed by consistent hashing (about 16 million channels per host), replacement in under 20 seconds, and delivery worldwide in about 500 ms.
8. How Discord scaled Elixir to 5,000,000 concurrent users (Discord)discord.comGuild fan-out costs (30–70 µs per send, up to 2.1 s per publish in a 30,000-user guild), grouping sends per node, 500,000 sessions per VM, and reconnect stampedes.
9. Instant messaging at LinkedIn: scaling to hundreds of thousands of persistent connections on one machinelinkedin.comSSE for messaging, 100,000 connections per node, about 20 KB per TLS connection, and the file-descriptor, heap and port limits hit on the way.
10. Uber’s Real-Time Push Platform (Uber Engineering)uber.comFrom polling (80% of gateway requests) to SSE with sequence numbers, 4 s heartbeats, message TTLs and dedupe, 1.5M connections and 250,000 messages/s, then gRPC streams.