Real-time connections: long polling, SSE and WebSockets

HTTP only answers when asked, so a server with news (a chat message, a driver’s position, a wicket) needs a trick to reach the client. Short polling asks repeatedly, long polling parks a request on the server until there is news, Server-Sent Events (SSE) stream events down one open response, and WebSockets turn the connection into a two-way pipe. The last two make each message cheap; the hard part is running millions of connections that never close.

Picture it first: token 47 at a food court

You order a dosa at a busy food court and get token 47. How do you learn it’s ready?

• Walk to the counter every two minutes and ask. Most trips are wasted, and on average you hear a minute late: short polling.
• Stand at the counter until the cook says “ready” or five minutes pass, then ask again. You hear at once, but you occupy counter space the whole time: long polling.
• Watch the display board. It lights up tokens as orders finish; if you look away, you check the last number shown: SSE.
• Use the intercom at your table. The kitchen calls you, and you can reply “extra crispy” on the same line: a WebSocket.
• Go home and get an SMS. Your app isn’t listening at all; the phone’s push service wakes it: a push notification.

Map it back: the counter is your server, each trip an HTTP request, counter space a held connection, the board’s numbering a sequence number, and the floor space where people can wait is your connection capacity.

The problem: HTTP only speaks when spoken to

A server cannot send a response nobody asked for, so on its own it cannot push (RFC 6202). Yet many products depend on the server knowing first: a chat message, a delivery partner’s position, an order going “out for delivery”, an LLM’s next token. Each technique trades two things: latency, how long after the event the client sees it, and idle cost, what it costs while nothing is happening, which with a million clients is most of the time.

Short polling: ask on a timer

The client calls GET /orders/8812 every T seconds and the server answers at once, news or not. It is plain HTTP: any server answers, caches work, nothing stays open.

The cost is arithmetic. 1 million clients polling every 5 s send 1,000,000 ÷ 5 = 200,000 requests a second. If an order changes about once a minute, 11 of every 12 polls come back empty: about 92% of the load says “nothing new”. At 800 bytes of headers per poll, that is 200,000 × 800 B ≈ 160 MB/s of headers.

An event lands at a random moment in the interval, so it waits T/2 on average and up to T, plus a round trip. Halving the delay doubles the load. Polling is still right when:

• Updates are rare or slow: a dashboard, or an export job’s status (see long-running jobs).
• Everyone reads the same thing. A live score is identical for ten million viewers. Serve it from a CDN with Cache-Control: max-age=2, and the origin sees about one request every 2 s per edge cache, however many clients poll.
• Empty answers are cheap: with an ETag, an unchanged poll gets a 304 and no body.

Add jitter to the timer so a million apps opened at 9 am don’t poll in lockstep.

Long polling: hold the request until there’s news

The client sends a request; the server holds it until an event arrives or a timeout fires, then answers; the client immediately sends the next one. The server always holds a request per client, so it can “push” by answering.

GET /orders/8812/events?after=41 HTTP/1.1
Host: api.example.com

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json

{"events":[{"seq":42,"status":"PICKED_UP"}],"next":42}

• A timeout of about 30 s. The wait must end before any proxy, balancer or NAT on the path gives up. RFC 6202 reports success with 120 s but calls 30 s the safer value; go higher and clients get a proxy’s 504 instead of an empty 200.
• A cursor closes the gap. Between a response and the next request, no request is parked. Events landing then must be buffered, so the client sends the last sequence number it saw (after=41) and gets everything newer. Without it, messages fall into the gap.
• Latency. About one network transit on average, but up to three (response, next request, response) when a second event follows the first.
• Idle cost. A million idle clients with 30 s timeouts make 1,000,000 ÷ 30 ≈ 33,000 requests a second, against 200,000 for 5 s polling.

RFC 6202’s catches: full headers on every message, often more bytes than the payload; held resources, since each waiting client pins a connection and a request, so a thread-per-request server with 200 threads stalls at 200 clients; one of the browser’s six or so HTTP/1.1 connections per origin tied up; and caches that must be told not to replay old answers.

Today long polling is mostly the fallback for networks that break everything else.

Server-Sent Events: one response, many messages

SSE, defined in the WHATWG HTML standard, is a long-poll response that never ends. The client makes one GET; the server replies 200 with Content-Type: text/event-stream and keeps writing events into the body:

retry: 5000

: heartbeat

id: 1042
event: score
data: {"match":"IND-AUS","runs":187,"wickets":3,"over":"31.4"}

id: 1043
event: wicket
data: {"batter":"No. 4","how":"caught"}

Each event is a few field: value lines and a blank line. data: carries the payload (always UTF-8 text, so binary must be encoded); event: names a type the browser dispatches to addEventListener("score", …); id: labels the event; retry: sets the reconnect delay in milliseconds. A line starting with : is a comment, and the standard suggests one every 15 s or so because some legacy proxies drop quiet connections.

const source = new EventSource("/matches/42/stream");
source.addEventListener("score", (e) => render(JSON.parse(e.data)));

Reconnect and resume are built in. When the connection drops, EventSource waits the reconnection time (a few seconds, or what retry: said), reconnects by itself and sends the last ID it saw in a Last-Event-ID header. If the server can replay from that ID, out of a log or a sequence-numbered table, clients resume with no gap and no client code. A 204 No Content tells the browser to stop retrying.

It is ordinary HTTP. Proxies, CDNs, auth middleware, HTTP/2 and HTTP/3 carry it, and it compresses like any response. Two things bite. A proxy or framework that buffers responses sits on your events (RFC 6202 notes that intermediaries may legally buffer a whole response). And over HTTP/1.1, MDN notes browsers allow only 6 connections per browser per domain, across all tabs, so a seventh tab’s stream hangs. Over HTTP/2 the streams share one connection, up to a negotiated limit (100 by default, per MDN).

It is one-way. The client sends with ordinary POST requests, often a fine split: Slack’s clients send messages through its web API and receive events over their socket.

EventSource is minimal. It takes a URL and a withCredentials flag: no request body, no custom headers. So it authenticates with cookies, or you call fetch(), read the body as a stream and parse the format yourself.

That second route is how LLM APIs stream: OpenAI’s API streams output over server-sent events when you set stream: true, and so does Anthropic’s, with named events such as content_block_delta, ping and error. Both take a JSON request body, which EventSource can’t send. The in-band errors teach a general lesson: once the 200 and the first tokens have gone out, the status code can’t change, so a failure mid-stream (Anthropic’s overloaded_error, an HTTP 529 when not streaming) must arrive as an event.

WebSockets: a two-way pipe that starts as HTTP

A WebSocket (RFC 6455) begins as one HTTP/1.1 request and then stops being HTTP. The same TCP connection becomes a channel where either side sends messages whenever it likes, framed in a few bytes each.

The opening handshake

The client sends a GET with Upgrade: websocket, Connection: Upgrade, a Sec-WebSocket-Key (a random 16-byte nonce in base64), Sec-WebSocket-Version: 13 and, from browsers, an Origin. A server that agrees answers 101 Switching Protocols with Sec-WebSocket-Accept: the key joined to the fixed GUID 258EAFA5-E914-47DA-95CA-C5AB0DC85B11, hashed with SHA-1 and base64-encoded. That proves the server really parsed a WebSocket handshake, so a confused HTTP server can’t be mistaken for one. It is not authentication.

Frames

• Headers of 2 to 14 bytes. Two bytes hold a final-fragment bit, a 4-bit opcode, a mask bit and a 7-bit length; payloads over 125 bytes add a 2- or 8-byte length, and client frames add a 4-byte masking key. A 50-byte chat message from a browser costs 2 + 4 + 50 = 56 bytes before TLS, against hundreds for a long-poll message.
• Opcodes: text (UTF-8), binary, continuation, and the control frames close, ping and pong. Control frames carry at most 125 bytes and can slip between fragments of a large message, so a ping never waits behind a 10 MB upload. A peer must answer a ping with a pong.
• Close codes: 1000 normal, 1001 going away (a server shutting down, a browser leaving the page), 1008 policy violation, 1009 message too big. 1006 means the connection died without a close frame; it never appears on the wire.

Why browsers mask every frame

During the protocol’s design, an experiment poisoned real caching proxies that didn’t understand Upgrade: a script made a fake upgrade to the attacker’s server, then sent bytes that looked like a GET for a popular script, and some proxies cached the attacker’s reply as that script for everyone behind them. So every client-to-server frame is XORed with a fresh, unpredictable 32-bit key chosen by the browser, and page scripts can no longer choose the bytes on the wire. Servers must reject unmasked client frames and never mask their own. Masking happens even under TLS, and it isn’t encryption: the key travels in the frame.

What WebSockets don’t do for you

• No reconnection. The browser API offers send() and close(), and fires a close event. RFC 6455 says to wait a random time before reconnecting (0 to 5 s is “reasonable”) and back off exponentially after failures; that code is yours.
• No resume, no acks. TCP acknowledges bytes to the operating system, not messages to your app, so frames written just before a connection dies can vanish silently.
• No backpressure in the browser API. MDN warns that a fast sender can fill a slow page’s memory.
• No scheduled heartbeat. Nothing sends pings unless you do, and page scripts can’t send them at all, so many apps add a heartbeat message of their own.
• No HTTP after the 101: no status codes, caching or per-message routing.

Subprotocols version your message format: the client offers Sec-WebSocket-Protocol: chat.v2, chat.v1 and the server echoes its pick. HTTP/2 has no Upgrade or 101, so RFC 8441 opens a WebSocket on one stream with an extended CONNECT, sharing the connection with other requests; RFC 9220 does the same for HTTP/3. Proxy and balancer support for these is patchier than for the HTTP/1.1 upgrade.

Side by side, and how to choose

A rule of thumb:

• Notifications, feeds, dashboards, live scores, order tracking, LLM output: SSE.
• Chat, collaborative editing, multiplayer games, trading screens: WebSockets, for frequent messages both ways.
• Rare updates, simple clients, hostile networks, or data everyone shares and a CDN can cache: polling.
• App in the background: mobile push (APNs, FCM), covered below.
• Server to server: webhooks.

Real products combine them: a socket in the foreground, push in the background, and a fetch of whatever was missed.

Newer options: WebTransport and WebRTC

WebTransport runs over HTTP/3, so over QUIC and UDP: independent streams with no head-of-line blocking between them, unreliable datagrams for data where only the latest value matters, backpressure, and connections that survive a switch from Wi-Fi to mobile data. You pay with an HTTP/3 server and a fallback for networks that block UDP (see TCP vs UDP). WebRTC data channels go peer to peer, for calls and games such as Zoom.

Scaling to millions of open connections

The protocol is the easy part. Each open connection is state pinned to one machine for hours, and that shapes the fleet.

What one connection costs

• Memory, not CPU. In 2012 WhatsApp held 2,277,845 TCP connections on one 24-core FreeBSD server with 42% of its CPU idle and 27 GB of about 103 GB of RAM free: at most (103 − 27) GB ÷ 2.28 million ≈ 33 KB per connection, kernel and app together. Discord ran up to 500,000 sessions on one Erlang VM.
• File descriptors. Every socket is one, and default limits suit ordinary servers; WhatsApp raised the kernel’s to 3 million files and 2.4 million sockets.
• Ephemeral ports. A proxy opening one upstream connection per client socket spends a source port on each: about 28,232 per proxy IP, gateway IP and port on Linux defaults, a limit long-lived WebSockets made Cloudflare plan around.
• Heartbeats. 10 million connections pinged every 30 s is 10,000,000 ÷ 30 ≈ 333,000 frames a second, just to say “still here”.

Sizing follows. 10 million concurrent users at 500,000 per node needs 20 gateways; at half capacity, so a dead node’s clients fit on the rest, 40. Memory: 10 million × 33 KB ≈ 330 GB fleet-wide, about 8 GB per node.

Split stateful gateways from stateless logic

Put a thin gateway tier in front that only terminates TLS and the socket, authenticates the handshake, tracks what each connection follows, and forwards messages in and out. Keep business logic in stateless services you deploy many times a day; every gateway deploy drops connections, so its code should be small and boring. Slack works this way: stateful gateway servers hold users’ WebSocket subscriptions in several regions near users, channel servers own channels on a consistent-hash ring, and messages reach users worldwide in about 500 ms.

Routing: which gateway holds Ravi’s socket?

1. Asha’s app writes a frame on her socket; the load balancer passes it to Gateway A, which holds it.
2. Gateway A forwards the message to any chat-service instance.
3. The chat service stores it as seq 42 of the conversation before delivering it.
4. The registry says Ravi is on Gateway B.
5. The service publishes to Gateway B’s topic on the backplane.
6. Gateway B, subscribed to its own topic, receives it.
7. Gateway B writes it to Ravi’s socket, and Ravi’s app acknowledges seq 42.

If step 4 finds no live connection, the message goes to the notification system for mobile push instead. The registry is just user → gateway entries in something like Redis, written on connect with a TTL that heartbeats refresh, and it doubles as presence.

The variant is a topic per user or channel: gateways subscribe to the topics their sockets care about, and services publish without a lookup, at the cost of millions of subscriptions churning (Slack’s gateways subscribe to the channel servers that own their users’ channels). Either way, fan out to big groups once per gateway, not per user: Discord found sending one event to 30,000 sessions one by one took 0.9 to 2.1 s, at 30 to 70 µs per send, and fixed it by grouping recipients by node. Fan-out patterns live in real-time updates and publish–subscribe.

Keeping connections alive: timeouts, deploys and storms

Idle timeouts and heartbeats

Every hop, whether balancer, proxy, NAT or mobile carrier, drops connections it thinks are idle. An AWS Application Load Balancer closes them after 60 s by default (1 to 4,000 s), and AWS advises sending at least a byte before each period ends; HTTP/2 PING frames don’t count. So heartbeat every 20 to 30 s with a WebSocket ping, a small app message or an SSE comment. Heartbeats also detect dead clients: TCP keep-alives wait two hours by default (see TCP vs UDP), and a phone in a tunnel never sends a close. After two missed heartbeats, close the socket and let the registry entry expire.

Stickiness and draining

A WebSocket or SSE stream stays on the gateway it first reached, even behind an L7 balancer (see load balancing: L4 vs L7). Balance by connection count at open, and since new gateways only get new connections, cap connection age at a few hours with jitter to rebalance.

A balancer’s deregistration delay waits for requests to finish (300 s by default on ALB), but a WebSocket never finishes. So a gateway drains itself: it fails its readiness check, then closes sockets with 1001 going away in batches over several minutes, and clients reconnect elsewhere and resume from their last sequence number. Rolling 40 gateways with a 5-minute drain each takes 40 × 5 = 200 minutes, another reason gateways should rarely change (see deployment strategies).

Reconnection storms

When a gateway crashes or an office’s Wi-Fi returns, every affected client reconnects at once, and each reconnect costs a TLS handshake, an auth check, re-subscription and a catch-up query. Slack found storms hurt because each reconnect downloaded a snapshot of the whole team; its edge cache, Flannel, served those boot requests near users for 4 million simultaneous connections. Discord saw reconnect bursts overload the process that owned its hash ring, and nearly 5 million sessions stampeding ten registry processes caused cascading outages until it capped calls in flight.

The defences:

• Jitter, then backoff. RFC 6455 asks for a random first delay and exponential backoff after failures. Size the window so clients ÷ window stays under capacity: 100,000 ÷ 20 s = 5,000 a second against 10,000. See backoff with jitter and retry storm.
• Cheap resume: send the last sequence number and fetch only the delta, never a snapshot.
• Admission control: cap concurrent handshakes per gateway and turn the excess away before TLS and database work.

Mobile networks and batteries

Phones switch networks and sleep, and a TCP connection dies with each switch, so mobile clients reconnect often. A background socket costs battery, and mobile operating systems restrict background apps; Android’s Doze mode even delays normal-priority push. So use the socket in the foreground. When it disappears and the registry entry expires, send a push through APNs or FCM; Google advises high priority for chat messages and food-delivery updates, so they can wake a dozing phone. When the app opens, it reconnects and fetches what it missed.

Going deeper: delivery, ordering, backpressure and security

Delivery guarantees over a socket

Out of the box a socket gives you at-most-once: a frame written just before the connection dies may never arrive, and nobody is told. For at-least-once:

1. Persist first, with a per-conversation sequence number and a unique message ID.
2. Push the message with both.
3. The client acknowledges, per message or by reporting its highest contiguous sequence number on reconnect.
4. The server replays everything after that number from the store.
5. The client deduplicates by message ID, since replays repeat messages.

Uploads mirror it: the client tags each message with an ID and retries until acknowledged, and the server deduplicates. That is idempotency on a socket; the theory is in delivery guarantees.

Ordering

One TCP connection keeps bytes in order, but a message crosses services, partitions and reconnects. Assign the sequence number in one place per conversation (a single writer, a Kafka partition per key or a database counter), and have the client hold back briefly on a gap (44 arrived, 43 didn’t) and request a replay.

Backpressure: the slow client

A phone on one bar of signal reads more slowly than you write, and without a limit its unsent messages pile up in gateway memory, times 500,000 connections. Discord saw the extreme: session processes waiting on timed-out calls let their queues balloon until the Erlang VM ran out of memory. Give every connection a bounded send queue, say 256 messages or 1 MB. When it fills, drop or merge what can go (only the latest location or typing indicator matters); if important messages back up, close the connection and let the client resume from its last sequence number. See backpressure and load shedding.

Security

• Authenticate the handshake. Browsers can’t add an Authorization header to new WebSocket() or EventSource, so use a session cookie or a short-lived, single-use ticket from an authenticated API, passed in the URL; long-lived tokens in URLs end up in logs. See authentication.
• Check Origin. Browsers don’t block cross-origin WebSocket handshakes the way CORS restricts fetch; they send the Origin, and RFC 6455 tells servers to answer an unexpected one with 403. Skip that while authenticating by cookie, and any page a logged-in user visits can open a socket as them: cross-site WebSocket hijacking.
• Re-check over time. A socket can outlive its token by days. Track expiry per connection, close with 1008 or demand a refresh, and push revocations to gateways.
• Cap sizes and rates. RFC 6455 warns of frames claiming 2⁶⁰ bytes. Limit message size (close with 1009), messages per second and connections per user and IP, and always use wss://.

Cheat sheet

How to say it in the interview

“Clients hold one connection each: SSE for one-way feeds, a WebSocket where the client talks often. They land on a thin, stateful gateway tier, about half a million sockets per node. The gateway authenticates the handshake, records user-to-gateway in a Redis registry and forwards inbound messages to stateless services. A service persists each message with a per-conversation sequence number, finds the recipient’s gateway and publishes to its topic; the gateway writes to the socket and the client acks. Heartbeats every 25 s keep us under the balancer’s idle timeout. Clients reconnect after a random delay and resume from their last sequence number, and users with no live socket get a push notification.”

Why not use WebSockets for everything? You must build reconnect, heartbeats, resume and acks; some proxies block the upgrade; and L7 routing and caching stop at the handshake. For one-way updates, SSE gives reconnect and resume for free over plain HTTP.

How does a message reach a user on a different server? A registry maps users to gateways. The sender persists the message, looks up the recipient’s gateway and publishes to that gateway’s topic on Redis pub/sub or Kafka. No live connection means mobile push.

What happens when a gateway crashes? Its clients see the connection drop or heartbeats stop, wait a random delay, reconnect to another gateway and send their last sequence number; the server replays the gap from the store. Jitter and admission control keep the wave from swamping the survivors.

SSE or WebSockets for streaming LLM output? SSE: tokens flow one way and each answer is one request and one stream, which is what OpenAI’s and Anthropic’s APIs use. Read it with fetch to POST a body, and expect errors as events once the 200 has gone out.

How do you make sure no message is lost? Persist before you push, number messages per conversation, have clients acknowledge or report their last sequence number, replay the gap on reconnect and deduplicate by ID. The socket and backplane are fast paths; the store is the truth.

Where to go next

• Real-time updates: fan-out patterns for pushing one change to many clients.
• Publish–subscribe: the backplane between your services and your gateways.
• Load balancing: L4 vs L7: why long-lived connections stay pinned, and how draining works.
• HTTP and HTTPS: HTTP/1.1 connection limits, HTTP/2 streams and idle-timeout races.
• TCP vs UDP: what WebSockets inherit from TCP, and the QUIC base under WebTransport.
• Webhooks: pushing events between servers instead of to browsers.
• Design WhatsApp and design Slack or Discord: these connections at full scale.
• Design ChatGPT: SSE token streaming in a full design.

References and further reading

1. RFC 6455 — The WebSocket Protocolrfc-editor.orgThe opening handshake and GUID, frame layout and opcodes, client masking and the cache-poisoning attack behind it, close codes, Origin checks and reconnect backoff.
2. RFC 6202 — Known Issues and Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTPrfc-editor.orgLong polling’s life cycle, latency of one to three transits, header overhead, held resources, proxy buffering, connection limits and the 30-second timeout advice.
3. HTML Living Standard — Server-sent events (WHATWG)html.spec.whatwg.orgThe text/event-stream format, the EventSource interface, reconnection time, Last-Event-ID, 204 to stop retries and the 15-second comment keep-alive.
4. Using server-sent events (MDN)developer.mozilla.orgThe six-connections-per-browser-per-domain limit for SSE over HTTP/1.1, and negotiated stream limits over HTTP/2.
5. Edit attributes for your Application Load Balancer (AWS)docs.aws.amazon.comThe 60-second default idle timeout (1 to 4,000 s), the advice to send a byte before it expires, and that HTTP/2 PING frames don’t reset it.
6. How Discord Scaled Elixir to 5,000,000 Concurrent Users (Discord, 2017)discord.comWebSocket sessions fanned out from guild processes, up to 500,000 sessions per VM, grouping sends by node, reconnect bursts and bounding in-flight calls.
7. Real-time Messaging (Slack Engineering, 2023)slack.engineeringStateful gateway servers that hold WebSocket subscriptions near users, channel servers on a consistent-hash ring, and messages sent over the web API.
8. Flannel: An Application-Level Edge Cache to Make Slack Scale (Slack Engineering, 2017)slack.engineeringWhy reconnection storms hurt when every reconnect downloads a full snapshot, and an edge cache serving 4 million connections.
9. 1 million is so 2011 (WhatsApp, 2012)blog.whatsapp.comOver 2 million TCP connections on one FreeBSD server, with its CPU and memory headroom and raised file and socket limits.
10. Streaming API responses (OpenAI)developers.openai.comLLM output streamed over server-sent events with typed delta events, plus a separate WebSocket mode.