Design ChatGPT

A consumer AI chat assistant looks like a messaging app: you sign in, type into a conversation and an answer appears. Underneath, every answer is a 10-to-30-second stream of tokens from GPUs that cost more per hour than the rest of the stack put together, and that you cannot add in minutes when 9 pm in India meets the American working day. The words must reach a phone on a patchy 4G link, survive that link dropping, and stop when the user taps stop. Every turn, the conversation is rebuilt into a prompt that fits the model’s window; edits turn it into a tree; and free and paying users compete for the same hardware. You design the product around the model — conversations, streaming, storage, quotas and safety — and treat the GPU fleet as an internal service with a known interface and capacity.

Understand the problem

Weak answers draw a chat app with an API call in the middle. Say two facts early. A reply is a stream that lives for 10 to 30 seconds, generated on the scarcest, most expensive hardware you run. And that hardware cannot be scaled in seconds: new GPUs take months to buy, and spare ones take minutes to load a model. Everything else exists to feed those GPUs well and to protect them. How a replica batches requests is the inference service’s problem; who gets to use it, what goes into each prompt and what happens to the words afterwards is yours.

Functional requirements

1. Users should be able to send a message and watch the reply stream in token by token, and stop it at any point.
2. Users should be able to come back to their conversations: see a list, open one on any device and continue it.
3. Users should be able to edit an earlier message or regenerate a reply, and switch between the resulting versions.
4. Users should be able to choose a model their plan allows, with usage limits that differ between free and paid plans.
5. Users should be protected by safety filters on what they send and on what the model writes.

Non-functional requirements

• Time to first token (TTFT): p50 under 1 s and p95 under 2.5 s on the default model, for prompts up to about 8K tokens. Paid plans keep this at peak; the free plan may stretch to a p95 of 5 s.
• Smooth streaming: at least 20 tokens a second, with no mid-answer stall over about 1 s.
• Durability: acknowledged messages and finished answers are never lost; an answer cut short by a crash is marked as interrupted, never silently truncated.
• Availability: 99.95% for reading history and sending messages. When GPUs are saturated, generation degrades; the app stays up.
• Scale (assumed): 100M daily active users × 20 messages a day, with a peak 2.5× the average because India’s evening (8–11 pm IST, 14:30–17:30 UTC) overlaps the US working day.
• Fairness: paying users never queue behind free traffic, and no user or script takes more than a fair share.
• Cost: GPU time dominates the bill; keep it near a tenth of a cent per typical message (derived below).

Out of scope

• Model serving internals — batching, KV cache, parallelism: Design an LLM inference service. We call generate(model, prompt, max_tokens), get a token stream back, and know roughly what one replica sustains.
• Routing across external model providers: Design an LLM gateway.
• Training and fine-tuning. We only store the thumbs-up and thumbs-down.
• Sign-in and billing. A session resolves to a user and a plan (see authentication).
• File uploads, tools, voice and memory across chats — follow-up questions at the end.

The set-up

Plan the approach

Satisfy each requirement simply, then spend your time in the deep dives. Say the shape early: “stream gateways that hold client connections, a chat API over a conversation store, an orchestrator that owns every generation, and admission control in front of the GPU fleet.”

Core entities

• User. ID, plan, home region, and preferences such as the default model and whether memory is on.
• Plan and quota. Allowed models, the limits on each (messages per rolling window, tokens per minute), a scheduling weight and a context budget.
• Conversation. ID, owner, title, active_leaf_id, last-activity time, a rolling summary and a deleted flag.
• Message. ID, conversation ID, parent_id, role (user, assistant or tool), and content with attachment references. Parent pointers make a conversation a tree.
• Generation (a run). One attempt at an assistant message: model, status (queued, running, completed, stopped, failed, blocked), prompt, cached and output token counts, finish_reason (stop, length, user_stopped, moderation, error), timings and the replica used. A retry is a new run for the same message.
• Attachment (a blob with its extracted text and token count) and feedback (thumbs up or down, with an optional reason).

Keep message and generation apart: one is what the user sees, the other the work behind it, with its own retries and cost.

API

Sending a message returns the reply as a server-sent events stream on the same HTTP response.

POST /v1/conversations/c_7Hq2/messages
Authorization: Bearer <session token>
Idempotency-Key: 5f1c2a9e-0b7d-4c3e-9a51-2d8e6f4b7a10
Accept: text/event-stream
Content-Type: application/json

{
  "parent_id": "m_41",
  "content": "Plan 3 days in Goa for two, under ₹25,000",
  "model": "large"
}

parent_id places the message in the tree: the current leaf for a normal reply, or an earlier message’s parent for an edit. The reply is a stream of named events; real provider APIs also name their SSE events, but these names and fields are our own.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-store

retry: 2000

id: g_9k1:0
event: start
data: {"run":"g_9k1","user_msg":"m_42","reply_msg":"m_43","model":"large"}

id: g_9k1:1
event: delta
data: {"text":"Day 1: North Goa."}

id: g_9k1:2
event: delta
data: {"text":" Check in near Baga"}

: keep-alive

id: g_9k1:57
event: done
data: {"finish":"stop","usage":{"prompt":2140,"cached":1792,"output":418}}

Each id is the run plus a sequence number, so the client knows the last event it saw. The other events are notice (“switched to the faster model”), retract (a safety stop) and error. Before the stream opens, quota and capacity problems are plain 429 Too Many Requests responses with Retry-After; after, errors arrive as events.

To resume after a drop, the client names the last event it saw:

GET /v1/runs/g_9k1/stream
Last-Event-ID: g_9k1:31

The server replays everything after 31, then continues live. For a long-finished run it sends the final message, and when nothing is left it answers 204 No Content, which the SSE specification defines as “stop reconnecting”.

The rest of the surface is ordinary:

POST /v1/runs/g_9k1:stop                         stop generating; keep the partial answer
GET  /v1/conversations?limit=30&cursor=…         newest activity first → items + next_cursor
GET  /v1/conversations/c_7Hq2                    messages on the active branch + version counts
POST /v1/conversations/c_7Hq2/messages/m_42:regenerate    new reply to m_42, streamed
PUT  /v1/conversations/c_7Hq2/active_leaf        body: leaf_id — switch to another version
POST /v1/messages/m_43/feedback                  body: rating up|down, optional reason

Opening a conversation returns the active branch, with what the version switcher needs:

{
  "id": "c_7Hq2",
  "title": "Goa trip on a budget",
  "active_leaf": "m_43",
  "messages": [
    { "id": "m_41", "parent": "m_40", "role": "assistant", "text": "…" },
    { "id": "m_42", "parent": "m_41", "role": "user", "text": "Plan 3 days in Goa…",
      "versions": { "index": 2, "count": 2 } },
    { "id": "m_43", "parent": "m_42", "role": "assistant", "text": "Day 1: North Goa…",
      "run": { "model": "large", "finish": "stop", "output_tokens": 418 } }
  ],
  "next_cursor": null
}

An edit needs no endpoint: it is a POST …/messages whose parent_id is the edited message’s parent. One practical detail: the browser’s EventSource only makes GET requests without a body or custom headers, so web and mobile clients read the stream with their HTTP library, parse text/event-stream themselves and send Last-Event-ID on reconnect.

Back-of-envelope estimates

These are assumptions; say so. For calibration, an NBER working paper (Chatterji et al., September 2025) reported about 2.6 billion ChatGPT consumer messages a day in June 2025, and 18 billion a week from 700 million users by July 2025 — the same order as the 2 billion a day assumed here. (Method: back-of-envelope estimation.)

Traffic. 100M × 20 = 2B messages a day ÷ 86,400 s ≈ 23,000 messages/s, and × 2.5 ≈ 58,000/s at peak.

Tokens. Assume 2,000 prompt tokens (system prompt plus history) and 400 output tokens per request. Peak output: 58,000 × 400 ≈ 23M tokens/s. Peak input is 116M tokens/s, but most of each prompt repeats the previous turn’s.

GPUs. Assume the default model runs on 8-GPU replicas that each keep about 80 streams going at 30 tokens/s: 2,400 output tokens/s per replica, prompt processing included — the kind of figure the inference article derives from memory bandwidth and KV cache size. 23M ÷ 2,400 ≈ 9,600 replicas, about 77,000 GPUs at peak, before headroom.

Open streams. An answer lives about 1 s to the first token plus 400 ÷ 30 ≈ 13 s: 14 s. 58,000 × 14 ≈ 810,000 open streams. Cross-check: 810,000 ÷ 80 per replica ≈ 10,000 replicas.

Stream gateways. Flushing every 50 ms gives about 20 events a second per stream. At an assumed 10,000 streams per node (200,000 small writes a second), 810,000 ÷ 10,000 ≈ 81 nodes: about 120 with headroom to lose a zone.

Bandwidth. Assume a token is about 4 characters; its SSE event (id, event, JSON wrapper, blank line) is about 60 bytes, so framing is over 90% of the bytes. Even at one token per event, 23M × 60 B ≈ 1.4 GB/s, about 11 Gbps fleet-wide. Size connections and per-event CPU, not bandwidth.

Storage. A turn is about 0.5 KB of question, 1.6 KB of answer and 0.4 KB of metadata: 2.5 KB. 2B × 2.5 KB = 5 TB a day, about 1.8 PB a year; three replicas with an assumed 3× text compression keep it near 1.8 PB on disk. Attachments go to blob storage.

Database writes. Each turn writes the user message, the reply (once, at the end — never per token) and the conversation’s last activity: about 175,000 writes/s at peak, or about 58,000 × (1 + 2 + 1) ≈ 230,000 DynamoDB write units (1 KB each) spread over millions of partitions.

Token buffer. Appending every 5 tokens: 23M ÷ 5 ≈ 4.6M appends/s. Redis documents half a million to a million XADDs a second per machine with pipelining; at a cautious 150,000 per primary, about 32 primaries hold under 90 GB of live and recently finished logs.

Cost. At an illustrative $2.50 per GPU-hour (the inference article’s figure), a replica costs $20 an hour for 2,400 × 3,600 ≈ 8.6M tokens: about $2.30 per million output tokens. A 400-token answer costs about $0.0009, a tenth of a cent, and 2B answers about $1.8M a day. But a fleet sized for peak is only 1 ÷ 2.5 = 40% busy on average: run all 77,000 GPUs all day and it costs 77,000 × $2.50 × 24 ≈ $4.6M a day, about 0.23 cents per message, unless the quiet hours do other work.

High-level design

Build one requirement at a time. The result works at moderate scale; the deep dives make it survive the real numbers.

Users should be able to send a message and watch the reply stream in

Start with the simplest path that streams. A chat API checks the session, saves the message and hands the work to an orchestrator, which builds the prompt and calls the inference fleet. Tokens come back one at a time; the chat API writes each as an SSE event on the still-open HTTP response, and the orchestrator saves the full reply at the end.

1. The client posts the message with an idempotency key, asking for text/event-stream.
2. The chat API checks the session, writes the message and a queued run, and keeps the response open.
3. The orchestrator builds the prompt and calls generate.
4. Tokens flow back as delta events; the reply is saved and done closes the stream.

Why SSE? During a reply data flows one way, and the next message is a new POST anyway. SSE is plain HTTP, so it rides HTTP/2, passes proxies and CDNs, and reuses the API’s authentication; the first deep dive compares the options (see also real-time connections).

Name the flaw now: the answer lives inside one response on one server. When a phone switches from Wi-Fi to 4G in a Bengaluru auto-rickshaw, the answer is gone, though the GPU finished it.

Where does the first second go? Here is an illustrative p50 budget for a 2,000-token prompt off-peak.

Two stages decide whether the budget holds. The admission queue can swallow the whole second at 9 pm, hence the deep dive on GPU scarcity. Prefill is short only because most of the prompt sits in a prefix cache from the previous turn, which makes prompt layout a latency decision, not just a cost one.

Users should be able to come back to their conversations

Two reads matter: the sidebar (“my conversations, newest first”) and opening one (“every message, in order”). Shape the conversation store around them — messages grouped by conversation, plus a per-user index sorted by last activity and read with cursor pagination, so a conversation jumping to the top doesn’t shuffle the pages. The storage deep dive picks the database.

A title worker names new conversations asynchronously with a small model, so the sidebar shows “Goa trip on a budget” instead of “New chat” a moment after the first reply. And because all state lives on the server, a second device opening a conversation mid-answer just attaches to the same run’s stream.

Users should be able to edit a message or regenerate a reply

Model the conversation as a tree, not a list: every message points at its parent. Regenerating adds a new assistant child to the same user message; editing adds a new user child to the edited message’s parent, followed by a fresh answer. Nothing is overwritten. The conversation records one active leaf, and the thread you see — also the history the model reads next turn — is the path from the root to that leaf.

The version switcher (“‹ 2/2 ›”) is just the siblings in creation order; picking one moves the active leaf to the newest leaf beneath it. Edits are cheap inserts, history stays immutable (a thumbs-down still points at the answer that earned it), and two devices editing at once can only create two siblings. The price is rebuilding the path on every read — cheap, because a conversation is one partition read of a few hundred small rows.

Users should be able to choose a model their plan allows

Before every generation the orchestrator consults a plan and quota service: the allowed models and the limits on each — illustratively, 50 messages per three hours on the large model for a paid plan, far fewer for free — plus a tokens-per-minute cap, so a pasted 100-page document isn’t one cheap message. Counters live in Redis with the same logic as a rate limiter. Output length is unknown up front, so reserve the maximum and refund the rest afterwards — reserve-then-reconcile, as in the LLM gateway. Over the limit, prefer the smaller model with a notice, or show when the limit resets; return 429 with Retry-After only when neither works.

Users should be protected by safety filters

A safety service runs fast classifiers. The input check runs in parallel with prompt building, and a blocked message never reaches a GPU. Output is checked in small windows as it streams, because waiting for the whole answer defeats streaming. Moderation models return a flag and a score per category — harassment, self-harm, violence and so on — so policy can differ by category: a self-harm signal may call for a supportive answer with helpline numbers rather than a block.

You now have a working design. At the real numbers it breaks in four places: answers die with their connections, prompts grow without limit, the store is still a box, and nothing protects the GPUs from the 9 pm peak. The deep dives fix those, then handle failures and geography.

Deep dives

How do we stream tokens reliably to millions of clients?

Two problems: which transport carries the tokens, and how to stop generation depending on it.

Bad: wait for the whole answer, or poll. A blank screen for 13 s feels broken. Polling every 500 ms means 28 polls per answer — 1.6M mostly empty requests a second at peak — and jerky text. Long polling holds each request until there is new text, but still pays a full request, headers included, per chunk.

Good: WebSockets. One long-lived, two-way connection with tiny frames — but a raw pipe: you build framing, resume and credential refresh yourself, stateful connections complicate deploys, and some corporate proxies break the upgrade. A text answer’s only upstream message is “stop”, which can be a POST; WebSockets earn their place for voice.

Great: SSE over HTTP/2. An ordinary HTTP response of type text/event-stream, multiplexed with the app’s other requests and fine through CDNs and proxies once response buffering is off. The WHATWG specification supplies what you would otherwise invent: an id per event, a Last-Event-ID header on reconnect, and a retry field for the reconnection delay. It also suggests a comment line every 15 seconds or so, because some proxies drop idle-looking connections — handy while a reasoning model thinks.

SSE fixes the format, not the dropped connection. For that, look at where the generation lives.

Bad: the generation lives inside the HTTP request. When the client’s socket dies, the handler holding it gives up, so a metro tunnel loses the answer and the GPU time already spent on it — and every deploy of the API tier kills up to 810,000 answers in flight.

Good: sticky reconnects to an in-memory buffer. Each server keeps its runs’ recent events, and the load balancer routes reconnects back by run ID. Short drops recover, but a crash or deploy still loses everything on that server.

Great: a per-generation token log. Split the roles. An orchestrator worker owns the run and appends small batches of tokens to a short-lived Redis stream per run, run:{g_9k1}, with XADD, using the event sequence number as the entry ID; the log is trimmed and expired after the run. Stream gateways become stateless: any gateway serves any client by reading the log with a blocking XREAD from the last ID it sent, and writing each entry as an SSE event with that ID. A client reconnecting with Last-Event-ID: g_9k1:31 reaches any gateway, which reads after entry 31 — exactly what XREAD returns — replays the backlog in one burst and carries on live.

The decoupling pays off four ways:

• Generation continues without a listener. Close the tab and reopen the app a minute later: the answer is in the conversation store, and the log lingers about 60 s for anyone still catching up.
• Deploys don’t kill answers. Clients of a draining gateway resume on another pod.
• Stop really stops. POST …:stop sets a cancel flag; the orchestrator cancels its request to the fleet, freeing the batch slot and KV cache memory at once, and saves the partial answer as user_stopped. Merely closing the browser’s connection would keep a GPU busy for nobody.
• Slow phones cost nothing. A slow reader just lags behind in the log; the gateway buffers nothing.

Two more details. Flush on time, not per token: sending what has accumulated every 30–50 ms cuts events by about a third and still looks smooth. And the log is a cache, not the record: Redis replicates asynchronously, and its documentation warns that a failover can lose the latest entries. The store holds the durable copy once the run ends, and a gateway that finds a gap asks the orchestrator for the text so far. One policy choice remains: paid runs keep generating with nobody listening, but at peak, free runs nobody has watched for 30 s are cancelled to return capacity to people who are waiting.

How do we fit the conversation into the context window?

The model remembers nothing between calls, so every turn rebuilds the prompt to fit a context window — say 128K tokens — shared between what you send and what comes back. It has six parts: the system prompt (instructions, safety policy and tool descriptions; about 2K tokens, identical for every user of a model version), memory about the user (about 0.5K), retrieved text and attachments, the history (the path from the root to the new message’s parent), the new message, and room for the answer, reserved up front (4–8K, more for reasoning models).

The window is a ceiling, not a target. Every prompt token costs prefill compute and money on every turn, because history is re-sent each time. At 700 tokens a turn, turn n re-sends about 700 × n tokens, and a 60-turn conversation sends 700 × (1 + 2 + … + 60) = 700 × 1,830 ≈ 1.3M history tokens in total: cost grows with the square of the length. So give each plan a product budget well under the window — 32K tokens for the default model here.

Bad: send everything until it fails. Fine for short chats, until a request exceeds the window and errors or is truncated from the wrong end; TTFT and cost creep up every turn before that.

Good: a sliding window. Keep the system prompt and the new message, reserve the answer’s room, then add history newest-first until the budget is full, counting with the model’s own tokenizer. Many products also pin the first user message, which often states the task. Cheap and predictable, but amnesiac: “my budget is ₹25,000”, said 40 turns ago, silently falls out.

Great: a rolling summary plus recent turns, with a stable prefix. When unsummarised history passes a threshold, a background job condenses the oldest block of turns into a “conversation so far” note of about 1K tokens. The prompt becomes system prompt, memory, summary, recent turns verbatim, new message. Very long conversations can also retrieve the few older turns most relevant to the new message — the machinery of RAG search. You keep what matters for a fraction of the tokens, and pay with summarisation calls, lossy summaries and care for the cache.

That last column needs one paragraph about the inference tier. A replica computes attention keys and values for every prompt token — the KV cache — and with prefix caching it keeps them, so a later prompt that starts with exactly the same tokens skips that work. Providers price the effect: OpenAI’s prompt-caching documentation (as of October 2026) bills cached input at 0.1× the normal rate on most current models, charges 1.25× to write a prefix with the default 30-minute lifetime, needs at least 1,024 tokens, and requires the whole prefix to match. Storing and routing this is the inference service’s job; keeping your prompts cacheable is yours:

• Static first, dynamic last. The system prompt is identical across users of a model version and is cached fleet-wide; the time or the user’s name goes at the end, as the provider guide also advises.
• Append, don’t rewrite. Turn n + 1’s prompt should be turn n’s plus two messages; a regenerate shares the whole prefix.
• Change the past in big, rare steps. A new summary invalidates everything after it (the guide notes that compaction prevents reuse from the first changed token), so re-summarise every 20 turns and drop history in blocks.
• Route for locality. Caches live on particular machines; keep an active conversation on one replica group by hashing its ID.

With 1,800 of 2,000 prompt tokens read from the cache at 0.1×, and the 200 new tokens written to it at 1.25× so the next turn can reuse them, the prompt costs 1.25 × 200 + 0.1 × 1,800 = 250 + 180 = 430 token-equivalents instead of 2,000 — a 78% cut in input cost — and most of the prefill work disappears too, which is where the TTFT budget’s 150 ms prefill came from.

How do we store and serve conversations at scale?

Two queries dominate: open a conversation, and list a user’s conversations newest first.

Bad: one relational database. A messages table indexed by conversation and OFFSET pagination for the sidebar: perfect for the first million users, but at 175,000 writes a second and 5 TB a day no single primary keeps up, and offset pages skip or repeat conversations as they jump to the top.

Good: Postgres sharded by user ID. A user’s data on one shard, with transactions and familiar tooling; it scales a long way. But resharding is your problem, anything that crosses users (shared links, support tools) gets awkward, and this volume means thousands of shards (see database sharding).

Great: a wide-column store keyed by the access patterns. In Cassandra or DynamoDB, use three tables:

• messages: partition key conversation_id, sort key a time-ordered message_id (ULID or Snowflake-style). Appending is a single-partition write; opening a conversation is one partition read that returns the whole tree in order.
• conversations: keyed by conversation_id — title, owner, active leaf, summary, last activity, and a version for conditional writes.
• conversations_by_user: partition key user_id, sorted by last activity, for the sidebar — a global secondary index in DynamoDB, or in Cassandra a table whose row you delete and re-insert when the conversation moves to the top.

The details that make it work:

• Stable pagination. The cursor encodes the last row’s activity time and ID; the next page is “rows in u_42 strictly older than the cursor, limit 30”, which new activity can’t shift. The index updates asynchronously (a DynamoDB global secondary index is eventually consistent), so the client adds a brand-new conversation to its own sidebar immediately.
• Big payloads go to blob storage. DynamoDB caps an item at 400 KB, and both stores prefer small rows, so a pasted document is stored as a pointer.
• No hot partitions in practice. A conversation writes a few times a minute against a partition ceiling of 1,000 write units a second; a script hammering one account hits its quota first. Very long conversations can be split into buckets (c_7Hq2#3).
• A history cache. Recent messages of active conversations sit in Redis, written through on save, so most turns skip the database.
• Search and titles off the write path. Change data capture feeds an index such as Elasticsearch, always filtered by user and a few seconds behind; workers write titles and summaries from orchestrator events.

Deletion is a feature. Deleting a conversation marks it deleted, so it vanishes everywhere at once; a purge job then removes messages, attachments, summaries, search documents, caches and token logs within a fixed deadline, while backups age out. To make “delete my account” true across backups, encrypt each user’s data with a per-user key and destroy the key — crypto-shredding (see encryption and key management). A “temporary chat” is a conversation with a short time-to-live that is never indexed or remembered.

How do we share scarce GPUs when demand spikes?

Picture 9:15 pm IST on the day a new model launches: demand is 30% above capacity, for hours, and you can’t buy GPUs tonight. Admission control decides who waits, who gets a smaller model, and who is asked to come back.

Bad: one first-come, first-served queue. Everyone’s TTFT rises until requests time out, clients retry into a retry storm, and one free user’s script with 50 parallel chats sits ahead of a paying customer.

Good: per-user rate limits and strict priority by plan. Token buckets per user stop hogging, and paid requests always go first. But strict priority starves the free tier through a long peak, and fixed limits know nothing about tonight’s actual capacity.

Great: capacity-aware admission with weighted fair queues and a degradation ladder.

• Queues per plan and pool, drained by a weighted fair scheduler — say 8 : 4 : 1 for pro, plus and free. Free traffic still moves, just slower, and each queue rotates between users, so ten parallel requests from one account take turns.
• Pull, don’t push. Dispatch only when a replica reports a free slot, so queues build in the scheduler, visible and cancellable, not inside GPUs — backpressure.
• Deadlines and a ladder. Each request carries a TTFT deadline by plan. When the predicted wait exceeds it, step down instead of queueing forever: (1) shorter answers and a smaller context budget; (2) the smaller model, with a notice; (3) for free users, a 429 with Retry-After; (4) pause titles, summaries and memory extraction until off-peak.
• Count tokens, not just messages. A 100K-token paste costs as much prefill as 50 normal messages. Public LLM APIs limit requests and tokens per minute, enforcing whichever runs out first; do the same per plan.
• Borrow across regions when another pool has room, as the multi-region deep dive shows.

The ladder works because models differ so much in cost. Say demand is 30M output tokens a second against 23M of capacity: 7.5M paid and 22.5M free. If the small model needs a fifth of the GPU time per token (an assumption) and pools can be rebalanced over tens of minutes, moving all free traffic to it costs 22.5M ÷ 5 = 4.5M large-model-equivalent tokens a second. Total: 7.5M + 4.5M = 12M of 23M. Everyone is served and paying users keep the large model; refusals are the last resort, not the first.

Clients cooperate: on a 429 they honour Retry-After, or back off exponentially with jitter as public rate-limit guides recommend, showing a countdown rather than retrying silently. Per-user counting reuses the rate limiter design; sharing between plans is noisy neighbours and fairness for GPUs.

How do we keep it safe without killing latency?

Safety comes in four layers, cheapest first:

1. Check the input before spending GPU time. A small classifier scores the message and any extracted attachment text in tens of milliseconds, in parallel with prompt building. A blocked message gets a fixed refusal and no generation. Paid plans can even start generating speculatively and discard the result on a fail, since blocks are rare; free users at peak wait for the verdict.
2. Moderate the output as it streams — the hard part, below. The model is trained to refuse, so this is defence in depth.
3. Watch behaviour, not just text. Count flags per account, device and network; repeat offenders get stricter thresholds, captchas or suspension, and mass sign-ups that farm free GPU time count as abuse.
4. Log for audit, carefully. Record the decision, category, scores, policy version and message ID — not extra copies of the text — in an access-controlled store with a retention limit.

Streaming makes output hard: partial text is harder to judge, and OpenAI’s documentation notes that moderation scores for a streamed generation arrive only once the full output exists. You can’t wait 13 seconds, so moderate in windows.

Bad: moderate the finished answer. The user has read it all before the verdict arrives.

Good: hold each sentence until it passes. Nothing flagged is shown, but the text moves in jerks and the first words arrive a whole window late.

Great: a short holdback with overlapping checks. Classify the last 100 or so tokens every 15 tokens — about every 0.5 s — and release text only after a check covering it has passed. The stream stays smooth, shifted by about 0.65 s (15 tokens plus an assumed 150 ms classifier call). On a flag, the orchestrator cancels the generation so the GPU stops, the client gets a retract event that replaces the partial answer with a safety message, and the run ends with finish_reason: moderation. Make the holdback risk-adaptive: conversations the input check scored as clearly harmless stream with no holdback, and retract covers the rare surprise.

Overlap costs classifier time: checking 100 tokens every 15 reads each output token about 100 ÷ 15 ≈ 7 times, or 23M × 7 ≈ 160M classifier tokens a second if every conversation used it — so only risky conversations get the overlapping mode, and the rest a cheap check every 50 tokens or so. A retracted answer must also leave the store, the search index and any shared link; the text survives only in the audit log, so wrong calls can be reviewed.

What happens when an inference replica dies mid-answer?

At 10,000 replicas, failure is routine. If each fails once a month, 10,000 ÷ (30 × 24) ≈ 14 failures an hour, each cutting about 80 answers off mid-sentence. A token watchdog in the orchestrator detects it: tokens normally arrive every 30 ms, so a second of silence — or no first token by the TTFT deadline plus a margin — means stalled. Fleet heartbeats often catch dead replicas sooner; the watchdog also catches stuck ones.

Bad: show an error. The user regenerates by hand, pays quota twice and loses the half-answer.

Good: retry from scratch. Invisible before the first token. After tokens were shown, a fresh attempt samples a different answer, so the partial text must be retracted — automatic, but visible.

Great: continue from the partial text. Send another replica a continuation request: the original prompt plus the partial answer, asking it to carry on — the recovery that provider documentation describes for interrupted streams. The orchestrator keeps numbering events where the log stopped, so the client sees a pause, not a restart. It costs one extra prefill, often partly cached, and may leave a slight seam in style; half-written tool calls fall back to restarting.

Two neighbouring failures need guards:

• The client retries a send. The POST timed out on a train, but the server had saved the message and started generating; a naive retry creates a duplicate message and a second run. So the client sends an Idempotency-Key per send; the chat API stores key → (message ID, run ID) for 24 hours with a conditional insert, and a repeat gets the existing run’s stream from the start (see idempotency).
• The orchestrator dies. Runs are claimed from a queue with a lease renewed every few seconds; when renewals stop, another orchestrator claims the run and continues it from the log. Each claim carries an epoch, and appends go through a tiny script that rejects older epochs, so a paused old owner can’t write — the fencing-token idea from distributed locking.

How do we run across regions?

Users and GPUs are both everywhere, but not in the same proportions or busy at the same hours, so treat data and compute separately.

• Data has a home. Each user is homed in one region — Mumbai for Indian users, Virginia for American ones — with their conversations, cache and token logs replicated across zones and asynchronously to a disaster-recovery region in the same country. One region owns each user’s writes, so nothing conflicts across regions (see multi-region and active-active).
• GPU capacity is a global pool. The home orchestrator sends each request wherever there is room. At 9 pm IST, Mumbai’s pool is full and Virginia’s is busy with the American working day, but Singapore, at 11:30 pm, has spare capacity.

Crossing regions is cheap here. Light in fibre covers about 200,000 km a second, so Mumbai to Singapore (about 3,900 km) costs at least a 40 ms round trip, and Mumbai to Virginia (about 13,000 km) at least 130 ms. The prompt is about 8 KB, and tokens return with a constant extra delay, not jitter. Against multi-second queues at peak, an idle GPU 50–100 ms away is an easy trade — as long as an active conversation stays in one GPU region to keep its prefix cache warm, and you borrow only when the home queue is longer than the extra round trip.

The final design

The connection tier, the durable record and the expensive work are separate tiers, joined by the token log.

Follow one message through the system:

1. Send. The client posts with an idempotency key; the edge routes it to any stream gateway, which holds the SSE response.
2. Save. The chat API deduplicates on the key, writes the message and a queued run, and moves the active leaf.
3. Start. The run goes on a queue, and an orchestrator claims it with a lease.
4. Prepare. The orchestrator loads the active branch from the cache, packs the prompt into the plan’s budget with a stable prefix, checks the quota and runs the input classifier in parallel.
5. Admit. The scheduler queues the run by plan and dispatches it to a pool with room — the large model, the small fallback or another region — or returns 429 to free users when the ladder runs out.
6. Stream. Tokens flow back through output moderation into the run’s token log; gateways tail it and push SSE events, resuming from Last-Event-ID after drops.
7. Finish. The orchestrator saves the reply and its usage and emits an event; workers write titles and summaries, index the text for search and record usage.

Check it against the requirements: TTFT about 0.4 s at p50 off-peak and protected for paid plans at peak; streams that survive drops and deploys; durable messages, with interrupted answers continued or marked; history readable while GPUs are saturated, because reads never touch the fleet; fairness from plan weights and per-user rotation; and about 0.09 cents of GPU time per message, never wasted.

What interviewers look for by level

Follow-up questions

What if users upload files and ask questions about them?

Uploads go straight to blob storage through signed URLs (see large file uploads); an async pipeline extracts text, counts tokens, and chunks and embeds big files. Small files fit the prompt’s attachment slot; large ones use retrieval, as in RAG search. Scan for malware, classify the extracted text, and treat it as untrusted: instructions hidden in a PDF are prompt injection and must never outrank the system prompt.

What if the assistant can use tools and act as an agent?

A generation becomes a loop: the model asks for a tool, the orchestrator runs it (a web search, code in a sandbox), appends the result to the tree as a tool message and calls the model again, streaming progress events. Longer, riskier runs need durable execution, step and budget limits, and approval before side effects — Design an AI agent platform.

What if we add voice conversations?

Speech flows both ways and people interrupt, so use WebRTC or WebSockets instead of SSE. The pipeline is streaming speech-to-text, the LLM and streaming text-to-speech, or one speech-native model. The target becomes time to first audio, and an interruption must cancel generation and playback within a few hundred milliseconds. Transcripts land in the same conversation tree.

What if a user shares a conversation publicly?

Share a snapshot: copy the active branch into a read-only record with an unguessable ID, strip memory and private attachments, moderate it and serve it from a CDN. Later edits don’t change it, and deleting the share or the conversation revokes it and purges the CDN. Keep shared pages out of search engines by default.

What if enterprise customers require data residency?

Pin the tenant’s home region and its GPU pool to the jurisdiction, and switch off global borrowing — which means reserved in-country capacity, since they can’t overflow to Singapore at 9 pm. Add per-tenant, optionally customer-managed keys, a no-training guarantee, admin audit logs and retention controls (see multi-tenancy and isolation).

What if we want long-term memory across chats?

An async job extracts durable facts after conversations — “vegetarian”, “works in Pune”, “prefers answers in Hindi” — and stores them per user with their source, where the user can see, edit and delete them. At prompt time the few relevant facts fill the 0.5K memory slot, in a stable order to keep the prefix cache warm. Memory never crosses users and stays off in temporary chats.

How to say it in the interview

“I’ll design a consumer chat assistant: 100 million daily users, about 2 billion messages a day, a 2.5× peak when India’s evening overlaps the US working day, and a first token in under a second. The GPU fleet is an internal service; my job is everything around it.

A message is a POST that returns an SSE stream. The chat API saves it under an idempotency key into a message tree, so edits and regenerations are branches and an active leaf picks what you see. An orchestrator owns the generation: it packs a per-plan token budget — system prompt, memory, rolling summary, recent turns, room for the answer — with a stable prefix so most of it hits the prefix cache, and checks quota and safety in parallel.

Generation is decoupled from the connection. Tokens go to a short-lived Redis stream per run that stateless gateways tail, so a dropped phone resumes with Last-Event-ID, answers finish if the tab closes, and stop cancels the GPU work.

At peak that’s 58,000 messages and 23 million output tokens a second: about 10,000 eight-GPU replicas and 810,000 open streams. GPUs are the whole bill and can’t be added tonight, so admission uses per-plan weighted fair queues and a ladder — shorter answers, the smaller model, then a 429 for free users — while background work pauses.

Output is moderated in overlapping windows, and a flag stops the GPU and retracts the text. Conversations live in a wide-column store partitioned by conversation and by user. Dead replicas are caught by a token watchdog and continued from the partial answer, and GPUs are borrowed across regions while data stays home.”

Concepts used in this design

• Real-time connections: long polling, SSE and WebSockets — why SSE carries the tokens.
• Real-time updates — pushing changes to clients.
• Design an LLM inference service and LLM inference serving — the fleet behind generate.
• Design an LLM gateway — reserve-then-reconcile token budgets.
• Redis — the token log, quota counters and history cache.
• Cassandra and DynamoDB — partition keys for the two hot queries.
• Pagination: offset vs cursor — a stable sidebar.
• Rate limiting and Design a rate limiter — per-user message and token limits.
• Backpressure and load shedding — admit only when a GPU has room.
• Noisy neighbours and fairness — weighted sharing between plans.
• Idempotency — retried sends without duplicates.
• Heartbeats and distributed locking — dead replicas, leases and fencing.
• Change data capture — search indexing off the write path.
• Encryption and key management — crypto-shredding for real deletion.
• Multi-region and active-active — home regions for data, a global pool for GPUs.

References and further reading

1. WHATWG HTML Living Standard, section 9.2: Server-sent eventshtml.spec.whatwg.orgThe event stream format, the id and retry fields, the Last-Event-ID header sent on reconnect, 204 to stop reconnecting, and the advice to send a comment line every 15 seconds or so.
2. Anthropic documentation: Streaming Messagesplatform.claude.comNamed SSE events, ping events, overload errors that arrive mid-stream, and recovering an interrupted stream with a continuation request built from the partial answer.
3. OpenAI API documentation: Streaming API responsesdevelopers.openai.comTyped semantic stream events, and the warning that streaming makes moderation harder because moderation scores arrive only after the full output.
4. OpenAI API documentation: Prompt cachingdevelopers.openai.comExact-prefix matching, cached reads at 0.1× and 30-minute writes at 1.25× the input price on current models, the 1,024-token minimum, per-machine caches, and keeping dynamic content at the end.
5. Redis documentation: Redis Streamsredis.ioXADD as an append-only log, XREAD after a last-seen ID with BLOCK, MAXLEN trimming, half a million to a million appends a second with pipelining, and asynchronous replication that can lose recent entries on failover.
6. OpenAI API documentation: Rate limitsdevelopers.openai.comLimits in requests and tokens per minute and per day, enforced on whichever is hit first; Retry-After; exponential backoff with jitter.
7. OpenAI API documentation: Moderationdevelopers.openai.comModeration categories with a flag and a score each, moderating input and output separately, and scores for streamed output arriving only at the end.
8. Amazon DynamoDB Developer Guide: Designing partition keysdocs.aws.amazon.comEach partition delivers at most 3,000 read units and 1,000 write units a second; one write unit covers an item of up to 1 KB.
9. Amazon DynamoDB Developer Guide: Storing large itemsdocs.aws.amazon.comThe 400 KB item size limit, and keeping large payloads in S3 with a pointer in the item.
10. How People Use ChatGPT (Chatterji et al., NBER Working Paper 34255, September 2025)nber.orgPublic usage figures: about 2.6 billion consumer messages a day in June 2025, and 18 billion messages a week from 700 million users by July 2025.