Design an LLM inference service

An LLM inference service turns a prompt into a stream of tokens, thousands of times a second, on hardware that costs more per hour than the rest of your servers put together. Picture a Bengaluru fintech whose support bot, loan-document summariser, IDE code-completion plug-in and nightly ticket-tagging jobs all call the same self-hosted open-weights models: a 70-billion-parameter model for hard work and an 8B model for quick, cheap work. Each request is really two jobs — read the whole prompt at once, then write the answer one token at a time — and the GPU memory that holds each request’s state decides how many users fit on a machine. In this problem you design the serving fleet behind that API: tokens streamed within latency targets, at the lowest cost per token.

Understand the problem

An LLM endpoint looks like any other HTTP API until you look at what one call costs. A typical request here sends 1,500 prompt tokens and gets 300 back, holds over half a gigabyte of GPU memory for about ten seconds, and shares the GPU with dozens of other requests. Agree on the vocabulary first, because every later decision trades one of these numbers against another.

• Time to first token (TTFT): from arrival to the first streamed token. It is queueing plus prefill, the forward pass that reads the whole prompt. Users feel it as “is it thinking?”
• Time per output token (TPOT), also called inter-token latency: the gap between tokens after the first. At 30 ms per token you stream about 33 tokens a second, faster than anyone reads.
• End-to-end latency = TTFT + (output tokens − 1) × TPOT. It matters most to callers that wait for the whole answer, such as a summariser.
• Throughput: tokens per second per replica or per GPU, with prompt and output tokens counted separately. It decides cost.
• Goodput: the request rate you can serve while meeting both latency targets for a chosen share of requests. The DistServe paper optimises per-GPU goodput because throughput that breaks the SLO is waste with a nice graph.

Callers fall into two tiers. Interactive callers — the support bot, the code-completion plug-in — need low TTFT and steady TPOT. Batch callers — tagging two million support tickets overnight, translating a product catalogue into Hindi and Tamil — care only that the job finishes by morning and costs little. Mixing them carelessly lets a batch job ruin the bot’s latency; separating them carelessly leaves GPUs idle all night.

Functional requirements

1. Product teams should be able to send a prompt to a named model and receive the answer as a token stream (or as one response), through a single OpenAI-compatible API.
2. Teams should be able to choose a service tier: interactive, with latency SLOs, or batch, cheaper and finished within 24 hours.
3. Teams should be able to use their own fine-tuned variants (LoRA adapters) of the shared base models without paying for a dedicated fleet.
4. Platform operators should be able to roll out model versions and change capacity without dropping traffic.

Non-functional requirements

• Latency, interactive 70B: TTFT p95 under 1 s for prompts up to 4K tokens; TPOT p95 under 50 ms. Interactive 8B (code completion): TTFT p95 under 300 ms.
• Goodput: at least 99% of interactive requests meet both targets at peak.
• Scale: 10M requests a day on the 70B model, averaging 1,500 prompt and 300 output tokens; up to 700 requests/s on the 8B model, averaging about 1,000 prompt and 50 output tokens.
• Context: up to 8K tokens on the interactive pools (the model supports 128K; see the follow-ups).
• Availability: 99.9% of requests succeed. Losing a GPU fails only the requests running on it, and those that have not streamed a token yet are retried invisibly.
• Cost: lowest cost per million tokens — GPUs at least 60% busy at peak, with batch work filling the troughs.
• Quality: any change that alters numerics, such as quantization, must pass the same evaluation suite as the BF16 reference before it serves traffic.

Out of scope

• The consumer chat product — conversation storage, edit and regenerate, resumable streams in the browser — is covered in Design ChatGPT.
• A multi-provider gateway with virtual keys, token budgets and fallback between providers is Design an LLM gateway. Our front end does authentication and per-team quotas only.
• Training and fine-tuning jobs, and how a shared cluster schedules them, belong to Design a GPU job scheduler. We serve the adapters training produces.
• Safety moderation, retrieval and tool calling happen in the calling services, before and after us.

The set-up

Plan the approach

Say the shape early: “a stateless front end and router, in front of pools of stateful GPU replicas. Each replica is an inference engine doing continuous batching over a paged KV cache. A control plane scales the pools and rolls out models.” The high-level design is short. The interview is won inside and between replicas: memory, batching, latency, parallelism, routing, autoscaling and failure.

Core entities

• Model deployment. Model name and version, weights location, precision (BF16 or FP8), parallelism (for example TP-4), and engine settings such as maximum context. Immutable: any change is a new version.
• Replica. One running engine on a fixed set of GPUs, either loading, ready or draining. It reports queue depth, KV cache use and latency.
• Pool. The replicas of one deployment serving one tier: the unit of routing and autoscaling.
• Request (sequence). Prompt tokens, sampling settings, maximum output tokens, tier, tenant and optional adapter. It exists only while it runs.
• KV block. The attention keys and values for 16 tokens in GPU memory, with a reference count and a content hash used for prefix reuse.
• Adapter. A LoRA fine-tune, about 130 MB, tied to one base model.

API

Expose the OpenAI-style chat API that most client libraries already speak. A fine-tune is addressed as its own model name; tier is our extension.

POST /v1/chat/completions
Authorization: Bearer <team API key>
Content-Type: application/json

{
  "model": "llama-70b-instruct",
  "messages": [
    { "role": "system", "content": "You are the support assistant for a payments app. ..." },
    { "role": "user",   "content": "My UPI refund from Tuesday hasn't arrived. Why?" }
  ],
  "max_tokens": 400,
  "temperature": 0.3,
  "stream": true,
  "tier": "interactive"
}

The response is a server-sent events stream: one event per token or small group of tokens, and a final event with usage, including how many prompt tokens came from the prefix cache, so the team can see what it is billed for.

HTTP/1.1 200 OK
Content-Type: text/event-stream

data: {"id":"req_7f3a","choices":[{"delta":{"content":"Refunds"}}]}

data: {"id":"req_7f3a","choices":[{"delta":{"content":" to"}}]}

data: {"id":"req_7f3a","choices":[{"delta":{},"finish_reason":"stop"}],
       "usage":{"prompt_tokens":1512,"cached_tokens":1024,"completion_tokens":287}}

data: [DONE]

When admission control says no, the answer is 429 Too Many Requests with Retry-After before any stream opens; a missing or draining model gets 503. Batch work and operations get their own small surfaces:

POST /v1/batches                              file of requests, model, completion_window "24h" → batch id
GET  /v1/batches/b_91                          status, counts, output file when done
PUT  /v1/deployments/llama-70b-instruct        new version: weights URI, precision fp8|bf16, tp 4, max_len 8192
POST /v1/deployments/llama-70b-instruct:canary send 5% of traffic to the new version
PUT  /v1/adapters/support-bot-v3               base llama-70b-instruct, LoRA weights URI

Back-of-envelope estimates

These numbers decide the replica shape, the fleet size and the bill. (For the method, see back-of-envelope estimation.)

Traffic. 10M requests a day ÷ 86,400 s ≈ 116 requests/s on average. Office hours in India and an evening peak give a peak-to-average ratio of about 2.6, so plan for 300 requests/s at peak on the 70B model. That is 300 × 1,500 = 450,000 prompt tokens/s and 300 × 300 = 90,000 output tokens/s.

Weights. Seventy billion parameters at 2 bytes each in BF16 is about 140 GB. One H100 SXM has 80 GB of HBM, so the model needs at least two GPUs just to load — and two would leave almost nothing for KV cache, so we use four.

KV cache per token. Every token a sequence has seen leaves a key and a value vector in every layer, so later tokens can attend to it without recomputation. Llama 3 70B has 80 layers and 64 query heads but only 8 key-value heads (grouped-query attention) of dimension 128. One token costs 2 (K and V) × 80 × 8 × 128 × 2 bytes = 327,680 bytes, or 320 KiB — eight times less than without grouped-query attention. The 8B model (32 layers, 8 KV heads) needs 128 KiB per token.

Per sequence. An 8K-token conversation holds 8,192 × 320 KiB = 2.5 GiB (2.7 GB) of KV cache. Our average request ends at 1,800 tokens: 1,800 × 327,680 B ≈ 0.59 GB.

How many requests fit on one replica. Take four 80 GB H100s, 320 GB in all. Engines leave a safety margin (vLLM uses 92% of GPU memory by default in current releases), so 294 GB is usable. Subtract 140 GB of weights and about 16 GB for activations, CUDA graphs and communication buffers, and 138 GB is left for KV cache — about 420,000 tokens.

Decode speed at batch size 1. To produce one token, the GPUs must read every weight once. With four-way tensor parallelism each GPU reads its 35 GB share per step: 35 GB ÷ 3.35 TB/s = 10.4 ms, so at most 96 tokens/s for a lone sequence. Kernels reach roughly 80% of peak bandwidth, so expect about 13 ms, or 76 tokens/s. The arithmetic is tiny: about 2 FLOPs per parameter, so 140 GFLOP per token against 140 GB read — 1 FLOP per byte, while an H100 can do 989 dense BF16 teraFLOPS ÷ 3.35 TB/s ≈ 295 FLOPs per byte. At batch 1 the tensor cores idle over 99% of the time.

Prefill speed. All 1,500 prompt tokens go through in one pass: 1,500 × 140 GFLOP = 210 TFLOP. The replica’s peak is 4 × 989 = 3,956 dense BF16 teraFLOPS; assume serving achieves 40% of that, about 1.6 PFLOP/s. Prefill takes about 130 ms, the floor under TTFT. The weights are read once for 1,500 tokens, so compute runs out, not bandwidth.

Fleet size. Size on compute, then check memory. At peak the 70B pool processes 540,000 tokens/s × 140 GFLOP = 75.6 PFLOP/s, so 75.6 ÷ 1.6 ≈ 47 fully busy replicas; at 65% busy, to keep queues short, 73 replicas, 292 GPUs. If the prefix cache serves 40% of prompt tokens (shared system prompts, chat history — an assumption to measure), demand falls to (270,000 + 90,000) × 140 GFLOP = 50.4 PFLOP/s: 49 replicas, 196 GPUs, about 25 eight-GPU nodes.

Now memory. With a 30 ms TPOT, a request lives about 0.3 + 299 × 0.03 ≈ 9.3 s, so 300 × 9.3 ≈ 2,800 sequences are in flight, about 57 per replica. At 0.59 GB each that is 34 GB of the 138 GB budget — room for long prompts and a warm prefix cache.

The 8B pool: 700 × 1,050 tokens = 735,000 tokens/s × 16 GFLOP = 11.8 PFLOP/s. One H100 at 40% of 989 teraFLOPS gives 0.4 PFLOP/s, so 30 busy GPUs, 46 at 65%. The whole fleet at peak is about 242 H100s, roughly 31 nodes.

Cost. Assume $2.50 per H100-hour — an illustrative on-demand cloud price; check your provider, since reserved prices vary widely. The 70B pool costs 196 × $2.50 = $490 an hour at peak and emits 90,000 × 3,600 = 324M output tokens an hour: $1.51 per million output tokens, with all prompt processing included. One replica serving one request at a time costs $10 an hour for 76 × 3,600 ≈ 274,000 tokens: $36 per million. Batching is the difference between a viable service and a bonfire.

High-level design

Build one requirement at a time. By the end of this section the service works; the deep dives make it fast and cheap.

Teams should get a token stream from a named model

The request path has three hops. The API front end authenticates the team’s key, checks its token quota, counts prompt tokens and holds the server-sent events connection. The router picks a replica of the right model. The replica is one inference engine — vLLM, SGLang or TensorRT LLM — on four NVLink-connected GPUs: a scheduler runs one forward pass at a time for every sequence it holds, a KV cache manager owns GPU memory, and one worker per GPU executes its shard of the model.

Each new token goes back as soon as it exists, and the front end forwards it as an SSE event; non-streaming callers get the same path, buffered. Weights are copied from an object store when a replica starts.

Two choices deserve a sentence. Streaming matters: a 300-token answer takes about 9 seconds, but the first words arrive in about 0.3 s. The router is separate from the front end because it needs live replica state — queue depth, KV use, cached prefixes — that the front end should not care about.

Teams should choose a service tier

Interactive requests go straight to the router. Batch requests arrive through /v1/batches, land in a durable queue, and a dispatcher feeds them to the same pools at low priority whenever a pool has headroom (a short queue, KV cache below about 70%). Engines preempt batch sequences first when interactive load rises.

This is where cost per token is won: interactive traffic at 3 a.m. is a fraction of the evening peak, and 2 million overnight ticket-tagging calls soak up GPUs that would otherwise idle. Batch callers get a lower price for a 24-hour deadline. Give batch its own pool only when its volume justifies it.

Teams should use their own fine-tunes

A LoRA adapter adds small low-rank matrices beside some weight matrices. At rank 16 on the 70B model’s four attention projections, one layer adds 8,192 × 16 + 16 × 8,192 = 262,144 parameters for the query projection, 8,192 × 16 + 16 × 1,024 = 147,456 each for key and value, and 262,144 for the output: 819,200 per layer × 80 layers ≈ 65.5M parameters ≈ 131 MB, about 0.1% of the base model.

So one replica holds the base model once plus dozens of adapters, and one batch can mix requests for different adapters. The caller names the adapter as its model; the engine applies it per request (vLLM caps the adapters active in a batch with max_loras and keeps more in CPU memory). You gain a fine-tune per team for 131 MB each; you pay a little compute per step and a load delay when an adapter is not resident, so the router prefers replicas that already have it.

Operators should roll out versions and change capacity

Add a small control plane. A model registry stores each deployment version: weights URI and hash, precision, parallelism and engine settings. An autoscaler (the pool controller) creates and removes replicas from those versions using live metrics. A new version rolls out like any canary: start a few replicas, have the router send them 5% of traffic, compare TTFT, TPOT, error rate and evaluation scores, then shift traffic and drain the old replicas — letting their streams finish rather than killing them.

You now have a working service: streamed answers, two tiers, per-team fine-tunes, and safe rollouts. The deep dives are about making each GPU do far more work without breaking the latency targets.

Deep dives

How do we fit more concurrent requests in GPU memory?

Batch size is throughput, and KV cache memory caps batch size. Every request you cannot fit is a request that waits.

Bad: reserve contiguous memory for the maximum length. Earlier serving systems gave each request one contiguous KV region sized for its maximum possible length, because the attention kernel wanted contiguous memory. Three kinds of waste follow: slots reserved for tokens not generated yet, slots that are never used because the answer ended early (internal fragmentation), and free gaps between regions too small for the next request (external fragmentation). The vLLM authors profiled such systems and found that only 20.4–38.2% of KV memory held actual token state. On our replica that is the difference between 51 and 233 concurrent requests.

Good: allocate small fixed-size blocks on demand — PagedAttention. Borrow virtual memory from operating systems. Split each sequence’s KV cache into blocks of 16 tokens (vLLM’s default; the paper found 16 large enough to keep the GPU busy and small enough to waste little). A per-sequence block table maps logical blocks to physical blocks anywhere in GPU memory, and the attention kernel follows the table. A new block is allocated only when the last one is full, so a sequence wastes at most one partly filled block. The paper reports 2–4× the throughput of FasterTransformer and Orca at the same latency.

Great: share blocks, and shrink them. Give each physical block a reference count and copy-on-write, and requests can share memory safely. Several samples of one prompt share its blocks; the paper measured 6.1–30.5% memory saved for parallel sampling and 37.6–66.3% for beam search, depending on the dataset. A system prompt shared by every support-bot request is stored once — the basis of prefix caching in the latency deep dive. Then cut bytes per token: an FP8 KV cache halves 320 KiB to 160 KiB, and cold blocks can be offloaded to CPU memory.

What you pay: the attention kernel does an extra table lookup, and memory can still run out mid-generation because nobody knows how long an answer will be. The engine then preempts a sequence; vLLM’s V1 engine frees its blocks and recomputes them later rather than swapping to CPU. Frequent preemption is a signal to add capacity or lower the batch limit, not something to tune away.

How do we batch without hurting latency?

Batching is how a memory-bound decode step serves many users for the price of one weight read. The question is how to batch requests that arrive at random times and finish at random lengths.

Bad: no batching, or static batching. One request per replica costs $36 per million tokens, as computed above. Static batching — collect a batch, run it until every member finishes, then take the next — wastes the slots of requests that finish early and makes new arrivals wait for the longest answer in the batch.

Good: continuous batching. The Orca paper (OSDI 2022) introduced iteration-level scheduling: the scheduler runs one forward pass at a time, so a finished sequence leaves after any step and a waiting one joins at the next. To batch sequences of different lengths together, Orca also uses selective batching: the big matrix multiplications are batched across all sequences, while attention runs per sequence over its own KV cache. On GPT-3 175B, Orca reported 36.9× the throughput of FasterTransformer at the same latency. Every serious engine now does this.

How big should the batch get? Decode stays memory-bound for a long way, so a bigger batch costs little extra time per step. Model it for our TP-4 replica: each step reads 35 GB of weights per GPU plus each sequence’s KV cache (about 1,650 tokens × 80 KiB per GPU ≈ 0.135 GB), at an achieved 2.68 TB/s, priced at $10 an hour:

Going from 1 to 8 sequences cuts cost about 8× while TPOT rises by 0.4 ms. At 128, compute is still not the limit (128 × 140 GFLOP ÷ 1.6 PFLOP/s ≈ 11 ms, under the 19.5 ms of memory traffic). The decode-only numbers are optimistic — real batches also carry prefill — but the shape is right: batch as large as memory and your TPOT target allow.

Great: continuous batching with chunked prefill. One problem remains. When a 6,000-token prompt arrives, its prefill costs 6,000 × 140 GFLOP = 840 TFLOP, about 525 ms of the replica’s compute. If that runs as one step, every other stream on the replica freezes for half a second: a TPOT spike that breaks the p95 target. Chunked prefill gives each step a token budget, say 512: decodes go first, and the rest of the budget is filled with a slice of the waiting prompt.

In vLLM’s V1 engine chunked prefill is on by default, with exactly this decode-first policy. The knob is max_num_batched_tokens: smaller budgets such as 2,048 protect inter-token latency, larger ones improve TTFT, and the docs suggest more than 8,192 for pure throughput. What you pay: long prompts reach their first token later, and the interference is reduced, not removed — every step that carries a prefill chunk is slower than a pure decode step. The DistServe authors make exactly this point, which leads to disaggregation below.

How do we serve a model bigger than one GPU?

At 140 GB the 70B model does not fit in one 80 GB GPU, and a 405B model (about 810 GB in BF16) does not fit in one eight-GPU node.

Bad: one GPU plus CPU offload. Stream the weights that don’t fit from host memory each step: over 60 GB across PCIe per token, roughly a second, against 13 ms from HBM. Fine for a laptop demo, useless for a service.

Good: tensor parallelism inside a node. Split every weight matrix across the node’s GPUs (the Megatron-style split vLLM implements), so each GPU holds and reads a quarter of each layer, and keeps the KV heads it computes. Per-token latency falls with the bytes each GPU reads. The cost is two all-reduce operations per layer — 160 per forward pass — each on the critical path, affordable only over NVLink (900 GB/s per H100 SXM). So the group stays inside one node, and the model bounds it too: with 64 query heads and 8 KV heads, TP of 2, 4 or 8 splits cleanly; beyond 8, KV heads must be copied.

Good, across nodes: pipeline parallelism. Give each node a contiguous range of layers. Only the hidden state crosses between stages — for a 405B model, 16,384 values × 2 bytes = 32 KB per token — so ordinary networking copes. You pay a hop per stage, and stages idle unless several micro-batches are in flight. Meta served Llama 3 405B in BF16 this way: tensor parallelism inside each of two 8-GPU machines, pipeline parallelism between them, micro-batching for throughput.

Great: use the smallest parallelism that meets memory and latency, then add replicas. More GPUs per replica means more synchronisation per token and rarely better throughput per GPU. Our 70B replicas use TP-4, and throughput comes from 49 independent replicas (data parallelism), which also gives you 49 small failure domains instead of a few big ones.

Quantization changes the arithmetic. Fewer bytes per weight means fewer GPUs, less to read per decode step and more room for KV cache. Keep the options straight:

The H100 does FP8 natively at twice its BF16 rate (1,979 versus 989 dense teraFLOPS), so FP8 helps prefill as well as decode. Meta reported FP8 inference for Llama 3 405B gave up to 50% more prefill throughput and a better throughput–latency trade-off in decode.

How do we cut time to first token and time per token?

Three techniques, each attacking a different term: prefix caching removes prefill work, speculative decoding makes each decode step yield more than one token, and disaggregation stops the two phases from slowing each other down.

Prefix caching: don’t recompute what you computed a second ago. The support bot sends the same 1,024-token system prompt — persona, refund policy, tone rules — with every request, and its KV cache is identical every time, because KV depends only on earlier tokens. vLLM’s automatic prefix caching hashes each full 16-token block together with its parent block’s hash (plus extras such as the adapter ID), so one hash identifies the whole prefix. A new request reuses every matching block and computes only the rest; idle blocks stay cached until memory is needed, least recently used evicted first. SGLang’s RadixAttention keeps the same idea in a radix tree and schedules the longest cached prefix first.

With 1,024 of 1,512 prompt tokens cached, prefill drops from about 130 ms to 488 × 140 GFLOP ≈ 68 TFLOP, about 45 ms. Multi-turn chat gains more, since each turn’s prompt extends the last. Two rules: keep shared text byte-identical and first — a timestamp at the top of the system prompt defeats the cache for everyone. And isolate tenants: a hit is measurably faster, which can leak whether someone else sent the same text, so vLLM accepts a per-request cache_salt that limits sharing to callers with the same salt. Salt per team.

Speculative decoding: guess cheaply, verify in bulk. At small batch sizes the replica has spare compute, and checking several tokens costs about the same as producing one, because the weight read dominates. So a small draft model (our 8B, same tokenizer) proposes k tokens one by one, and the 70B target scores them all in one forward pass. A modified rejection-sampling rule accepts a prefix of the guesses and resamples at the first rejection, so the output follows exactly the target model’s distribution (Leviathan et al.; Chen et al.).

How much it helps depends on the acceptance rate α and the draft’s cost c relative to a target step. With k guesses, expected tokens per target pass are (1 − α^(k+1)) ÷ (1 − α), and the speed-up is that divided by (1 + k × c). For α = 0.8, k = 4 and c ≈ 0.11 (the 8B reads 16 GB per step against the 70B’s 140 GB): 3.36 ÷ 1.44 ≈ 2.3× lower TPOT. The papers measured 2–3× on T5-XXL and 2–2.5× on the 70B Chinchilla with a 4B draft, both at batch size 1.

Now try batch 128. Verifying 5 positions for 128 sequences is 640 tokens: 640 × 140 GFLOP ≈ 90 TFLOP, about 56 ms of compute plus about 8 ms of drafting, for 128 × 3.36 ≈ 430 tokens. That is about 6,700 tokens/s — the same as the 6,560 without speculation, and worse once prefill shares the step. Speculation turns spare compute into lower latency; a busy GPU has nothing spare to turn. Enable it for latency-critical, lightly loaded pools (code completion at 2 a.m., a premium tier) and let the engine switch it off as batches grow.

Disaggregation: separate prefill and decode replicas. Chunked prefill reduces interference but every chunk still slows the decodes it rides with, and one replica shape must suit both phases. DistServe (OSDI 2024) runs prefill and decode on different GPUs, each sized and parallelised for its own SLO — TTFT for prefill, TPOT for decode — and reported up to 7.4× more requests, or 12.6× tighter SLOs, within the latency targets. The price is moving the KV cache from the prefill replica to the decode replica for every request.

The transfer is sizeable: DistServe’s own example is 1.13 GB for a 512-token request on OPT-66B, 90 Gbps at 10 requests a second. Grouped-query attention makes our model far lighter per token, but the fleet still moves over a terabit per second, so disaggregation assumes an RDMA fabric. vLLM ships it as an experimental feature whose docs say plainly it does not improve throughput; its value is tuning TTFT and TPOT separately and controlling tail inter-token latency. Start colocated with chunked prefill, and disaggregate when tail TPOT is the SLO you keep missing and your network can carry the KV cache.

How do we route and load-balance requests?

Replicas are not interchangeable. Each holds different cached prefixes and adapters, a different queue, and a different amount of free KV memory — and requests differ by 100× in cost. A 6,000-token summarisation and a 50-token completion are both “one request”.

Bad: round-robin or least-connections. Both treat requests as equal. Round-robin happily sends the next request to a replica whose KV cache is 95% full, where it waits or forces a preemption, while a neighbour sits at 30%. It also scatters the turns of a conversation across replicas, so each turn recomputes the whole history. (For the general L4 versus L7 trade-offs, see load balancing.)

Good: least outstanding work. Score replicas by what they actually owe: prompt tokens waiting for prefill, sequences decoding, and KV cache utilisation, all reported by the engines every few hundred milliseconds. Route to the lowest score. This alone fixes the hot-replica problem.

Great: prefix-aware routing on top of load. Hash the first few blocks of the prompt — the same block hashes the engine uses — and prefer replicas that already hold that prefix, unless they are overloaded. Pin conversations softly: the conversation ID maps to a replica, falling back to the next-best replica when that one is full, much like consistent hashing with bounded load. SGLang’s router keeps a tree of which prefixes each worker holds, updated as workers evict; the llm-d project reports 3× output throughput and 2× faster TTFT from prefix-aware routing compared with round-robin on Llama 3.1 70B. The same affinity applies to LoRA adapters.

Admission control belongs here too. The router can predict TTFT: queued prompt tokens ÷ the pool’s prefill rate, plus this request’s own prefill. If that breaks the SLO on every replica, queueing the request only makes everyone late. Return 429 with Retry-After at once for interactive calls and push batch calls back to their queue — backpressure and load shedding applied to tokens. Within a pool, weighted fair queuing per team stops one team’s burst from starving the rest (noisy neighbours), and separate pools per model and tier keep an 8B flood away from 70B capacity.

How do we autoscale when a replica takes minutes to start?

Two things make LLM autoscaling unusual: the usual signal is useless, and new capacity is slow.

Bad: scale on GPU utilisation. The common GPU “utilisation” metric reports the share of time any kernel is running, so a replica crawling along at batch 2 and one saturated at batch 200 both read close to 100%. CPU usage is meaningless too. Scale on what users feel and what predicts it: waiting requests and queued prompt tokens, KV cache utilisation, and p95 TTFT and TPOT against the SLO. Scale out when KV use stays above about 80% or predicted TTFT nears the target; scale in slowly. (General lessons are in autoscaling done right.)

Good: understand the cold start, and cut it. Count what a brand-new 70B replica must do:

• Get a GPU node: several minutes from a cloud provider — if one is available at all during a launch-day rush.
• Pull the serving image: often 10 GB or more.
• Fetch 140 GB of weights: at 10 Gb/s, 140 GB × 8 ÷ 10 Gb/s = 112 s at full line rate; from local NVMe at about 6 GB/s, about 25 s; from host memory into four GPUs over PCIe, under a second.
• Start the engine: load weights, profile memory, compile and capture CUDA graphs. vLLM documents ways to shorten this: reuse a saved compile cache, pass the KV cache size to skip profiling, or skip CUDA graphs at the cost of slower decode.

Great: keep capacity warm, and scale ahead of demand.

• Warm pool. Keep a few GPU nodes running with the image pulled and weights on local NVMe. Idle nodes cost money — the price of turning minutes into seconds.
• Standby replicas. vLLM’s sleep mode moves weights to host memory and frees the GPU without stopping the server, then wakes it without a restart — a way to park the interactive model at night while batch work borrows the GPUs.
• Predictive scaling. Traffic follows Indian office hours and an evening peak; scale out 15 minutes ahead of the curve and let reactive scaling handle surprises.
• Drain on scale-in. Stop routing to the replica and let its streams finish — about 20 s for 400 tokens at 50 ms — before stopping it.
• Scale to zero only for small models. The 8B model’s 16 GB loads from NVMe in about 3 s; a rarely used fine-tune is better as a LoRA adapter on a shared base. Keep a floor of 70B replicas per zone.

Night-time GPUs can also go to evaluation and fine-tuning jobs through the GPU job scheduler, reclaimed before the morning ramp.

How do we handle failures mid-generation?

A replica holds no durable state: the KV cache is a cache of computation that any replica can rebuild from the prompt. Failures are simple to reason about, if not free.

A GPU or node dies. Every sequence on that replica, about 57 at peak, loses its KV cache at once. The router retries only requests that have not sent a token to the caller yet; they never knew. Generation has no side effects, so this is safe — with temperature above zero the retry is a different sample, but the caller never saw the first. For streams already in progress, either send an error event and let the caller retry, with an idempotency key so usage is not billed twice, or resume: send the prompt plus the tokens already streamed to another replica as a new prompt. You pay a fresh prefill, and the continuation is a valid sample, not the one you would have got. Bill only delivered tokens.

A GPU gets slow or stuck. This is worse than a clean death: with tensor parallelism one slow GPU slows all four, because every layer waits for the all-reduce. Watch for driver Xid and ECC errors in node logs, thermal throttling and hung collectives. A step-time watchdog — kill the engine if a step takes 10× the normal time — turns a hang into a clean failure.

Health checks must run the model. An HTTP 200 from the server process proves nothing about the GPUs. The pool controller sends each replica a tiny generation — a fixed 16-token prompt, 1 output token, a 2 s deadline — every few seconds. Two failures in a row: stop routing, drain, cordon the node and replace it.

Everything else is ordinary resilience: stateless front ends and routers in several zones, per-request deadlines, cancelling generation (and freeing KV blocks) when a caller disconnects, timeouts and backoff on retries, and canaries for new versions, because a bad version fails every request at once.

The final design

Put it together. The data path is front end, router and GPU pools; the control plane decides how many replicas exist and whether each one is healthy.

Follow one support-bot request:

1. Request. The app calls POST /v1/chat/completions with stream: true. The front end authenticates the team, checks its token quota, counts prompt tokens and opens the SSE stream.
2. Route. The router picks the 70B interactive pool, hashes the prompt’s first blocks, and chooses the replica with the best mix of cached prefix and outstanding work. If predicted TTFT breaks the SLO everywhere, it returns 429 with Retry-After at once.
3. Serve. The engine reuses the cached system-prompt blocks, prefills the rest in chunks alongside running decodes, adds the sequence to the continuous batch and allocates KV blocks as tokens appear. Adapter requests use LoRA weights already resident.
4. Stream. Each token flows back through router and front end as an SSE event. The final event carries usage — prompt, cached and completion tokens — for quotas and showback.
5. Batch. Batch requests wait in a durable queue and enter at low priority when pools have headroom; they are preempted first when interactive load rises.
6. Operate. The pool controller reads queue depth, KV usage and p95 TTFT/TPOT, scales pools from warm nodes and standby replicas, probes every replica with a one-token generation, drains sick ones, and rolls out new versions from the registry as canaries. Weights come from object storage through each node’s NVMe cache.

Check it against the requirements. Latency: prefill of a 1,500-token prompt takes about 130 ms (45 ms with a cached system prompt), chunked prefill keeps TPOT around 30–45 ms with about 60 sequences per replica, and admission control protects the p95. Scale: 49 TP-4 replicas (196 H100s) for 300 requests/s on the 70B, 46 GPUs for the 8B, KV memory a quarter full. Availability: a GPU failure costs one replica’s in-flight requests, and those not yet streaming are retried invisibly. Cost: about $1.51 per million output tokens at peak, with batch work filling the night. Quality: quantized and new versions ship only after evaluation and a canary.

What interviewers look for by level

Follow-up questions

What if a team needs 128K-token context?

Do the KV arithmetic first: 131,072 tokens × 320 KiB = 40 GiB per sequence, so a replica with 138 GB of KV room holds three such requests — and a 128K prefill is 18 PFLOP, over 10 s of the replica’s compute. Serve long context from a separate pool so it cannot starve the interactive one: FP8 KV cache to double capacity, KV offload to CPU memory for idle conversations, prefix caching for the repeated document, and a price per token that reflects the memory held. Often the better answer is retrieval that sends 6K relevant tokens instead of 128K.

What if we have to choose a serving engine?

As of October 2026: vLLM (v0.32), the origin of PagedAttention, has the broadest model and hardware support, with continuous batching, prefix caching, chunked prefill, LoRA, speculative decoding and experimental disaggregation. SGLang (v0.5.x) is strong where prefixes are reused heavily — agents, multi-call programs, few-shot prompts — thanks to RadixAttention; its paper reports up to 6.4× throughput on such workloads. TensorRT LLM (1.x) is NVIDIA’s optimised runtime, often deployed behind Triton Inference Server, NVIDIA’s general model server; it suits all-NVIDIA fleets willing to adopt its tooling. Hugging Face TGI’s documentation now says it is in maintenance mode and recommends vLLM or SGLang, and its repository is archived, so don’t start new deployments on it. Whatever you pick, benchmark goodput on a replay of your own traffic, not peak tokens per second on a synthetic one.

What if GPUs are scarce during a big launch?

Degrade in a planned order: pause the batch tier, cap max_tokens for free-tier callers, route low-stakes traffic from the 70B to the 8B, then shed by priority with 429s. Keep reserved capacity for the products that pay. Product-level degradation — what users see when the model is busy — is covered in Design ChatGPT; here the job is to make sure that priority and preemption exist in the router and engine.

What if we serve a mixture-of-experts model?

A mixture-of-experts model activates only a few experts per token, so compute per token is like a much smaller model, but every expert’s weights must still be in memory. You pay memory like a big model and compute like a small one, and decode needs larger batches to amortise reading the experts. Engines add expert parallelism — different experts on different GPUs — on top of tensor and data parallelism; the rest of this design carries over.

How to say it in the interview

“I’ll design a self-hosted inference service for a 70B and an 8B model: about 300 requests a second at peak on the 70B, averaging 1,500 prompt and 300 output tokens, with p95 TTFT under a second and TPOT under 50 milliseconds, plus a cheap batch tier.

Prefill reads the whole prompt in one pass and is compute-bound; decode makes one token per step, re-reading all 140 gigabytes of weights, so it is memory-bandwidth-bound. Batching is everything: one stream on four H100s costs about 36 dollars per million tokens, a full batch well under 2.

Memory limits the batch. The KV cache costs 320 kilobytes per token, so a TP-4 replica with 138 gigabytes free holds about 420 thousand tokens. I’d run vLLM-style engines: paged KV blocks, continuous batching, chunked prefill so long prompts don’t freeze other streams, and prefix caching for shared system prompts.

Sizing from FLOPs gives about 49 four-GPU replicas. The router scores replicas on cached prefix and outstanding tokens, pins conversations softly, and returns 429 early when predicted TTFT breaks the SLO; batch work fills idle capacity at low priority.

I autoscale on queue depth, KV usage and TTFT — not GPU utilisation — with warm nodes and standby replicas, because a cold 70B replica takes minutes. Health checks run a real one-token generation, and a dead replica’s requests are retried if they haven’t streamed yet. Speculative decoding and disaggregated prefill are for specific latency problems: speculation needs spare compute, and disaggregation needs an RDMA network to move the KV cache.”

Concepts used in this design

• LLM inference serving — the engine-level concepts behind every replica.
• Back-of-envelope estimation — the FLOPs, bytes and Little’s-law arithmetic used for sizing.
• Latency vs throughput vs bandwidth — why decode is a bandwidth problem and batching a throughput tool.
• Reliability: SLIs, SLOs and error budgets — TTFT and TPOT targets and goodput.
• Real-time connections: SSE and WebSockets — streaming tokens to callers.
• Load balancing: L4 vs L7 and consistent hashing — request-aware routing and soft affinity.
• Backpressure and load shedding — admission control and early 429s.
• Noisy neighbours and fairness — fair queuing between teams.
• Autoscaling done right — signals, warm pools and scale-in.
• Canary, blue-green and feature flags — rolling out model versions and quantized builds.
• Idempotency and timeouts, retries and backoff — safe retries after a replica dies.
• Cost-aware design — cost per million tokens as a design metric.
• Design ChatGPT, Design an LLM gateway and Design a GPU job scheduler — the neighbouring systems.

References and further reading

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023)arxiv.orgThe vLLM paper: KV blocks and block tables, the measured 20.4–38.2% effective KV memory in earlier systems, waste bounded to one block, sharing savings, block size 16, and 2–4× throughput.
2. Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022)usenix.orgIteration-level scheduling and selective batching, and the 36.9× throughput gain over FasterTransformer at the same latency on GPT-3 175B.
3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., OSDI 2024)arxiv.orgTTFT and TPOT as separate SLOs, per-GPU goodput, prefill–decode interference, the limits of chunked prefill, and KV transfer bandwidth arithmetic.
4. Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman and Matias, ICML 2023)arxiv.orgDraft-and-verify with identical output distribution, the acceptance rate α, expected tokens per target call (1 − α^(γ+1)) / (1 − α), and the walltime formula.
5. Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)arxiv.orgSpeculative sampling on the 70B Chinchilla with a 4B draft: 2–2.5× faster decoding at batch size 1, and why small-batch sampling is memory-bandwidth bound.
6. SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al.)arxiv.orgRadixAttention: KV reuse in a radix tree with LRU leaf eviction, cache-aware scheduling, and a router that tracks each worker’s cached prefixes.
7. vLLM documentation: Automatic Prefix Caching (design)docs.vllm.aiBlock hashes chained from the parent block, caching of full blocks only, reference counts and eviction, and per-request cache salts for tenant isolation.
8. vLLM documentation: Optimization and Tuningdocs.vllm.aiChunked prefill on by default in V1 with decode-first scheduling, tuning max_num_batched_tokens, preemption by recomputation, and when to use TP, PP and DP.
9. NVIDIA H100 Tensor Core GPU: product specificationsnvidia.comH100 SXM: 80 GB HBM3 at 3.35 TB/s, 989 dense BF16 teraFLOPS (1,979 with sparsity), FP8 support and 900 GB/s NVLink.
10. The Llama 3 Herd of Models (Llama Team, Meta, 2024)arxiv.orgLayer, head and KV-head counts for the 8B and 70B models; serving 405B with TP inside a node and PP across nodes; FP8 inference and its quality pitfalls.