LLM inference serving

Serving a large language model means running billions of weights on GPUs so that many users can stream answers at once, at a sane cost per token. The hard part is not HTTP: answers come out one token at a time, each token re-reads the whole model from GPU memory, and each conversation holds gigabytes of state. In an interview you need the two phases, the KV-cache arithmetic, the latency metrics and the tricks that keep a GPU busy.

Picture it first: a wedding lunch on banana leaves

At a wedding lunch in Madurai, guests sit in long rows, a banana leaf before each. Servers walk the row with buckets of rice and sambar, dropping one ladle on every leaf they pass. A walk takes about as long for six guests as for sixty, so the caterer fills every seat. A new guest’s leaf is laid in one go with the same starters for everyone: salt, pickle, chips. Servers glance at each leaf to judge what it needs next. When a guest gets up, a new one sits down at once; nobody waits for the row to finish. The limit is not how fast servers walk but how many leaves fit in the hall.

Map it back: a walk down the row is one decode step, a ladle is a token, the buckets carried the full length of the row are the weights read on every step, the row of guests is the batch, laying the leaf is prefill, what sits on a leaf is that guest’s KV cache, the hall is GPU memory, the common starter set is a shared prefix, and seating a guest mid-meal is continuous batching.

How a token gets made: prefill and decode

Models read and write tokens, chunks of text averaging about four characters of English (3.94 for the Llama 3 tokenizer). Generation is autoregressive: the model reads everything so far, predicts one token, appends it and repeats. A 300-token answer is 300 passes. Every request has two phases:

• Prefill. All prompt tokens go through the model in one parallel pass of big matrix multiplies, so it is compute-bound. It writes the KV cache for the prompt and emits the first token: queueing plus prefill is the time to first token (TTFT).
• Decode. Each step feeds in the newest token, attends to the cached keys and values of all earlier ones, and emits one more. Every weight is fetched from GPU memory to do just two floating-point operations per sequence, so it is memory-bandwidth-bound; the step time is the time per output token (TPOT).

The arithmetic that explains everything

A forward pass costs about 2 FLOPs per parameter per token: one multiply and one add per weight. Take a 70-billion-parameter model in FP16, 2 bytes per weight, on NVIDIA H100 SXM GPUs (80 GB of HBM3 at 3.35 TB/s, about 989 dense FP16 TFLOPS; the 1,979 on the spec sheet assumes sparsity):

• Weights: 70 × 10⁹ × 2 B = 140 GB, more than one GPU holds.
• Prefill of a 2,000-token prompt: 2 × 70 × 10⁹ × 2,000 = 280 TFLOP. Four GPUs offer 3,956 TFLOPS; at half of peak that is about 0.14 s.
• Decode at batch 1: reading 140 GB at 3.35 TB/s takes 42 ms, so even if the weights fitted, one sequence would get at most 1 ÷ 0.042 ≈ 24 tokens/s while the arithmetic units idled.

The way out is batching. One read of the weights serves every sequence in the batch, so 64 sequences cost almost the same weight traffic as one, for nearly 64 times the throughput. An H100 balances at 989 ÷ 3.35 ≈ 295 FLOPs per byte, and FP16 decode does about one FLOP per weight byte per sequence, so batches into the hundreds stay bandwidth-bound. What stops you sooner is the KV cache: each sequence reads its own every step, and each needs memory.

The KV cache

Attention lets each new token look back at earlier tokens through their keys and values, vectors computed in every layer. Recomputing them each step would make decode quadratic, so the server keeps them: the KV cache.

KV bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value

With the Llama 3 paper’s architecture table (head dimension = model width ÷ attention heads = 128) and FP16 values:

• Llama 3 8B (32 layers, 8 KV heads): 2 × 32 × 8 × 128 × 2 B = 128 KiB per token, so an 8,192-token conversation holds 1 GiB.
• Llama 3 70B (80 layers, 8 KV heads): 320 KiB per token: 2.5 GiB at 8K tokens and 40 GiB for one sequence at the full 128K context.

Those 8 KV heads are grouped-query attention (GQA): the 70B model’s 64 query heads share 8 key/value heads, which Meta chose to shrink the cache. One KV head per query head would mean 2.5 MiB per token; multi-query attention goes down to one. An FP8 cache halves it again, at some accuracy risk. The cache, not compute, sets how many requests run at once, as Fig. 2 shows.

Paged KV memory

Early servers reserved one contiguous slab per request, sized for the maximum length (say 2,048 tokens), since nobody knows how long an answer will be. Short answers left slabs mostly empty and memory fragmented: the vLLM paper (Kwon et al., SOSP 2023) measured that only 20.4–38.2% of KV memory in such systems held real token states.

PagedAttention borrows virtual memory from operating systems. The cache is cut into fixed-size blocks (16 tokens by default), each sequence has a block table mapping logical blocks to physical blocks anywhere in GPU memory, and a new block is allocated only when the last one fills, so waste is at most one part-filled block per sequence. Blocks carry reference counts and are copied on write, so several samples of one prompt, or requests with the same system prompt, share blocks; beam search saved 37.6–55.2% of memory in the paper. vLLM delivered 2–4× the throughput of FasterTransformer and Orca at the same latency.

When the cache still runs out, the scheduler preempts a sequence, freeing its blocks to swap back from CPU memory or recompute later in one prefill. vLLM’s V1 engine recomputes by default and logs a warning; frequent preemption means you need more KV memory.

What you can promise: TTFT, TPOT and throughput

An LLM endpoint has no single latency. Promise these instead:

• TTFT: arrival to first token; queueing plus prefill, so it grows with prompt length and load.
• TPOT, or inter-token latency (ITL): the gap between streamed tokens, about one decode step; it grows with batch size and with prefills sharing the step.
• End-to-end latency = TTFT + TPOT × (output tokens − 1).
• Throughput: output tokens per second per GPU, the number that sets cost.
• Goodput: DistServe’s term for the highest request rate a GPU sustains while, say, 90% of requests meet both TTFT and TPOT targets.

A chat SLO might be TTFT p95 under 500 ms and TPOT under 50 ms. 50 ms per token is 20 tokens/s. DistServe puts reading speed at 250 words per minute, about 4.2 words/s, or 5–6 tokens/s at roughly 1.3 tokens per word, so 20 tokens/s stays about four times ahead of the reader. A 400-token answer still takes 0.5 + 0.05 × 399 ≈ 20 s end to end, so every chat product streams, usually over server-sent events through proxies that flush rather than buffer.

Bigger batches raise throughput and TPOT together, so set the SLO per product (SLIs, SLOs and error budgets) and run the largest batch that meets it.

Batching: static, continuous and chunked

Static batching collects a batch, runs it until the longest sequence finishes, then takes the next. A one-line reply sits in a slot doing nothing while a 2,000-token essay finishes, and new requests wait outside. Orca (Yu et al., OSDI 2022) fixed this with iteration-level scheduling: the scheduler runs one step at a time, so a finished sequence leaves after any step and a waiting one joins at the next. Orca reported 36.9× the throughput of NVIDIA FasterTransformer on GPT-3 175B at the same latency. Modern engines all work this way, under the name continuous batching.

A joining request needs its prefill, and a 6,000-token prefill inside one step stalls every running sequence for that step: a TPOT spike for everyone. Chunked prefill (Sarathi-Serve, Agrawal et al., 2024) splits long prefills into pieces that ride along with decode steps under a per-step token budget. vLLM’s V1 engine enables it by default and schedules all pending decodes before any prefill chunk. The budget, max_num_batched_tokens, is the knob: small values (vLLM suggests 2,048 as an example) protect inter-token latency, large ones (above 8,192) favour TTFT and throughput.

Fitting the model: memory and parallelism

The memory budget is weights + KV cache + runtime overhead, and only what the weights leave is useful for batching. Two levers make room.

Quantisation stores weights in fewer bits: a 70B model is 140 GB in FP16, 70 GB in FP8 and 35 GB at 4 bits, and fewer bytes also mean faster decode. Weight-only 4-bit methods such as AWQ (MLSys 2024) keep activations in 16 bits and protect the ~1% of weights that matter most. You gain memory and speed; you pay in quality, which benchmarks can hide: Meta’s FP8 Llama 3 405B matched BF16 on standard benchmarks yet occasionally produced corrupted responses until they capped the scaling factors. Evaluate on your own traffic.

Parallelism spreads the model over GPUs:

• Tensor parallelism (TP), the Megatron-LM scheme (Shoeybi et al., 2019), gives each GPU a slice of every layer’s matrices and combines the slices with all-reduces inside each layer. That chatter needs NVLink (900 GB/s on H100 SXM), so TP stays inside one server. It also divides the weight read: at TP = 4 each GPU reads 35 GB, about 10.4 ms per step instead of 42.
• Pipeline parallelism (PP) gives each GPU or node a range of layers, so only activations cross the slower link between nodes. Meta served Llama 3 405B in BF16 on two 8-GPU machines: tensor parallel within each, pipeline parallel across. You pay in pipeline bubbles.
• Expert parallelism (EP) spreads a mixture-of-experts model’s experts over GPUs; each token visits only a few.
• Data parallelism means replicas: whole copies behind a router, for throughput once one copy fits.

Making decode faster

Speculative decoding (Leviathan, Kalman and Matias, ICML 2023) attacks the one-token-per-step limit. A small draft model proposes k tokens; the big model checks all k in one pass, which costs about one decode step since decode is bandwidth-bound; a modified rejection-sampling rule accepts a prefix of them, so the output distribution is exactly the big model’s. If each draft token is accepted with probability α, one target pass yields on average (1 − α^(k+1)) ÷ (1 − α) tokens: with α = 0.8 and k = 4, 3.4 tokens per pass. The paper measured 2–3× on T5-XXL with identical outputs. You pay with draft compute and rejected work, and the gain shrinks at large batches; it shines on predictable text such as code.

Prefix caching reuses KV cache that an earlier request computed for the same prefix: a system prompt, tool definitions, earlier chat turns. vLLM hashes each full 16-token block together with everything before it, so a block matches only if the whole prefix does; it is on by default, and per-tenant cache salts keep tenants’ blocks apart. SGLang’s RadixAttention keeps finished requests’ caches in an LRU radix tree and runs waiting requests with the longest shared prefix first, for up to 6.4× throughput on multi-call workloads. You gain TTFT and prefill compute; you pay with memory, and prompts must be byte-stable: fixed instructions first, variable parts (time stamps, names) last.

FlashAttention (Dao et al., 2022) computes exact attention in tiles held in on-chip SRAM, cutting traffic to HBM, which speeds up long contexts without approximation.

Serving at fleet scale

Routing replicas

A round-robin load balancer fits replicas badly. Requests differ 100-fold in cost, so balance on outstanding tokens (queued prompt plus expected output), not request counts. And the KV cache is local to a replica, so send requests sharing a prefix to the same one: hash the system prompt or session ID with consistent hashing, and spill to the least-loaded replica when the owner is saturated. OpenAI documents exactly this for its prompt cache: caches live on individual machines, requests are routed by a hash of their first tokens, and beyond about 15 requests a minute for one prefix, traffic overflows to machines without it.

Disaggregated prefill and decode

Prefill and decode interfere: a burst of long prompts stalls decodes, and one parallelism plan cannot suit both. Disaggregation runs them on separate GPU pools and ships the KV cache between them. DistServe (Zhong et al., OSDI 2024) served 7.4× more requests, or met 12.6× tighter SLOs, than colocated systems; Splitwise (Patel et al., 2023) reported 1.4× throughput at 20% lower cost. The price is the transfer: a 2,000-token prompt on the 70B model carries 2,000 × 320 KiB ≈ 0.66 GB of cache, so 10 requests a second need about 52 Gbit/s between pools. vLLM still calls its support experimental, aimed at tail inter-token latency rather than throughput; NVIDIA’s Dynamo (1.6, October 2026) runs vLLM, SGLang or TensorRT-LLM workers as prefill and decode pools with KV-aware routing.

Admission, autoscaling and cold starts

• Admit by tokens. Cost each request as prompt tokens plus max_tokens, enforce per-tenant tokens-per-minute quotas (rate limiting), and reject early with 429 and Retry-After rather than queue past the TTFT target (backpressure and load shedding). Priority tiers let interactive traffic jump batch jobs; vLLM’s scheduler supports priorities.
• Scale on the right signal. CPU is idle, and GPU “utilisation” only says a kernel was running. Scale on waiting requests, KV-cache usage and TTFT p95, which vLLM exports as vllm:num_requests_waiting, vllm:kv_cache_usage_perc and vllm:time_to_first_token_seconds (autoscaling done right).
• Respect cold starts. A new 70B replica pulls 140 GB of weights (70 s at 2 GB/s), loads four GPUs and captures CUDA graphs: minutes, not seconds. Keep warm headroom and scale early; scale-to-zero suits only traffic that can wait.
• Share one base model. Multi-LoRA serving puts many small fine-tuned adapters on one base model, chosen per request, instead of a replica per customer.

The gateway and the cost per token

Put an LLM gateway, a specialised API gateway, in front: it authenticates, counts tokens per tenant in Redis, caches exact repeats, writes usage events to Kafka for billing, caps max_tokens, and falls back to a smaller model or another provider when the fleet is full (design an LLM gateway).

Cost per million output tokens = GPU price per hour ÷ (tokens/s per GPU × 3,600) × 10⁶. At batch 1 our 70B replica makes at most 24 tokens/s per GPU, 86,400 per GPU-hour: 11.6 GPU-hours per million tokens. If a benchmark at your SLO shows 2,400 tokens/s for the replica (600 per GPU), that is 2.16 million per GPU-hour: 0.46 GPU-hours per million, 25 times cheaper. Batching is the cost story (cost-aware design).

The serving stacks

As of October 2026, vLLM (0.32) and SGLang (0.5) are the open-source defaults, both with continuous batching and prefix caching; TensorRT-LLM (1.2) is NVIDIA’s own engine. Hugging Face’s TGI is in maintenance mode with its repository archived, and Hugging Face now recommends vLLM and SGLang. Dynamo coordinates engines across nodes, and llama.cpp runs models locally.

Limits and numbers

Hardware figures are NVIDIA’s; model figures are from the Llama 3 paper; speed-ups are each paper’s own, against its own baseline.

Self-host or call an API

Start with an API: no GPU planning, no on-call, quality you could not train. Self-host when volume is steady enough to keep batches full, you need a fine-tuned or open model, data may not leave your network, or you need latency no shared fleet will promise. Many teams do both behind one gateway, with the API as overflow.

Common interview usage

Going deeper: failure modes

• KV-cache exhaustion. The cache fills, sequences are preempted, and recomputing them burns GPU time exactly when you have none, so latency spikes in waves. Cap concurrency (max_num_seqs), alert on preemptions and KV usage, and shed load at the gateway.
• Long prompts starving decode. One 30,000-token paste stalls every stream on the replica. Use chunked prefill with a modest budget, cap prompt length per tier, or give long prompts their own pool.
• Noisy neighbours. A tenant asking for 8,000-token outputs holds slots for minutes. Enforce max_tokens per tier, count tokens in quotas, and schedule fairly (noisy neighbours and fairness).
• Out of memory at start-up. A 70B model configured for 128K contexts needs 40 GiB of KV for one sequence; if that does not fit beside the weights, the engine refuses to start. Size the context to the product.
• Retry storms. Clients retrying 429s in lock-step multiply load on a saturated fleet. Return Retry-After, use backoff with jitter, and fail over to a fallback model (retry storm).
• Autoscaling lag. Spikes arrive in seconds, replicas in minutes. Keep warm spares and degrade meanwhile: a smaller model, shorter outputs, queued batch work.
• Cost runaway. Unbounded outputs, looping agents and full regenerations all bill by the token. Cap steps and tokens; alert on spend per tenant.
• Silent prefix-cache misses. A time stamp at the top of the system prompt changes every prefix and the hit rate falls to zero. Track hit rate as an SLI.

Cheat sheet

How to say it in the interview

“Prefill pushes the prompt through in one compute-bound pass and sets time-to-first-token; decode makes one token per step and re-reads every weight, so it is bandwidth-bound: about 24 tokens a second for a 70B model at batch 1. So the game is batching. I’d run vLLM with continuous batching and paged KV memory on four-GPU tensor-parallel replicas; at 320 KiB of KV per token, each holds roughly 100 four-thousand-token chats. I’d target TTFT p95 under 500 ms and TPOT under 50 ms, stream over SSE, put a gateway in front for token quotas and failover, route sessions to the replica holding their prefix, and autoscale on queue depth and KV usage with warm spares, because a cold replica takes minutes.”

Why is decode slow, and why does batching help? Each step reads all the weights for one token per sequence, leaving the arithmetic units idle. A batch shares that read, so 64 sequences cost about the same weight traffic as one, until compute or KV memory runs out.

How many concurrent users can one replica serve? Divide free KV memory by KV per sequence. On a 70B model at TP = 4, about 132 GB of KV space ÷ 1.34 GB per 4K-token sequence ≈ 98 sequences; then check that TPOT at that batch still meets the SLO.

How do you keep TTFT low with long RAG prompts? Retrieve less but better, cache the fixed instructions as a prefix, use chunked prefill, and add prefill capacity, in its own pool if prefill dominates.

What do you autoscale on? Waiting requests, KV-cache usage and TTFT p95. CPU says nothing, and a new replica takes minutes to load weights, so scale early and keep spare capacity.

Would you self-host? Not at first. I’d self-host for steady high volume, a fine-tuned open model, data that must stay inside, or a latency SLO a shared fleet won’t sign, keeping the API as overflow behind the same gateway.

Where to go next

• Design an LLM inference service: this page turned into a full design.
• Design ChatGPT and design an LLM gateway: streaming, sessions, quotas and failover.
• Design RAG search over company documents and design an AI agent platform: prefill-heavy and many-call workloads.
• Embeddings and semantic search and vector databases: the retrieval half of RAG.
• Latency vs throughput vs bandwidth: the trade-off behind every batching decision.
• Rate limiting and design a rate limiter: token-based quotas.
• Autoscaling done right and cost-aware design: running GPU fleets without waste.
• Caching 101: prefix and response caching are caches with the usual hit-rate economics.

References and further reading

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023)arxiv.orgKV blocks and block tables, 20.4–38.2% effective KV memory in earlier systems, 16-token default blocks, copy-on-write sharing, swap versus recompute, and 2–4× throughput over FasterTransformer and Orca.
2. Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022)usenix.orgIteration-level scheduling and selective batching, and the 36.9× throughput gain over FasterTransformer on GPT-3 175B at the same latency.
3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., OSDI 2024)arxiv.orgTTFT and TPOT, per-GPU goodput under an SLO, the 250 words-per-minute reading speed, KV transfer bandwidth, and 7.4× more requests or 12.6× tighter SLOs.
4. Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman and Matias, ICML 2023)arxiv.orgDraft-then-verify decoding with an unchanged output distribution, and the 2–3× speed-up on T5-XXL.
5. SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)arxiv.orgRadixAttention: an LRU radix tree of KV cache reused across requests, cache-aware scheduling, and up to 6.4× throughput.
6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., 2019)arxiv.orgIntra-layer (tensor) model parallelism implemented with a few communication operations per layer.
7. The Llama 3 Herd of Models (Llama Team, Meta, 2024)arxiv.orgLayer, head and KV-head counts for 8B and 70B, GQA, 128K context, 3.94 characters per token, FP8 inference and its hidden quality risk, and two-node pipeline parallelism for 405B.
8. NVIDIA H100 Tensor Core GPU: product specificationsnvidia.comH100 SXM: 80 GB of HBM3 at 3.35 TB/s, 1,979 FP16 tensor TFLOPS with sparsity, and 900 GB/s NVLink.
9. vLLM documentation: Optimization and tuning (v0.32, October 2026)github.comPreemption by recompute in V1, chunked prefill on by default with decode-first scheduling, max_num_batched_tokens trade-offs, and tensor, pipeline, expert and data parallelism.
10. OpenAI API documentation: Prompt cachingdevelopers.openai.comA provider’s own description of KV-prefix caching: exact prefix matches, a 1,024-token minimum, discounted cached input, and routing by a hash of the first tokens to machine-local caches.