Prefill: the whole prompt, one pass. Decode: one token per step.
Prefill is compute-bound and sets TTFT. Each decode step re-reads every weight, so it is bandwidth-bound and sets TPOT.
One request at a time?
Bad: Batch of one
Each token re-reads 140 GB: 42 ms on one GPU’s bandwidth, ≤ 24 tokens/s.
Good: Share the weight read
64 sequences share each read: nearly 64× the throughput.
320 KiB
of KV cache per token, Llama 3 70B
A 4K-token chat holds 1.3 GB. The 70B needs four GPUs to hold about as many chats as the 8B holds on one.
Continuous batching: a freed slot refills on the next step
Static batches idle until the longest answer ends. Orca schedules per step, for 36.9× FasterTransformer’s throughput at the same latency.
140 GB won’t fit one 80 GB GPU. Split every layer.
Tensor parallelism inside one NVLink server: each GPU holds and reads 35 GB, 10.4 ms a step. Replicas add throughput.
Temperature 0, the same prompt, sent twice. You expect identical tokens.
The catch
Batch-mates reorder floating-point additions, so tokens can differ. vLLM doesn’t promise reproducible results by default. Need the same answer again? Cache it.
Self-host, or call an API?
- Start with an API: no GPUs, no on-call
- Self-host for steady volume that fills batches
- Or for fine-tuned models, data rules, strict latency
- Keep the API as overflow
Say it like this
Decode re-reads every weight, so batch: vLLM, continuous batching, paged KV. Promise TTFT p95 under 500 ms, TPOT under 50 ms. Self-host for steady volume or private data; else an API.