$36
per million tokens, one request at a time
Assume $2.50 an H100-hour. One stream on four GPUs makes 76 tokens a second. Batch 128 and it’s $0.42, each step 19.5 ms instead of 13.1.
Each decode step re-reads all 140 GB of weights: 1 FLOP per byte. An H100 can do 295.
The trick
So batch: extra sequences ride that read almost free, joining and leaving at every step.
320 KiB
of KV cache per token, per request
Weights fill 140 GB of four 80 GB H100s. About 138 GB is left for KV, roughly 420,000 tokens. How many requests is that?
Where does KV memory go?
Bad: Reserve the 8K maximum
Most slots sit empty or stranded in gaps. Only 51 requests fit.
Good: 16-token blocks, on demand
Each request wastes under one block, so 233 fit.
A 6,000-token prompt lands while 128 answers stream.
Prefilled in one step, it freezes them all for over half a second. Sliced into a 512-token budget per step, decodes first, tokens keep coming every ≈ 45 ms.
Which replica gets the next support-bot request?
Not the emptiest. R2 already holds the shared 1,024-token system prompt and has room, so prefill drops from 130 ms to 45. Routing is cache placement.
A cold 70B replica needs ≈ eight minutes
- Scale on queues, KV and TTFT, not GPU utilisation
- Keep warm nodes with weights on NVMe
- Park standby replicas in host memory
- Scale out 15 minutes ahead of demand
Remember
Decode is memory-bound, so batch hard with paged KV, continuous batching and chunked prefill. Route on cached prefix and load, reject early, keep replicas warm.