Layer 3 · Go deeper · 9 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How many 4K-token chats fit on a four-GPU 70B replica?
How many GPU-hours does a million output tokens take?
Build-up · 6 steps
Follow one chat through a replica
A 2,000-token prompt joins a replica already decoding three chats. Watch where its KV cache lives, when its first token appears, and what happens when memory runs out.
Step 1 of 6
Wait for the next step
A, B and C are mid-answer. The scheduler works one decode step at a time, so request D, a 2,000-token prompt, waits only until the next step, not until the whole batch finishes.
4 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 4
Interviewer
“Eight replicas serve a chat app, and each turn resends the whole history. How do you spread requests across them?”
Your move. Pick an approach:
Deep dive 2 of 4
Interviewer
“Evening traffic doubles and TTFT climbs across your GPU fleet. What should the autoscaler watch?”
Your move. Pick an approach:
Deep dive 3 of 4
Interviewer
“A RAG feature sends 8,000-token prompts to a 70B model on four GPUs. TTFT p95 must stay under 500 ms. What do you do?”
Your move. Pick an approach:
Deep dive 4 of 4
Interviewer
“A startup launches a support chatbot. Traffic is modest and spiky. Do you self-host an open model, or call an API?”
Your move. Pick an approach:
Build-up · 6 steps
Follow a request through the fleet
Self-hosting is more than one replica. Follow a chat request through the gateway, router and GPU pools, then see what scales the fleet and what catches overflow.
Step 1 of 6
Gateway: auth and token quotas
The gateway authenticates, costs each request as prompt tokens plus max_tokens, and checks the tenant’s tokens-per-minute quota in Redis. Over quota, it rejects early with 429 and Retry-After instead of queueing past the TTFT target.