Layer 3 · Go deeper · 11 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How many requests fit on one four-GPU 70B replica?
How many GPUs does the 70B pool need at peak?
What does a million output tokens cost?
Build-up · 5 steps
Build the high-level design
One functional requirement at a time, ending with a service that works before any of the deep dives make it fast and cheap.
Step 1 of 5
Stream from a model
The front end authenticates, checks quota and holds the SSE stream. The router needs live replica state; the replica, an engine on four NVLink-connected GPUs, owns all request state. Weights load from object storage.
6 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 6
Interviewer
“Each token holds 320 KiB of KV cache, and nobody knows how long an answer will be. How do you allocate GPU memory?”
Your move. Pick an approach:
Deep dive 2 of 6
Interviewer
“Requests arrive at random times and finish at random lengths. How do you batch them without hurting latency?”
Your move. Pick an approach:
Deep dive 3 of 6
Interviewer
“The 70B model is 140 GB in BF16, and one H100 holds 80 GB. How do you serve it?”
Your move. Pick an approach:
Deep dive 4 of 6
Interviewer
“A teammate wants speculative decoding, an 8B draft checked by the 70B, on every pool. Where do you turn it on?”
Some pools run busy, with batches of 128. Others, like code completion at 2 a.m., are lightly loaded.
Your move. Pick an approach:
Deep dive 5 of 6
Interviewer
“Replicas hold different cached prefixes and free KV memory, and requests differ 100× in cost. How does the router choose?”
Your move. Pick an approach:
Deep dive 6 of 6
Interviewer
“Evening traffic is climbing, and a new 70B replica takes minutes to start. How do you scale the pool?”
Your move. Pick an approach:
Build-up · 6 steps
Follow one support-bot request
The data path is front end, router and GPU pools. The control plane decides how many replicas exist and whether each one is healthy.
Step 1 of 6
Request
The app calls the chat-completions API with stream set to true. The front end authenticates the team, checks its token quota, counts prompt tokens and opens the SSE stream.