Layer 3 · Go deeper · 10 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How many GPUs does that peak need?
How many answers are streaming at any moment?
What does one answer cost in GPU time?
Build-up · 6 steps
Build the high-level design
One functional requirement at a time, ending with a design that works before any scaling.
Step 1 of 6
Send and watch it stream
The chat API saves the message and hands the work to an orchestrator, which builds the prompt and calls the fleet. Tokens flow back as SSE events on the same open response.
6 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 6
Interviewer
“Phones drop off Wi-Fi mid-answer, and the API tier gets redeployed. Where should each generation live?”
Your move. Pick an approach:
Deep dive 2 of 6
Interviewer
“A conversation hits 60 turns, and every turn re-sends its history. How do you build each prompt?”
Your move. Pick an approach:
Deep dive 3 of 6
Interviewer
“Assume 175,000 writes a second at peak and 5 TB of turns a day. Where do conversations live?”
Your move. Pick an approach:
Deep dive 4 of 6
Interviewer
“It’s 9:15 pm IST on launch day, and demand is 30% above GPU capacity for hours. Who waits?”
Your move. Pick an approach:
Deep dive 5 of 6
Interviewer
“Moderation scores for streamed output arrive only once the full answer exists. How do you moderate a 13-second stream?”
Your move. Pick an approach:
Deep dive 6 of 6
Interviewer
“About 14 replicas fail an hour, each cutting 80 answers off mid-sentence. How do you recover?”
Your move. Pick an approach:
Build-up · 7 steps
Follow one message through it
The connection, the durable record and the expensive work are separate tiers, joined by the token log. Follow one message through them.
Step 1 of 7
Send
The client posts the message with an idempotency key. The edge routes it to any stream gateway, which holds the SSE response.