Layer 3 · Go deeper · 11 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How many gateway pods carry the peak?
How much would storing every chat’s prompt and response add up to?
What does a month of chat cost at list prices?
Build-up · 6 steps
Build the high-level design
One functional requirement at a time, ending with a gateway that works in one region at moderate traffic.
Step 1 of 6
One API, any model
Apps send an OpenAI-style request with an alias like chat-default. Stateless pods resolve it from an in-memory alias table, and one adapter per provider translates both ways, so swapping a model never touches 300 codebases.
6 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 6
Interviewer
“A request’s cost is known only when its stream ends, seconds after you said yes. How do you rate-limit by tokens?”
Your move. Pick an approach:
Deep dive 2 of 6
Interviewer
“Rupee budgets span 18 pods, and one batch job sends 2,000 requests a second on one key. Where does the budget counter live?”
Your move. Pick an approach:
Deep dive 3 of 6
Interviewer
“Provider B is having a bad hour: 429s, overloads and 40-second waits for a first token. How does the router fail over?”
Your move. Pick an approach:
Deep dive 4 of 6
Interviewer
“Users expect the first words in a second, not a whole answer after ten. How does the gateway pass a stream through?”
Your move. Pick an approach:
Deep dive 5 of 6
Interviewer
“Support bots and KYC summarisers resend the same long system prompt on every call. What should the gateway cache?”
Your move. Pick an approach:
Deep dive 6 of 6
Interviewer
“Budgets need usage in seconds, showback in minutes, and finance a monthly chargeback that matches the invoices. How do you record usage?”
Your move. Pick an approach:
Build-up · 7 steps
Follow one KYC request through it
The hot path is the gateway pods, Redis, the guardrails and the chosen deployment. Everything else is asynchronous or pushed ahead of time. Follow one KYC-summary request.
Step 1 of 7
Arrive
The app calls POST /v1/chat/completions with model: chat-default and its virtual key. The load balancer picks any pod in any zone.