Design an LLM gateway

An LLM gateway is the one door every internal app walks through to reach a language model. Picture a Bengaluru-headquartered fintech whose 300 internal teams and apps — a support chatbot, a KYC document reader, a coding assistant, nightly ticket taggers — call two commercial model APIs, a cloud-hosted model service and the company’s own self-hosted open-weights fleet. Without a gateway, every team holds provider keys, writes its own retries, sends customer data wherever it likes and learns about the bill at month end. The gateway looks like a reverse proxy, but an unusual one: request prices vary a thousandfold, you learn the price only after the response has finished streaming, and the upstreams are other companies with their own limits and outages.

Understand the problem

Say first why this is not just an API gateway. A normal gateway counts requests that cost about the same; here a 50-token classification and a 100,000-token contract summary are both “one request”. Responses stream for 5 to 60 seconds, and the upstreams are other companies’ APIs, each with its own limits, errors, streaming format and prices.

Functional requirements

1. Apps should call any approved model through one OpenAI-compatible API — chat completions (streamed or not) and embeddings — by a stable alias such as chat-default, not a provider model ID.
2. Admins should issue virtual keys per team and app with policies: allowed aliases, the most sensitive data class the key may send, and its limits.
3. Teams should be held to token-aware rate limits and monthly rupee budgets, with clear errors when they run out.
4. Requests should be routed, retried and failed over across providers and regions, so one provider’s bad hour is not everyone’s outage.
5. Every request should produce usage, cost and audit records, with optional caching and guardrails (PII redaction, policy checks).

Non-functional requirements

• Low overhead. Admission adds at most 20 ms at p99 (about 5 ms at p50), excluding the provider’s time. For streams that is all we add to time to first token; chunks are then relayed without buffering.
• More available than any one provider. The gateway targets 99.99% (about 4 minutes in a 30-day month); an alias backed by two providers targets 99.95% end to end.
• Throughput. 5,000 requests/s at a sale-day peak, 1,000 on an average day.
• Budget accuracy. Hard budgets overshoot by at most 1%; token-per-minute limits hold within about 5%.
• Fast control plane. Key, price and routing changes are live on every pod within 30 seconds.
• Privacy. Usage and audit metadata kept 13 months; prompt and response bodies stored only if a team opts in (30 days, redacted, encrypted); restricted data never leaves the self-hosted fleet.

Out of scope

• Training and fine-tuning.
• The model servers. Batching, KV caches and GPUs belong to the LLM inference service; to us that fleet is one more upstream.
• Prompt management UI — templates, versioning, evaluation dashboards.
• End-user products. Chat storage and resumable streams are Design ChatGPT; agent loops are the AI agent platform; retrieval is RAG search over company documents.

The set-up

Plan the approach

Say the shape first: “a stateless proxy that authenticates, admits, routes and relays; Redis for counters; Kafka for usage; a control plane that pushes config snapshots.” Build it requirement by requirement, then spend the time in the deep dives — nearly all of which are about tokens: counting them before you know them, paying for them, and not adding latency.

Core entities

• Team. Owns budgets and receives the showback report: kyc-ops, support-bot.
• Virtual key. What an app sends instead of a provider key: one team, one app, allowed aliases, a data-class ceiling, RPM and TPM limits, an expiry. Stored only as a hash.
• Model alias. The stable name apps use, mapped to weighted deployments plus the capabilities each must have.
• Deployment. One place a model runs: provider and account, region, pinned model ID, context window, capabilities, prices per 1M input, cached-input and output tokens, and the provider’s own limits.
• Route policy. Retries, fallbacks, timeouts, cache and guardrail settings per alias.
• Budget. Rupees per team per month (or per key, or per agent run), with a soft alert and a hard stop.
• Usage record. One per attempt: IDs, key, team, alias, deployment, token counts by type, cost, price version, latency, status.
• Request log. The audit trail; bodies only on opt-in.

As config, an alias looks like this:

alias: chat-default
needs: [tools, json_schema]           # every candidate must support these
fallback_alias: chat-small            # last resort, if the caller allows it
deployments:
  - id: b-mid-mumbai
    provider: provider-b
    account: bu-payments              # provider workspace per business unit
    model: mid-2026-08                # pinned, never "latest"
    region: ap-south-1
    context_tokens: 200000
    capabilities: [tools, json_schema, vision]
    data_classes: [public, internal, confidential-redacted]
    price_usd_per_1m: { input: 2.00, cached_input: 0.20, output: 10.00 }
    provider_limits: { rpm: 10000, input_tpm: 10000000, output_tpm: 2000000 }
    weight: 60
  - id: a-mid-mumbai
    provider: provider-a
    # … same fields …
    weight: 40

API

The data plane speaks the OpenAI chat-completions dialect, because almost every SDK and framework already does; teams change a base URL and a key, nothing else. Gateway extras travel as x-gw-* headers so the body stays compatible.

POST /v1/chat/completions
Authorization: Bearer <virtual-key>
Content-Type: application/json
x-request-id: 7c1e2b
x-gw-data-class: confidential
x-gw-cache: off

{
  "model": "chat-default",
  "messages": [
    { "role": "system", "content": "You summarise KYC documents for reviewers. ..." },
    { "role": "user",   "content": "Summarise this address proof: ..." }
  ],
  "max_completion_tokens": 600,
  "temperature": 0.2,
  "stream": true,
  "stream_options": { "include_usage": true }
}

model is an alias. max_completion_tokens matters more than usual: it is what the gateway reserves at admission, so a missing value falls back to a per-alias default, not the model’s maximum. Response headers say where the request went and what quota is left; the x-ratelimit-* names mirror a major provider’s, so client libraries already understand them:

HTTP/1.1 200 OK
Content-Type: text/event-stream
x-request-id: 7c1e2b
x-gw-deployment: b-mid-mumbai
x-ratelimit-limit-requests: 600
x-ratelimit-remaining-requests: 571
x-ratelimit-limit-tokens: 200000
x-ratelimit-remaining-tokens: 187424
x-ratelimit-reset-tokens: 4s
x-gw-budget-remaining-inr: 37600

The stream uses one schema whichever provider served it. With include_usage, the last chunk before [DONE] has empty choices and the token counts; the gateway adds its own cost block.

data: {"id":"gw-7c1e2b","object":"chat.completion.chunk","model":"chat-default","choices":[{"index":0,"delta":{"content":"The document"}}]}

data: {"id":"gw-7c1e2b","object":"chat.completion.chunk","model":"chat-default","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"gw-7c1e2b","object":"chat.completion.chunk","model":"chat-default","choices":[],
       "usage":{"prompt_tokens":1874,"completion_tokens":212,"total_tokens":2086,
                "prompt_tokens_details":{"cached_tokens":1536}},
       "x_gateway":{"deployment":"b-mid-mumbai","cost_inr":0.27,"price_version":17}}

data: [DONE]

Check the cost: 338 uncached input tokens × $2 per million + 1,536 cached tokens × $0.20 per million + 212 output tokens × $10 per million = $0.0031, or ₹0.27 at an assumed ₹88 per dollar. When a limit says no, the gateway answers before any stream opens:

HTTP/1.1 429 Too Many Requests
Retry-After: 1
x-ratelimit-limit-tokens: 200000
x-ratelimit-remaining-tokens: 1180
x-ratelimit-reset-tokens: 60s
Content-Type: application/json

{"error":{"type":"rate_limit_error","code":"key_tokens_per_minute",
  "message":"This key has 1,180 tokens/min left; the request needs 2,474 (1,874 input + 600 max output)."}}

A spent budget is also a 429, with code budget_exhausted and no Retry-After, because waiting will not help — the convention one major provider uses for its monthly spend cap. POST /v1/embeddings and GET /v1/models (only the aliases this key may use) follow the OpenAI shapes. The admin API is off the hot path:

POST /admin/v1/keys                    team, app, aliases, data-class ceiling, rpm, tpm, expiry → key shown once
POST /admin/v1/keys/vk_8f2a:revoke     live on every pod within 30 s
PUT  /admin/v1/budgets/kyc-ops         monthly ₹5,00,000 · alert at 80% · hard stop at 100%
PUT  /admin/v1/aliases/chat-default    deployments, weights, needs, fallback alias, timeouts
PUT  /admin/v1/prices/v18              per deployment: input, cached input, output per 1M; valid_from
GET  /admin/v1/usage?team=kyc-ops&group_by=alias,day

Back-of-envelope estimates

These numbers decide pods, provider contracts and how scary a budget bug is (method: back-of-envelope estimation).

Requests. 1,000/s on average × 86,400 s = 86.4M requests a day, 60% chat and 40% embeddings. At the 5,000/s peak: 3,000 chat and 2,000 embedding requests a second.

Tokens. Assume a chat request carries 1,000 input tokens and returns 250; an embedding request carries 400. At peak: 3,000 × 1,000 × 60 = 180M input tokens/min and 3,000 × 250 × 60 = 45M output tokens/min for chat, plus 2,000 × 400 × 60 = 48M tokens/min of embeddings. One provider’s highest self-serve tier currently allows 10M input and 2M output tokens a minute for a mid-tier model, per organization — peak chat needs 18 times that. So load must spread across providers, regions and the self-hosted fleet, and the gateway must know each deployment’s headroom.

Concurrency. A 250-token answer streams for about 8 seconds (a second to the first token, then about 35 tokens a second). By Little’s law, 3,000/s × 8 s = 24,000 open streams at peak, each holding a client connection and an upstream stream.

CPU and pods. Per chat request, budget about 1 ms of CPU for parsing, auth, policy and token estimation; about 20 µs for each of roughly 150 streamed events (3 ms); and 1 ms to settle — about 5 ms of CPU, and 1 ms per embedding. Peak: 3,000 × 5 ms + 2,000 × 1 ms = 17 cores, plus about 5 for TLS and scanning: 22 cores. At 50% utilisation that is 11 four-core pods; to survive losing one of three zones, 11 × 1.5 ≈ 17, so 18 pods, each holding about 1,350 open streams. Connections and draining long streams matter more than CPU.

Counter store. One admission and one settlement script per request plus a cache lookup for chat: 2 × 5,000 + 3,000 ≈ 13,000 Redis operations/s. One primary could do it; run three for headroom. The general rate limiter needed 30 shards for 1.2M checks a second — LLM traffic is expensive per request, not numerous.

Logs. A 1 KB usage event per attempt is 5 MB/s into Kafka at peak and 86 GB a day; at an assumed 10× columnar compression, 13 months is about 3.4 TB. Bodies are different: at about 4 characters per token a chat request is about 6 KB with JSON, so 51.8M chats a day is 311 GB a day and 30 days is 9.3 TB. Storage is cheap; holding 1.5 billion customer prompts is not. Hence opt-in only.

Spend. Assume one provider’s October 2026 list prices: a small model at $0.10 / $0.50 per million input / output tokens and a mid-tier model at $2 / $10, with embeddings on the self-hosted fleet. Average chat is 600/s × 2,592,000 s = 1.56B requests a month: 1.56T input and 389B output tokens. Send 80% to the small model, 10% to the mid-tier and 10% to the self-hosted fleet:

• Small: 1,244B × $0.10/M + 311B × $0.50/M = $124k + $156k ≈ $280k
• Mid-tier: 156B × $2/M + 39B × $10/M = $311k + $389k ≈ $700k
• Total ≈ $1M a month, about ₹8.6 crore at ₹88 per dollar

Ten percent of requests are 71% of the bill, and a 1% budget leak is $10k a month.

Latency. Twenty milliseconds holds only if the slow admission checks run in parallel.

High-level design

Build it one requirement at a time; by the end you have a gateway that works in one region at moderate traffic.

Apps should call any approved model through one API

Put stateless gateway pods behind an L7 load balancer. Each pod holds an alias table in memory and one adapter per provider, which turns the OpenAI-style request into the provider’s native one and the provider’s responses back into OpenAI-style chunks.

Adapters are where the bugs live, so give them contract tests. For an Anthropic-style API the adapter moves the system message to a top-level field, always sets an output limit, maps tool schemas and results, and maps stop reasons back (end_turn to stop, max_tokens to length, tool_use to tool_calls). A parameter a provider cannot honour is rejected with a 400 naming it, never silently dropped — a team that asked for JSON-schema output and quietly did not get it will ship a bug. Aliases point at pinned model versions, so an upgrade is a config change rolled out by canary, not a provider’s silent “latest”.

Admins should issue virtual keys with access policies

Keys, teams, aliases, prices and policies live in a config DB (PostgreSQL) behind an admin API, and the hot path never reads it. The control plane compiles everything into an immutable, numbered snapshot and pushes it to every pod, which swaps it in with one pointer change; if the config DB is down, pods keep serving the last snapshot. That is the control plane versus data plane split.

A virtual key is a random 256-bit string shown once; the snapshot holds only its SHA-256 hash, so authentication is a hash and a map lookup. (See authentication, sessions, tokens and JWT for why opaque, revocable keys suit this.) Then come cheap policy checks — alias allowed, key not expired, data class within the key’s ceiling. Revocation is a snapshot change, live within 30 seconds.

Teams should be held to token-aware limits and budgets

Add a Redis cluster. At admission one atomic script checks the key’s request and token buckets and the team’s budget and reserves what the request might use; when the response ends a second script settles with the real numbers. The bucket mechanics are those of the rate limiter; what is new is a cost you only know afterwards — the first deep dive.

Requests should be routed, retried and failed over

A router inside each pod turns an alias into candidate deployments. Per deployment it tracks latency and errors, a circuit breaker and a mirror of the provider’s limits; it drops deployments that cannot serve the request, picks one by weight and health, and on a retryable failure before the first token tries the next.

Every request should produce usage, cost and audit records

When a request settles, the pod writes one usage event to Kafka and moves on. Consumers keep budgets current, write per-minute rollups to a usage DB for showback, and archive audit records. Two optional modules sit on the admission path: an exact-match cache in Redis and a guardrail service for PII redaction and injection checks.

That works for one region at moderate traffic. Now make it accurate, fast and safe.

Deep dives

How do we rate-limit and budget by tokens when the output length is unknown?

Two facts make this harder than ordinary rate limiting. Tokens, and the money they cost, are known only when the response ends — seconds after you had to say yes or no. And the gateway is itself a customer of providers that limit it by requests and tokens per minute, so your limits must protect both your budget and that shared quota.

Bad: requests per minute only. One request can be 50 tokens or 100,000, so 600 requests a minute means anywhere from 30,000 to 60 million tokens. It stops neither a runaway bill nor one team draining everyone’s provider quota.

Good: count tokens after the response. Charge each response’s usage to the key’s token bucket and reject while the bucket is empty. Envoy AI Gateway documents exactly this, including the consequence: with a 1,000-token hourly limit, a stream admitted into an empty bucket can use 1,200 tokens, and later requests are rejected until the window recovers. Simple and exact — but overshoot grows with concurrency. Ten requests arriving in the same second all see a full bucket, because none has been charged yet.

Great: reserve, then reconcile. At admission, charge the most the request could cost; at the end, refund what it did not use.

1. Estimate input tokens with the model family’s tokenizer, or fall back to characters ÷ 4, the published rule of thumb for English. Code, JSON and Indian-language scripts need more tokens per character, so pad the fallback (say 1.3×) and calibrate it against real usage.
2. Cap the output at max_completion_tokens, or a per-alias default such as 1,024 — never the model’s maximum.
3. Reserve atomically in one Redis script: a request, the input estimate and the output cap from the key’s buckets, and their rupee cost from the team budget, priced at the most expensive deployment in the alias because none is chosen yet. If anything is short, return 429 with Retry-After from that bucket’s refill rate.
4. Settle at the end. The gateway always sets stream_options.include_usage on OpenAI-style upstreams (stripping the usage chunk if the caller did not ask); on Anthropic-style streams, message_start carries input tokens and message_delta cumulative output tokens. Charge actuals at the chosen deployment’s price; refund the rest.
5. Plan for broken endings. An interrupted stream may never send its usage chunk, so count the relayed text and mark the record estimated. Holds carry a 15-minute TTL; if a pod dies mid-stream the hold expires and is charged in full, the conservative choice.

# Admission: one atomic script per identity in Redis.
est_in  = tokenizer.count(msgs) if tokenizer else ceil(chars(msgs) / 4 * 1.3)
max_out = min(req.max_completion_tokens or alias.default_out, alias.max_out_cap)
hold = reserve(
    key=vk.id, team=vk.team, hold_id=req.id, ttl_s=900,
    rpm=1, in_tpm=est_in, out_tpm=max_out,
    inr=est_in * alias.max_price.input + max_out * alias.max_price.output)
if not hold.ok:
    return http_429(code=hold.reason, retry_after_s=ceil(hold.wait_s))

# ... route, stream, relay ...

# Settlement: refund the difference; buckets may go briefly negative.
used = stream.usage or estimate_from_relayed_text(stream)
settle(hold_id=req.id, in_tokens=used.input, out_tokens=used.output,
       inr=rate(used, deployment.price, price_version=snapshot.prices))

If the input estimate was low, actual usage can exceed the hold. Let the bucket go briefly negative rather than fail a request already served; the debt is bounded by the estimator’s error.

Reserving is pessimistic: in the figure seven requests (7 × 1,300 = 9,100 tokens) would have fit, but five were admitted. Teach callers to send realistic max_completion_tokens, and for soft limits reserve the key’s observed p99 output length instead of the cap.

Mirror the providers’ limits. Keep a shared bucket per deployment at about 90% of the provider’s limits, reserve against it too, and model each provider’s rules — one documents that it counts output as it is produced (so max_tokens does not count) and, for most models, ignores cache reads in its input limit. Correct drift from headers such as x-ratelimit-remaining-tokens or anthropic-ratelimit-input-tokens-remaining. When a mirror runs low, the router shifts weight away before the provider returns 429; if capacity is truly short, interactive traffic goes ahead of batch (see backpressure and load shedding and noisy neighbours and fairness).

Rupee budgets across 18 pods use the same reserve and settle, with three ways to keep the counter:

• Bad: sum the usage DB at admission. Minutes stale, so a runaway job overshoots by whatever it spends in the lag.
• Good: one atomic counter per team in Redis, reserved and settled with the token buckets. Exact, and well within 13,000 operations a second.
• Great: leases for hot teams. A 2,000-request-a-second batch job hammers one key on one shard. Instead each pod leases a ₹200 slice and spends it locally. The store counts leased money as spent, so the error either way is what sits in slices: unspent slices can stop a team up to pods × slice early, and admitting while a renewal is in flight can overshoot by at most the same. Near the limit slices shrink, and in the last 2% every request goes to the central counter.

Add a soft alert at 80%, the hard stop at 100%, and a per-request cost cap, say ₹20: a 200,000-token prompt to the mid-tier model reserves about ₹35 of input alone (200,000 × $2 per million × ₹88), the kind of “denial of wallet” OWASP warns about. Budgets nest (key, team, business unit) in the same script.

How do we route and fail over across providers?

Providers fail in more ways than “down”: a 429 because your organization hit its token limit, one model overloaded in one region, a deployment that takes 40 seconds to start answering. The router must fail over without making things worse.

Bad: a static priority list. Always call B; on any error retry twice, then try A. During an outage every request still pays for the doomed attempts on B. Retrying a 400 elsewhere just gets another 400. And retrying B’s 429 when your quota is spent is a retry storm aimed at your own limit.

Good: health-aware weights with circuit breakers. Spread traffic across healthy deployments by weight, track error rate and time to first token, and open a breaker on a failing deployment for a cooldown before probing it again; LiteLLM’s proxy, for instance, cools a deployment down after a configurable number of failures a minute. Failover becomes fast and automatic — but the fallback may not be able to do the job.

Great: filter on capability, then score, then retry by error type.

1. Filter on a capability matrix. Each deployment declares its context window, tool calling, JSON-schema output, vision, and the data classes and regions it may handle. A request needing tools and 12K of context never goes to a model without tools, however healthy, and confidential data never reaches a US deployment.
2. Score the survivors on weight, error rate, time to first token and headroom in the provider-limit mirror, so traffic moves before a provider says 429.
3. Retry by error type, within a fleet-wide retry budget of about 10% extra attempts, so retries never multiply load during an outage.

The last row matters most. Once a token has reached the client, the gateway must not switch models and carry on, or the caller gets half of one model’s answer glued to half of another’s; a major provider’s own guidance is not to replay a request after consuming output. So the gateway fails over freely until the first byte goes out, then sends an error event with partial usage and lets the app decide.

Hedging — sending a request to two deployments and keeping the faster — trims tail latency for short, latency-critical calls such as autocomplete, but doubles token spend for the hedged share. Hedge only after the first attempt passes its p95 without a token, and cancel the loser at once.

Two habits make failover real. Keep standing traffic on every fallback, even 10–20%: a deployment idle for a week has untested credentials and unknown latency. And respect ramp limits — one provider asks customers above 1M tokens a minute to grow by no more than 50% every 15 minutes — so shift weights in steps. More in timeouts, retries and backoff.

How do we stream without adding latency?

Streaming is the user experience: first words in a second, not a whole answer after ten. Providers stream with server-sent events, and so does the gateway.

Bad: buffer, then send. Output checks become trivial, but time to first token becomes time to last token. The accidental version is common: many proxies and compression filters buffer by default (NGINX’s proxy_buffering is on unless you turn it off), so one ingress hop undoes your work.

Good: relay chunk by chunk. Parse each upstream event as it arrives, write it and flush; turn off buffering and compression for text/event-stream on every hop. Added latency per chunk drops well under a millisecond.

Great: relay, translate, time out and cancel.

• Translate event by event. An Anthropic-style stream is message_start (with input tokens), then content_block_start, many content_block_deltas and content_block_stop, then message_delta (stop reason, cumulative output tokens) and message_stop, with pings between. The adapter is a small state machine that turns each text delta into a chat.completion.chunk at once, turns partial tool-argument JSON into tool-call deltas, emits the usage chunk and [DONE] at the end, and skips unknown event types.
• Three timeouts, not one: connect (about 2 s); first token, per alias (30 s for a large reasoning model, 5 s for autocomplete), after which the request fails over since the client has seen nothing; and idle between chunks (about 20 s), which catches a stream stalled mid-answer.
• Heartbeats. Load balancers drop connections silent for longer than their idle timeout, often 60 seconds, and reasoning models can pause longer than that. Send SSE comment lines during long pauses, as providers send ping events.
• Backpressure, not buffering. If a client reads slowly, let a small per-stream buffer fill (say 64 KB), then stop reading upstream so TCP pushes back.
• Cancel upstream when the client leaves. Close the upstream stream at once so the provider can stop generating; settle from the text already relayed, since tokens produced may still be billed.
• Errors inside the stream become an OpenAI-shaped error event, with usage settled for what was delivered.

Output guardrails fight streaming, because a sent token cannot be recalled. Scan a sliding window for local patterns — card numbers, API keys, phone numbers — holding back only the last few dozen characters, and keep full buffering for the few aliases whose answers justify the wait.

How do we cache safely?

A wrong cache hit here can be another customer’s data or a confident answer to a different question. Of three caches, the one you control least usually pays most.

Bad: cache by prompt text for everyone. Two teams’ identical questions with different system prompts collide, personalised answers leak between users, and a temperature-0.9 creative answer is frozen forever.

Good: an exact-match cache, scoped and opt-in. Key it on everything that shapes the answer: the team (never shared), the alias and the pinned model version, normalised parameters (temperature, output cap, tools, response format) and the canonical JSON of all messages. Cache only on opt-in (x-gw-cache: on), only at low temperature, never for responses containing tool calls — a cached “call refund_payment” would repeat an action — and for hours, not weeks. Embeddings are the clear winner: re-indexing a document store re-embeds mostly unchanged chunks. For chat, exact hits are rare outside repeated batch jobs.

Risky: semantic caching reuses answers to similar prompts: embed the prompt, search a vector index in the team’s namespace, and reuse the nearest answer above a similarity threshold. It raises hit rates and adds a failure exact caching never had: a false hit, where a close prompt has a different correct answer.

If you build one, keep it on a short leash: per-team namespaces, opt-in per alias, FAQ-like questions only, a threshold tuned on a few hundred labelled prompt pairs, a one-day TTL, and hits in the audit log. Every request also pays for an embedding and a vector search (see embeddings and semantic search).

Great: protect the provider’s prefix cache. Providers cache their work on a prompt prefix and charge much less to reuse it. One provider charges 1.25× the input price to write a prefix with a 5-minute lifetime and 0.1× to read it, refreshing the lifetime on each use. Support bots, KYC summarisers and agents resend the same long system prompt and tools on every call, so hit rates are high — and nothing is ever wrong, because the model still answers the full prompt. If 60% of the mid-tier model’s input tokens hit, the saving is 0.6 × $311k × 0.9 ≈ $168k a month, before the write premium. The gateway must not break it:

• Never touch the start of the prompt. Put injected metadata at the end, keep tool order stable, and never add timestamps or request IDs to a system prompt.
• Route for locality. Caches are isolated per provider organization or workspace, so send requests sharing a prefix to the same deployment, for example by hashing the first 1,024 tokens into the weighted pick.
• Count cached tokens separately in usage, budgets and limit mirrors; they cost and count differently.

Do not cache at all for personalised answers, anything about money, orders or accounts, tool calls, restricted data, or questions whose answer depends on today’s date.

How do we account for usage and cost accurately?

Usage data has three clocks: budgets need it in seconds, team showback in minutes, and finance a monthly chargeback that matches provider invoices.

Bad: a synchronous database write per request, or provider invoices alone. The first makes a slow database minute a slow gateway minute; the second arrives monthly, per provider account, with no idea which team spent what.

Good: a usage event per request into Kafka, aggregated downstream. The request path only appends to a producer buffer. Fast and isolated — but not yet right.

Great: idempotent, attempt-level events, priced with versioned tables and reconciled against the providers.

• One event per attempt, with a stable ID. A request that failed over produces two events, and if the first deployment billed tokens before it broke, the team pays for those too. The ID request_id + attempt makes at-least-once delivery safe: pods retry publishing, spool to local disk if Kafka is briefly away, and consumers drop duplicates (see idempotency).
• Key by team so each team’s events are ordered in one partition; give a very large team a few sub-keys so its partition does not run hot. See Kafka.
• Count token types separately — uncached input, cache writes, cache reads, output, reasoning — because each is priced and limited differently. A single “total tokens” field can never be priced correctly later.
• Version the prices. Each price table has a valid_from; events carry the version in force, old versions are never edited, and a wrong price is fixed in a new version followed by re-rating the affected days from archived events. Providers bill in dollars and budgets are in rupees, so each month’s table fixes one exchange rate and finance trues up centrally.
• Price the self-hosted fleet at internal cost per million tokens, so showback compares it honestly.
• Reconcile daily. With one provider account per business unit, the provider’s usage report is an independent check. Drift beyond 1% against the usage DB pages someone; it catches estimated streams, missing token types, price typos and lost events.

These are the same patterns as in metering and usage-based billing and Design usage metering and billing: the gateway is a metering system whose meter happens to be a proxy.

How do we enforce data and safety policies?

The gateway is the last place in your network to see a prompt before it leaves for another company: the natural enforcement point for data rules, though a coarse filter, not a replacement for each app’s defences.

Classify, then route. A request’s data class is the highest of the key’s default, the caller’s x-gw-data-class tag and what the PII scanner finds; above the key’s ceiling, it is rejected. The class then narrows the deployment list.

Redact PII before external calls. Most Indian identifiers have checkable shapes: a 12-digit Aadhaar number with a Verhoeff check digit, a PAN of five letters, four digits and a letter, card numbers passing the Luhn check, IFSC codes, UPI IDs like name@bank, phone numbers and emails. Regexes plus checksums are cheap; names and addresses need an entity model, run on a separate pool in parallel with the limit and cache checks. Redact reversibly when the answer needs the value: replace ABCPK1234F with a placeholder like PAN_1, keep the mapping only in the request’s memory, and restore it in the reply, holding back a few characters so a placeholder split across chunks is still caught.

Check for prompt injection. OWASP’s 2025 list for LLM applications puts prompt injection first and sensitive information disclosure second, and also lists improper output handling and unbounded consumption. A gateway classifier catches crude attacks in user input; it cannot see an injection hidden in a retrieved document or tool result, which is why apps that act on model output need their own controls (see the AI agent platform).

Fail closed or open per class. If the scanner is down, confidential and restricted traffic stops; public traffic continues with an audit flag.

Audit everything, store little. Keep metadata for every request — key, team, alias, deployment, data class, policies fired, tokens, cost, latency, outcome — plus a prompt hash for investigations. Store bodies only on opt-in, redacted, encrypted with a per-team key (see encryption and key management), deleted after 30 days, with every read logged. That fits the purpose-limitation and deletion duties of India’s Digital Personal Data Protection Act, 2023; have counsel confirm the details.

How do we keep provider keys safe and the gateway available?

The gateway holds keys that can spend $1M a month, so the boring disciplines matter.

Provider keys live in a vault or KMS, are fetched at startup with the gateway’s own workload identity, and stay in memory — never in snapshots, logs or error messages. One provider account per business unit limits what a leaked key can spend and separates limits, caches and usage reports. Rotate with two keys active during the overlap, and restrict keys to your egress IPs where providers allow it.

Availability. Pods are stateless across three zones, sized so two zones carry the peak, with the last good snapshot on local disk. Redis calls time out in about 5 ms: rate limits then fall back to local per-pod buckets, and budgets to the leases already held plus a small emergency allowance. A draining pod stops taking new requests and lets streams finish (up to, say, 10 minutes) before exiting, and every change ships by canary.

The final design

The hot path is the gateway pods, Redis, the guardrail service and the chosen deployment; everything else is asynchronous or pushed ahead of time.

Follow one KYC-summary request:

1. Arrive. The app calls POST /v1/chat/completions with model: chat-default and its virtual key; the load balancer picks any pod in any zone.
2. Authenticate and classify. The pod finds the key’s hash in its snapshot, checks the alias is allowed, and classes the request confidential because the scanner finds a PAN.
3. Admit in parallel. One Redis script reserves tokens and rupees against the key, team and provider-mirror buckets; meanwhile the guardrails replace the PAN with a placeholder and score the prompt for injection, and the exact cache misses.
4. Route and stream. The router drops deployments without tools or outside India, picks provider B by weight and headroom, and sends the redacted prompt with B’s key from memory. Chunks are translated, the placeholder restored, and each flushed as it arrives. A 429 or a first-token timeout would have sent it to A.
5. Settle. At message_stop the pod settles the hold with real token counts and publishes a usage event keyed by team.
6. Account. Consumers update the budget within seconds, write priced rollups for showback, archive the audit record, and the next day the reconciler checks totals against B’s usage report.
7. Change. Admins change keys, budgets, deployments or prices through the control plane, which pushes a new snapshot to every pod within 30 seconds; provider keys come from the vault, never the snapshot.

Check the requirements: admission about 12 ms at p99, inside 20 ms, with chunks never buffered; failover before the first token, breakers and warm fallbacks make an alias more available than any one provider, while 18 stateless pods over three zones carry 24,000 open streams; token limits hold within the estimator’s error and budgets within 1%; every request leaves a usage record and audit trail, bodies are opt-in, and restricted data never leaves the self-hosted fleet.

What interviewers look for by level

Follow-up questions

What if a team needs guaranteed capacity?

Pay-as-you-go limits are best effort. For a must-not-fail workload, such as the support bot during a sale, buy provisioned throughput from a provider or dedicated self-hosted replicas. Model it as a dedicated deployment first in that team’s alias, spilling over to shared ones, and charge the reservation to the team whether or not they use it. When shared capacity is scarce, the reserved team keeps its slice and batch traffic from others is shed first.

What if one provider has a multi-hour outage?

Breakers open in seconds and traffic shifts; the question is whether the others can absorb it. Know in advance their token headroom (from the mirrors), their ramp-rate rules (shift in steps), and whether each alias’s fallback gives acceptable answers — run each alias’s evaluation set against its fallbacks weekly, not mid-outage. If capacity still falls short, degrade in a planned order: pause batch aliases, cap output for low-priority keys, move some traffic to the self-hosted fleet, and say on a status page what “degraded” means.

What if teams want to A/B test two models?

Add a split to the alias — 90% current, 10% candidate — chosen by a stable hash of the user or session ID so each user sees one model. Usage events carry the variant, so cost, latency and the app’s quality metrics compare cleanly. For riskier changes, shadow 1% of requests to the candidate and discard its answers, remembering that every shadowed token is paid for.

What if an agent makes 50 model calls per task?

Then the budget that matters is per run. The agent platform sends a run ID header, and the same Redis script keeps a per-run budget of rupees and calls, so a looping agent stops at, say, ₹20 or 50 calls instead of burning the team’s month. Usage events carry the run ID, making cost per task measurable. Loop limits, tools and approvals belong to the AI agent platform; the gateway is the backstop.

What if we must run on-premises only?

Keep the gateway and drop the commercial deployments. Aliases point at self-hosted pools, prices become internal cost per million tokens, and budgets become chargeback for GPU time. Keys, token limits, routing, streaming, usage and audit all survive; failover now means another pool or a smaller model, and capacity planning moves to the inference service’s GPUs.

How to say it in the interview

“I’ll design an internal LLM gateway for about 300 apps calling two commercial APIs, a cloud model service and our self-hosted fleet: 5,000 requests a second at peak, about a million dollars a month of spend, and under 20 milliseconds of added latency.

It’s a stateless, OpenAI-compatible proxy. Apps send an alias like chat-default and a virtual key; each pod holds a config snapshot pushed by a control plane, so auth and policy are in memory, and provider keys never leave the gateway.

The core problem is that cost is in tokens and the output length is unknown at admission. So I reserve estimated input plus max_tokens from per-key token buckets and a team rupee budget in Redis, then settle with real usage when the stream ends and refund the rest. I mirror providers’ limits to shift traffic before they return 429, and small per-pod leases keep budget overshoot under 1%.

Routing filters deployments by capability and data class, then picks by weight and health with circuit breakers. I retry 429s, 5xx and timeouts on another deployment, but only before the first token reaches the client — never mid-stream. Streams are relayed chunk by chunk, translated to one format, with first-token and idle timeouts, and cancelled upstream when the client disconnects.

Every attempt emits an idempotent usage event to Kafka for budgets, showback, audit and daily reconciliation with provider invoices. PII is redacted before external calls and restricted data stays on the self-hosted fleet. For caching I’d lean on providers’ prefix caching and an exact cache for embeddings, and treat semantic caching with care.”

Concepts used in this design

• API gateway — the general pattern, specialised here for tokens, streams and third-party upstreams.
• Rate limiting and Design a rate limiter — token buckets in Redis, reused with tokens as the cost.
• Redis — buckets, budget holds, leases and the exact cache.
• Kafka — the usage-event log, keyed by team.
• Circuit breaker and timeouts, retries and backoff with jitter — failing over without a retry storm.
• Real-time connections: long polling, SSE and WebSockets — how token streams reach clients.
• Noisy neighbours and fairness — sharing scarce provider quota between 300 teams.
• Control plane vs data plane — config snapshots that keep the database off the hot path.
• Idempotency — usage events delivered twice, counted once.
• Metering and usage-based billing and Design usage metering and billing — rating, versioned prices and showback.
• Embeddings and semantic search — what semantic-cache similarity measures.
• Encryption and key management — provider-key custody and encrypted bodies.
• Authentication: sessions, tokens and JWT — opaque, hashed, revocable virtual keys.
• Design an LLM inference service — the self-hosted fleet behind one of the deployments.

References and further reading

1. OpenAI API: Rate limits guidedevelopers.openai.comRPM and TPM limits per organization and project, the x-ratelimit-* response headers, 429 slow_down and the ramp-rate rule of thumb, and the warning not to replay a stream after consuming output.
2. Claude API: Rate limitsplatform.claude.comSeparate RPM, ITPM and OTPM limits enforced with a token bucket, input estimated at the start and adjusted, cache reads excluded from ITPM, anthropic-ratelimit-* headers, tier limits, and the spend-cap 429 without retry-after.
3. OpenAI API reference: Create chat completiondevelopers.openai.comstream_options.include_usage, the final usage chunk with empty choices, the warning that an interrupted stream may never send it, and max_completion_tokens replacing max_tokens.
4. Claude API: Streaming messagesplatform.claude.comThe SSE event sequence (message_start, content_block_delta, message_delta, message_stop), cumulative usage in message_delta, ping events and mid-stream error events.
5. OpenAI API: Key concepts (tokens)developers.openai.comThe rule of thumb that one token is about 4 characters or 0.75 words of English text.
6. Claude API: Prompt cachingplatform.claude.comPrefix caching with a 5-minute default lifetime, cache writes at 1.25× and reads at 0.1× of the input price, workspace-level cache isolation, and the per-model price table used for the cost arithmetic.
7. Envoy AI Gateway (now Agent Router): Usage-based rate limitingtheagentrouter.aiToken-based limits charged after the response completes, and the documented overshoot when an admitted stream exceeds the remaining budget.
8. LiteLLM proxy: Virtual keysdocs.litellm.aiVirtual keys with allowed models, per-key and per-team spend and limits, model aliases, and cost computed from a model price table.
9. LiteLLM proxy: Fallbacks (provider failover)docs.litellm.aiOrdered fallbacks after retries, separate context-window and content-policy fallbacks, and cooling down a deployment after repeated failures.
10. OWASP Top 10 for LLM Applications 2025genai.owasp.orgThe 2025 risk list, including prompt injection, sensitive information disclosure, improper output handling and unbounded consumption (denial of wallet).