$1M
a month of model spend
Assume October 2026 list prices. The mid-tier model takes 10% of requests but 71% of the bill. A 1% budget leak is $10k a month.
What should a limit count?
Bad: Requests per minute
600 a minute can be 30,000 tokens or 60 million.
Good: Tokens, reserved then settled
Reserve input + max_tokens up front; refund the rest when the stream ends.
Ten requests hit a key’s 10,000-token limit in one second
Charge afterwards and all ten get in, 13,000 tokens, 30% over. Reserve 2,000 each and five get in; the unused 3,500 is refunded.
18×
one provider’s top tier, needed at peak
Assume 1,000 input tokens a chat: peak needs 180M a minute. One provider’s top self-serve tier allows 10M, so spread load and mirror every provider’s limits.
Provider B dies mid-answer. Why not finish it on provider A?
The rule
Fail over freely until the first token goes out. After that, send an error event with partial usage; never glue two models’ answers.
Two refund questions, 0.98 similar, about different orders
A semantic cache hands one customer another’s answer. The provider’s prefix cache can’t; the model still answers the full prompt.
Before a prompt leaves your network
- Class = highest of key, tag and PII scanner
- Confidential, like a PAN: redact before commercial APIs
- Restricted, like Aadhaar: self-hosted fleet only
- Scanner down? Confidential and restricted stop; public continues
Remember
Reserve tokens and rupees up front; settle when the stream ends. Fail over only before the first token, and log every attempt.