429 or 503?
Bad: 503 for an over-budget caller
503 means we’re over capacity, whoever is asking.
Good: 429: you’re over your budget
Per-caller limits answer 429, even at 20% CPU.
Same traffic, different memory, different bursts
Each limiter allows 5 a second. A fixed window lets 10 through in half a second at its edge; a token bucket allows only the burst you chose.
500
requests in flight at 500 ms, not 50
Little’s law: 1,000 a second × 0.5 s. Arrivals never changed, so a 1,200-a-second rate limit never fires. Only a cap on work in flight sees it.
Count the right key
500 phones behind one carrier NAT address share a counter, while one IPv6 host can rotate through 2⁶⁴ addresses. Split by account or device; merge per /64.
Your per-IP limit reads the leftmost X-Forwarded-For entry. Protected?
The trap
No. The client writes that header and can fake a new address every request. Key on the address your own edge appended.
Enforce in layers, cheapest first
The edge sees only an IP, the gateway knows the key and plan, and only the service knows a request’s real cost. Outbound, smooth to your partners’ limits.
Ship limits safely
- 429 with Retry-After, rounded up
- Never cache a 429
- Shadow-mode every new limit first
- Fail open, except for OTP and login
Say it like this
A token bucket per API key, 50 a second with bursts to 100, enforced in layers from edge to service. Over budget gets a 429 with Retry-After; clients back off with jitter.