1M/s
rate-limit checks at peak
Each one adds latency to a real request. Budget ≈ 0.6 ms typical with a hard 3 ms cap, and the limiter must never take the API down.
Where do the counters live?
Bad: In each pod
With 100 pods, round-robin turns 100 a minute into 10,000.
Good: Shared, in Redis
A rate-limit service runs one atomic script per check; every pod sees one count.
A limit of 100 a minute let 200 through in 20 s
Fixed windows reset on the minute. Send 100 in the last 10 seconds and 100 in the next 10, and neither window looks wrong.
Token bucket: one knob for bursts, one for the average
Capacity 40 absorbs a screen firing a dozen calls; refilling at 20 a second holds the long-run rate. Store tokens and a timestamp; refill lazily.
One partner sends 400k requests a second on one key. Its shard handles 40k.
The trick
Pods lease 100 tokens per call and spend them locally. 4k Redis calls a second instead of 400k.
Redis stalls mid-rush. Let everyone in, or no one?
Give up after 3 ms and never retry. Then each rule decides, in advance. Most fail open; OTP sends and logins fail closed.
A bad rule is a self-inflicted outage. Ship it safely:
- Validate and replay the last hour
- Publish immutable snapshot v43
- Shadow a day: log “would deny”, allow
- Canary 1 pod → 10% → all; v42 one click away
Remember
A token bucket per caller, checked by one atomic script in sharded Redis. Bounded timeouts and per-rule fallbacks mean the limiter never takes the API down.