8,950
req/s offered to an 8,000 req/s pool
4,000 − 50 + 5,000. Queue the excess and every request waits 1.2 s more after 10 s. Shed it evenly and 10.6% of everyone’s requests fail.
Isn’t a global rate limit enough?
Bad: One bucket for everyone
It protects servers, not tenants; whoever sends most gets most.
Good: A bucket per tenant
Checked first, so one tenant can’t drain the shared tokens.
A 500-a-second quota turns a fleet-wide incident into one tenant’s 429s
Merchant 417 gets 500 requests through and 4,500 back as 429s. The pool runs at 56%, and the other 999 tenants never notice.
One FIFO makes a one-job tenant wait 300 seconds
Tenant A drops 60,000 jobs ahead of tenant B’s one, on 200 workers. A queue per tenant, served one turn each, cuts B’s wait to milliseconds.
Nothing on the dashboard says who. Tag every metric with the tenant ID?
Careful
That’s 300,000 series for one histogram. Label the top N, fold the rest into “other”, keep IDs in logs.
Limit, schedule fairly, then isolate
- A token bucket per tenant at the gateway
- Fair queues: one turn per active tenant
- Requests and limits; batch at low priority
- Cells, shuffle shards, and silos for whales
3.6%
chance another tenant shares both workers
Give each tenant a random pair of 8 workers, 8 × 7 ÷ 2 = 28 pairs. Four fixed shards of two would put 1 in 4 tenants beside the noisy one.
Say it like this
One tenant mustn’t become everyone’s incident. Limit each tenant before the global limit, queue work per tenant, set requests and limits, and give the whales their own capacity.