1,000 units, four tenants want 1,550. Who gets what?
A hard split caps each at 250 and leaves 250 idle. Weighted max-min fills small demands first, splits the rest by plan, and idles nothing.
2.4 s
p99 for 12 small tenants, from 120 ms
They share W’s database shard and send 0.5% of requests, so the fleet p99 creeps from 250 to 270 ms. The dashboard stays green.
Bad: Rate limit of 100/s
If reports slow from 50 ms to 5 s: 100 × 5 = 500 in flight, unnoticed.
Good: Cap 10 in flight
Counts work in progress, not arrivals, so it holds however slow things get.
W’s export passed every gateway limit. How did it flood the queue?
Fan-out
It was one allowed call that fanned out into 2 million jobs. Enforce fairness again where jobs are consumed.
A queue per tenant, served in turns
A weighted fair scheduler runs small sellers’ jobs next round; W gets the other 2,500 a second. W pays 5 extra minutes; nobody else pays 8.
Five layers, each guarding a different resource
Only the first two reject a tenant outright. The middle layers make heavy work wait in its own lane. Under overload, the shedder drops the lowest priority.
Signs it’s time for a silo
- Limits fire daily, though it pays well
- A fifth or more of its shard’s load
- Keeps appearing in incident reviews
- Needs guarantees the pool can’t make
Remember
One tenant’s burst becomes everyone’s latency. See it per tenant; layer limits, fair queues and bulkheads; give whales their own silo.