Noisy neighbours and fairness

A noisy neighbour is a tenant whose load degrades everyone who shares its infrastructure: one customer’s bulk export, flash sale or runaway script eats the CPU, connections, cache or worker time that a thousand other tenants were counting on. Fairness is the set of mechanisms that stop this, so each tenant gets a predictable share of the pool while spare capacity still goes to whoever can use it. In a pooled SaaS product you either design it in, or your biggest customer designs your outage.

Picture it first: the booking counter and the travel agent

A railway booking counter opens at 10 am. First in line is a travel agent holding 400 forms. At 45 seconds a form, he needs 400 × 45 s = 18,000 s, 5 hours, and everyone behind him waits. Nobody broke a rule; the line just doesn’t know whose turn it is.

A sensible station manager has five fixes:

• Four forms per visit, then back of the line. Everyone gets a turn.
• A separate counter for agents, so bulk work never blocks the general line.
• A daily cap per agent licence.
• A window for senior citizens that is served first when the hall is busy.
• Closing the gate when the hall is full.

Map it back. The counter is your workers, connections and CPU, the line is your job queue, and the agent is a tenant running a bulk export. The fixes are round-robin scheduling, a bulkhead, a quota, priority and load shedding.

What one tenant can use up

Everything shared is something a tenant can hog. CPU and memory are the obvious ones; outages usually come from further down this list.

Make it concrete. A B2B order-management SaaS runs the back office for 2,000 online sellers, at the peak of the Diwali sale. One large seller, W, clicks “Export all orders” for 2 million orders, and the app enqueues one job per order on the shared queue.

• Queue. 200 workers each finish 20 jobs a second: 200 × 20 = 4,000 a second, comfortably above the other sellers’ 1,500. But 2 million of W’s jobs now sit ahead of everybody, so a job enqueued just after them waits 2,000,000 ÷ 4,000 = 500 seconds, over 8 minutes.
• Cache. The export reads every order through a shared 32 GB LRU cache: 2,000,000 × 10 KB = 20 GB of cold data that evicts everyone else’s hot keys. If the hit rate falls from 95% to 60%, misses rise from 5% to 40% of reads: 8× the database read load.
• Downstream quota. Another seller sends a 500,000-message festive SMS blast through the platform’s shared provider account, limited to 100 messages a second: 500,000 ÷ 100 = 5,000 seconds, about 83 minutes, with every other seller’s delivery OTPs queued behind it.

Azure’s antipattern guide notes that most noisy tenants don’t know they are hurting anyone, and that many modest tenants peaking together is the other shape of the problem. That one is about capacity: capacity decides whether anyone suffers; fairness decides who.

What “fair” should mean

AWS’s Builders’ Library states the goal: every client of a multi-tenant system should get a single-tenant experience. When demand exceeds capacity, though, someone gets less than they asked for; the definitions differ in who.

Equal share with hard partitions. Split capacity N ways and cap each tenant at its slice. Predictable, but a quiet tenant’s slice sits idle while a busy one is throttled.

Max-min fairness. Satisfy the smallest demands in full, split what remains equally among the rest, and repeat. No tenant can gain without taking from one that has less, and it is work-conserving: idle capacity goes to whoever wants it.

Weighted max-min. The same, with shares proportional to a weight, usually the plan: free 1, standard 2, enterprise 4.

Work one example: 1,000 units of capacity, and four tenants wanting 1,550. A (free) wants 50, B (free) 500, C (standard) 200, D (enterprise) 800.

• Hard equal split: each is capped at 1,000 ÷ 4 = 250, so A gets 50, B 250, C 200, D 250. That uses 750; 250 sit idle while B and D are throttled.
• Max-min: A and C are satisfied (250 in total), and the other 750 is split 375 each between B and D.
• Weighted max-min: weights 1 + 1 + 2 + 4 = 8, so one weight is worth 1,000 ÷ 8 = 125. A needs 50 of its 125 and C 200 of its 250, so both are satisfied. The remaining 750 goes to weights 1 + 4 = 5, 150 per weight: B gets 150, D gets 600.

For a SaaS product, a practical definition of fair has three parts:

1. A guaranteed floor per tier, delivered even when the system is saturated.
2. A burst allowance above the floor, so a sale or a month-end close succeeds.
3. Borrowing when idle, returned at once when the owner needs it. AWS calls this stacking capacity rather than hard-allocating it: a client may burst into unused capacity, but when the others return, their traffic takes priority over the borrower’s burst.

DynamoDB does this at the storage layer. In its documented example, 400 write units are spread over four partitions, 100 each; when one partition receives 150 a second while the others take 50, adaptive capacity lets it sustain 150, within the table’s total and the per-partition ceiling of 3,000 reads and 1,000 writes a second. Burst capacity also banks up to 300 seconds of unused throughput.

Finally, measure shares in cost, not requests: a query returning 1 MB is not one returning 1 KB. Some AWS services charge each request as the cheapest kind, then debit its true cost afterwards; Kubernetes charges big list requests several “seats”.

Seeing it: per-tenant telemetry

You can’t be fair to tenants you can’t tell apart. Three habits make noisy neighbours visible:

• Tag everything with the tenant. The tenant ID resolved at the edge (see routing requests to tenants) goes on every log line, trace span, job and message.
• Watch the worst tenants, not the average. Show p99 per tenant, the top-N tenants by requests, CPU-seconds, database time and queue depth, and how many tenants are breaching their SLO right now.
• Attribute cost per tenant (CPU, database time, bytes, jobs). It feeds detection and metering.

Why not averages? Suppose W’s export also hammers its database shard, which 12 small tenants share. Together they send 0.5% of all requests. During the export their p99 climbs from 120 ms to 2.4 s. The fleet p99 is the latency 99% of all requests beat, and 0.5% of requests can barely move it: it creeps from 250 ms to about 270 ms, the dashboard stays green, and 12 customers open tickets.

One catch: with 20,000 tenants, a tenant label on every metric multiplies your time series 20,000 times, so keep per-tenant detail in logs and traces and emit metrics only for the heaviest tenants. Amazon SQS fair queues show the right shape of metric: beside the queue-wide oldest-message age, ApproximateAgeOfOldestMessageInQuietGroups says how long the tenants who are not noisy are waiting.

Limits at the front door

The cheapest layer rejects excess early. Algorithms are covered in rate limiting and design a rate limiter; here is what changes when the caller is a tenant.

Rate limits per tenant, sized by tier. Key the limit on the tenant, the billing entity, not only the API key, or a tenant with 50 keys gets 50 shares. A typical shape: free 10 requests a second with bursts to 20, standard 100 with bursts to 300, enterprise 1,000 with negotiated overrides, all kept in the control plane and pushed to the gateways. Excess gets 429 with Retry-After. Stripe calls its request rate limiter by far the most important; it rejects millions of requests a month.

Concurrency limits, often the better tool. A rate limit counts arrivals; a concurrency limit counts work in progress. By Little’s law, requests in flight = arrival rate × time in the system. A tenant allowed 100 report requests a second at 50 ms each has 100 × 0.05 = 5 in flight. If the database slows and reports take 5 s, the same rate means 100 × 5 = 500 in flight: 100 times the threads, connections and memory, and the rate limiter hasn’t noticed. A cap of 10 in flight holds however slow things get. Stripe’s concurrent requests limiter fired only about 12,000 times in a month, yet it ended the resource contention on its most expensive endpoints.

Quotas. Daily or monthly caps on exports, storage or seats protect cost and planning, not milliseconds. Show tenants their usage.

Fair queues for background work

Front-door limits can’t help once work is accepted: W’s export was one permitted API call that fanned out into 2 million jobs. AWS’s article on queue backlogs warns that once work sits in one shared queue, it is hard to isolate one workload from another. Backlogs outlast the spike, too: if an unthrottled tenant queues 10 times what consumers can handle during the 30 minutes it takes an operator to respond, draining takes 300 minutes.

So stop serving in arrival order. Give each tenant its own queue and let a scheduler decide whose job runs next:

• Round robin. One job from each non-empty queue in turn. Fair in jobs, not in work: a tenant whose jobs take 2 s gets 100 times the worker time of one whose jobs take 20 ms.
• Deficit round robin (DRR). Each queue earns a quantum of credit per round, say 100 ms of worker time × its tier weight, runs jobs while its credit covers them, and carries the rest forward. Fair in work, constant time per job (Shreedhar and Varghese, 1995).
• Weighted fair queuing (WFQ). Stamp each job with a virtual finish time from its cost and its tenant’s weight; run the smallest. More precise; needs a priority queue.

All three are work-conserving: if only W has jobs, W gets every worker. Rerun the Diwali numbers under DRR. The other sellers’ 1,500 jobs a second are each well under their fair share, so they run in the next round. W gets the rest, 4,000 − 1,500 = 2,500 jobs a second, so its export takes 2,000,000 ÷ 2,500 = 800 seconds, about 13 minutes instead of 8. W pays 5 extra minutes; nobody else pays 8.

A queue per tenant gets awkward at 50,000 tenants, because consumers must poll them all. Three ways around it:

• Hash each tenant to a few queues and pick the shortest. AWS Lambda keeps a fixed set of queues, maps each customer to a few of them, and enqueues into whichever has the fewest messages. A flooding customer backs up its own queues; others route around them.
• Sideline the excess into a spillover queue that is worked when capacity is free.
• Let the queue do it. Amazon SQS added fair queues to standard queues in July 2025. Producers set MessageGroupId to the tenant ID; when one group holds a disproportionate share of in-flight messages, SQS returns other groups’ messages first. It never caps a tenant’s rate, and consumers need no changes.

Priorities and load shedding

Per-tenant limits assume the system has the capacity it was sized for. When it doesn’t (a slow dependency, a lost zone, a sale bigger than forecast), the question becomes “what do we drop first?”. That is load shedding, decided by system state and request priority rather than by who sent the request; see backpressure and load shedding.

Stripe runs two shedders behind its two limiters:

• A fleet usage load shedder reserves capacity for critical methods such as creating charges. With a 20% reservation, non-critical traffic beyond its 80% gets a 503.
• A worker utilization load shedder ranks traffic in four classes (critical methods, POSTs, GETs, test mode) and, as workers back up, sheds from the bottom, starting with test-mode traffic. It sheds and restores slowly to avoid flapping.

In SaaS, priority combines what a request does with who sent it: checkout before exports, live before sandbox, paid before free. Kubernetes’ API Priority and Fairness shows both layers in one design. FlowSchemas put each request in a priority level with its own concurrency limit, in “seats”, so a flood from pods can’t block leader election; idle levels lend seats to busy ones. Within a level, flows (by user or namespace) are fair-queued, so one buggy client can’t starve its peers.

Since tenants’ load is usually uncorrelated, AWS notes, a sudden surge is most likely one tenant: shed its unplanned excess before failing everybody’s. Keep status codes honest: 429 means “you are over your share”, 503 means “we are overloaded”.

Bulkheads and database guards

Bulkheads give heavy work its own resources: interactive calls on one worker pool, exports and imports on another, analytics on a read replica. A runaway export fills only its own pool, so interactive latency is protected by construction. You pay in utilisation, since reserved pools sit partly idle; letting the heavy pool borrow idle interactive capacity claws some back. Azure adds two cheap moves: run non-urgent heavy work off-peak, and cap rows or query time on expensive operations. See bulkheads.

Most noisy neighbours end up in the shared database, so guard it directly:

• Timeouts per role or transaction. PostgreSQL’s statement_timeout is off by default, and the docs advise against setting it in postgresql.conf because it would hit every session. lock_timeout stops a statement queueing forever behind a bulk update; idle_in_transaction_session_timeout ends sessions that hold locks while doing nothing.
• Per-tenant connection caps in the app’s pool or the pooler: no tenant holds more than, say, 10 of 100 connections.
• Memory per query. work_mem applies to each sort or hash step, in each session. Three sorts in each of 50 report sessions at 64 MB can use 3 × 50 × 64 MB = 9,600 MB. Give heavy roles a bounded budget on their own replica.

-- Interactive request path
BEGIN;
SET LOCAL statement_timeout = '3s';
SET LOCAL lock_timeout = '500ms';
-- tenant-scoped queries ...
COMMIT;

-- Reports run as their own role, on a replica
ALTER ROLE reports SET statement_timeout = '120s';
ALTER ROLE reports SET work_mem = '64MB';

On DynamoDB the per-partition ceiling acts as a bulkhead: a whale’s hot key throttles at its own partition. Splitting that key is covered in multi-tenant data.

Isolation: shuffle sharding, cells and silos

Everything so far divides one pool fairly; isolation shrinks how many tenants share it.

Shuffle sharding

Take 8 workers. Plain sharding splits them into 4 fixed pairs. A tenant that floods its pair, or sends a poison request that crashes workers, takes down everyone on it: 1 tenant in 4.

Shuffle sharding gives each tenant its own pair from all 8. There are C(8,2) = 8 × 7 ÷ 2 = 28 possible pairs, so only about 1 tenant in 28 shares both workers with the noisy one, 7 times better than 4 fixed shards, as AWS notes. A tenant that shares one worker retries on its other one and carries on.

The arithmetic improves as the fleet grows. With 100 nodes and 5 per tenant there are C(100,5) = 75,287,520 possible sets. Two randomly placed tenants share no node 77% of the time, one node 21%, and two or more under 2%; sharing all five is a 1-in-75-million event. Route 53 gives each domain 4 of 2,048 virtual name servers, about 730 billion combinations, assigned so no two domains share more than two. A DDoS on one domain hits its four; every other domain keeps at least two healthy ones.

It needs clients that fail over within their shard, and nodes that don’t cascade. Kubernetes deals each flow a “hand” of queues the same way; with hands of 8 from 64 queues and 4 heavy flows, its documentation puts a light flow’s chance of being stuck behind them at about 1 in 2,000.

Cells, and moving the whale

Cells or deployment stamps apply the idea to the whole stack: identical copies of the system, each holding a few hundred tenants, so a noisy tenant hurts only its own cell.

When one tenant is a large, steady share of its pool, stop managing it and move it. Signals: its limits fire daily though it pays well; it is a fifth or more of a shard’s load; it keeps appearing in incident reviews; or its contract promises performance the pool can’t guarantee. Give it a dedicated stamp or silo, the bridge model from tenancy models, with the same code and a price that covers it.

Why autoscaling isn’t the answer

Autoscaling adds capacity; fairness decides who gets it when there isn’t enough. Alone, it falls short four ways:

• Too slow. Instances take minutes; a 2-million-job dump lands in a second. AWS frames load shedding as what buys autoscaling time.
• Unbounded demand. W’s export uses whatever you give it, paid for out of everyone’s margin.
• Bottlenecks that don’t scale: a primary database, a lock on one row, a provider’s 100-SMS-a-second quota.
• Order. Double the workers behind a FIFO queue and the small tenant waits 2,000,000 ÷ 8,000 = 250 seconds instead of 500. Still four minutes.

Putting the layers together

No single control is enough, because each protects a different resource. Stack them from the edge inwards, with a load shedder watching the whole system.

Going deeper: how fairness fails

Limits that punish your best customers

Limits sized from the average hit your largest payers first. AWS names the paradox: a quota both raises and lowers availability, because a throttled tenant sees errors even when the service had capacity to spare. Defences: tiered limits with per-tenant overrides, limits that grow with a tenant’s business, bursting into idle capacity, usage dashboards customers can see, and dark-launching new limits, as Stripe does, to see whom they would hit.

Retry storms after a 429

A client that retries at once doubles the load it was rejected for. Send Retry-After, ship SDKs with exponential backoff and jitter, and give clients a retry budget; the AWS SDK stops retrying when a service throttles it often. One subtlety from AWS: behind a least-outstanding-requests load balancer, a server returning instant errors looks idle and attracts more traffic, so slow those errors to normal latency. See timeouts, retries and backoff with jitter and retry storm.

Fair at one layer, unfair at the next

A fair gateway in front of a shared FIFO queue protects nothing, as W’s export showed. Walk each request’s path (gateway, threads, job queue, workers, connection pool, cache, third-party quotas) and make every shared resource tenant-aware or bulkheaded. Layers can also interact badly: Kubernetes warns that when a flow-controlled server calls another that calls back, priority inversion or deadlock is possible, and recommends classifying the inner calls higher or exempting them.

The infrastructure under you is shared too

Your VMs share hosts, so an unseen co-tenant can steal CPU time, memory bandwidth, disk and network. Your cloud account’s API quotas are shared by all your tenants, so one tenant’s provisioning burst can stall everyone’s onboarding. Azure’s advice to customers applies to you: reserved capacity, a tier with stronger isolation, or single-tenant instances where performance matters. Tenants can also game fairness (30 sign-ups, 30 shares) unless you key on the billing entity, and some are deliberately hostile: the DDoS problem shuffle sharding was invented for. See the noisy neighbour antipattern.

Cheat sheet

How to say it in the interview

“In the pool, one tenant’s burst becomes everyone’s latency, so I design fairness in layers. First visibility: tenant ID on every log, trace and job, and alerts on per-tenant p99. At the gateway, per-tenant rate limits sized by tier, plus concurrency limits on expensive endpoints. Background jobs go into per-tenant queues behind a weighted fair scheduler, so a two-million-job export can’t block a tenant with five jobs. Heavy work gets its own pool; the database gets connection caps and statement timeouts. Under overload I shed by priority. Shuffle sharding and cells bound the blast radius, and a tenant that dominates its pool moves to its own stamp.”

How do you stop one tenant’s batch job starving everyone? Never let it share a FIFO queue. Use per-tenant queues behind a weighted fair scheduler such as deficit round robin, cap each tenant’s concurrent jobs, and run batch work in its own pool. The batch still uses all idle capacity; it just stops cutting the line.

Rate limit or concurrency limit? Both. Rate limits stop request floods cheaply at the edge. Concurrency limits protect resources held for the duration: by Little’s law, a 100-fold slowdown at the same rate means 100 times the work in progress. For reports and exports, the concurrency limit matters more.

When do you move a tenant to a silo? When it is a large, steady share of its pool (say a fifth of a shard), its limits fire daily though it pays well, or its contract needs guarantees the pool can’t make. Limits buy time; a dedicated stamp or silo, priced to cover it, is the lasting fix.

How does shuffle sharding help? Each tenant gets its own random subset of nodes, so two tenants rarely share many. With pairs from 8 workers there are 28 combinations; with 5 of 100 nodes, over 75 million. A flood hits its own subset fully and others only partly, and they fail over to healthy nodes.

Where to go next

• Tenancy models: silo, pool and bridge: pools, silos and moving a whale out.
• Multi-tenant data: hot tenants on shards and DynamoDB partitions.
• Rate limiting and design a rate limiter: the algorithms behind per-tenant limits.
• Backpressure and load shedding: what to drop under overload.
• Bulkheads: separate pools for separate work.
• Cell-based architecture: bounding the blast radius.
• Autoscaling done right: what scaling can and can’t fix.
• Observability: logs, metrics and traces: tenant-tagged telemetry.
• Multi-tenancy and isolation: keeping tenants’ data apart, not just their load.
• Design a multi-tenant SaaS platform: these controls in a full design.

References and further reading

1. AWS Builders’ Library: Fairness in multi-tenant systemsaws.amazon.comFairness as a single-tenant experience, stacked versus hard-allocated quotas, why a sudden surge is usually one tenant, cost-based debiting, retry throttling and the quota paradox.
2. AWS Builders’ Library: Workload isolation using shuffle-shardingaws.amazon.comThe 8-worker example (4 shards versus 28 shuffle shards) and Route 53’s 2,048 virtual name servers with 4 per domain.
3. AWS Builders’ Library: Avoiding insurmountable queue backlogsaws.amazon.comWhy a shared queue defeats isolation, the 30-minute spike that takes 300 minutes to drain, Lambda’s hash-to-a-few-queues trick and spillover queues.
4. Kubernetes documentation: API Priority and Fairnesskubernetes.ioFlowSchemas, priority levels with lendable concurrency, seats for expensive lists, fair queuing of flows with shuffle sharding, and the recursion caution.
5. Azure Architecture Center: Noisy Neighbor antipatternlearn.microsoft.comThe two shapes of the problem, provider and client mitigations, restricting expensive operations, and how to detect it per tenant.
6. AWS Compute Blog: Building resilient multi-tenant systems with Amazon SQS fair queuesaws.amazon.comThe July 2025 launch: MessageGroupId as the tenant key, in-flight imbalance detection, no per-tenant rate cap, and the quiet-group metrics.
7. Amazon DynamoDB Developer Guide: Burst and adaptive capacitydocs.aws.amazon.comPer-partition ceilings of 3,000 reads and 1,000 writes a second, up to 300 seconds of retained burst capacity, and the 400-WCU adaptive capacity example.
8. Stripe: Scaling your API with rate limitersstripe.comThe request rate limiter, concurrent requests limiter, fleet usage load shedder and worker utilization load shedder, with how often each fires.
9. PostgreSQL documentation: Client connection defaultspostgresql.orgstatement_timeout, lock_timeout and idle_in_transaction_session_timeout, and why not to set statement_timeout server-wide.