Noisy neighbour

A noisy neighbour is one tenant, customer, workload or process that takes a disproportionate share of a resource it shares with others, so everyone else slows down or fails even though their own traffic hasn’t changed. It shows up at every layer: a VM on a busy host, a container with no limits, a big customer flooding a pooled API, a hot database partition, a shared queue or cache. More capacity rarely fixes it. Fairness does: per-tenant limits, fair scheduling, and stronger isolation for the tenants who need it.

Picture it first: the society water tank

An apartment society of 40 flats shares one 20,000-litre overhead tank, filled by a pump each night. On a normal morning each flat uses about 300 litres between 7 and 9 am: 12,000 litres in all, with plenty to spare.

Then flat 12 hosts a wedding. Two hundred guests, a catering crew washing vessels, every bathroom busy: the flat draws 9,000 litres before 9 am. The total is 39 × 300 + 9,000 = 20,700 litres, more than the tank holds, and at 8:40 the taps in all 40 flats run dry. The other 39 flats did nothing different. Their pipes are fine. They just share a tank with someone who needed a lot of water that morning.

Every fix costs something. A daily allowance per flat annoys families with a real need, a flow restrictor slows everyone’s bucket, and a separate tank for the caterer sits half-empty most days.

Map it back:

• The tank is any shared capacity: CPU, IOPS, database connections, a partition’s throughput, queue workers, cache memory.
• Flat 12 is the noisy tenant: one customer, job or co-located process.
• The allowance is a quota, the restrictor is a rate limit, filling drums at night is moving batch work off-peak, and the separate tank is a silo.

Where neighbours collide

The same shape repeats at every layer: a finite resource, consumers of very different sizes, and nothing capping any one consumer’s share.

Azure’s guidance describes a second form: no tenant is large, but many peak together and their sum exceeds capacity. That is under-provisioning by another name, with different fixes; the comparison table below separates them.

Symptoms you’d see

The signature: many tenants get worse at once, their own traffic is flat, and the timing lines up with one tenant or job you can name.

• Latency and errors for tenants who changed nothing. Tickets arrive from customers on different plans and features at once; as Azure puts it, a request that succeeds at other times seems to fail at random.
• p99 spikes with no deploy and no growth. The median barely moves; the tail does, because requests queue behind the big tenant’s.
• Steal time on VMs. top shows it as st and mpstat as %steal: time a virtual CPU waited while the hypervisor served another virtual processor.
• Throttling and evictions in containers. A pod’s container_cpu_cfs_throttled_periods_total (from cAdvisor) climbs when it hits its CPU limit; pods get evicted when a node runs short of memory.
• A disk at its cap. IOPS or throughput sits flat at the volume’s provisioned limit while latency per I/O climbs.
• Hot-partition throttling. DynamoDB throttles a partition asked for more than 3,000 read units or 1,000 write units a second, even when the table as a whole has spare capacity.
• Queue age rising for small tenants, whose jobs sit behind one tenant’s bulk import.
• Cache hit rate falling for everyone after one tenant loads a large catalogue.

Why it happens

Nobody designs a noisy neighbour. It is the default outcome of sharing without limits.

Pooling is the business model

Many tenants on one fleet and one database is why SaaS and the cloud are cheap; Amazon’s Builders’ Library calls resource sharing a central benefit of multi-tenant systems. The pool model shares everything, so every resource in it is a shared tank.

Tenants are not the same size

Tenant sizes are usually skewed. Suppose, for illustration, the top 10 of your 1,000 tenants send half the traffic. One of ten giants having an unusual day (a sale, a migration, a buggy sync loop) is a weekly event. Amazon adds that load from different tenants is usually uncorrelated, so when total load jumps abruptly, one tenant is the likeliest cause.

Nothing caps any one tenant

Take a pooled order API serving 1,000 merchants. It handles 8,000 requests a second and normally runs at 4,000, a comfortable 50%. Merchant 417 usually sends 50 a second. At 8 pm it starts a Flipkart-style sale and jumps to 5,000.

Offered load is now 4,000 − 50 + 5,000 = 8,950 req/s, 950 more than the pool can serve. Two things can happen:

• The pool queues the excess. The backlog grows by 950 requests every second. After 10 seconds, 9,500 are waiting, adding 9,500 ÷ 8,000 ≈ 1.2 s to every request; after a minute it is about 7 s, and clients have timed out long before.
• The pool sheds the excess evenly. 950 ÷ 8,950 ≈ 10.6% of every tenant’s requests fail.

Either way, 999 tenants pay for one tenant’s sale. A global rate limit doesn’t help: it protects the servers, not the other tenants, because whoever sends the most still gets the most.

Shared queues serve in arrival order

One queue in front of a worker pool is fair to jobs, not to tenants. If a tenant drops 60,000 jobs in at once, every job behind them waits for all 60,000. Webhook delivery, report exports, email sends and media processing are the usual suspects.

Containers without requests or limits

In Kubernetes, a container’s request is what the scheduler reserves for it and, on a busy node, its weight in the split of CPU time; its limit is a hard ceiling. A pod with neither is BestEffort: it may use whatever the node has spare, and it has almost no weight when the node is contended. A batch pod with a request but no limit can take every idle core and all free memory. Oversubscribed hosts make the same bet one layer down.

Batch and online share a cluster

Exports, reindexing and model training run on the same database, node pool or Kafka cluster as checkout, because a second cluster costs money. A consumer group re-reading a topic from the beginning pulls old segments off disk, and the brokers’ disks and network are shared with every live consumer. Monolithic persistence covers the database version of this mix. And “unlimited” plans remove the one lever you would use.

How to detect it

Aggregate metrics can’t show a noisy neighbour: CPU is high and p99 is up, but nothing says who.

Tag usage with the tenant, carefully

Record per-tenant request rate, errors and latency, plus a cost unit that reflects real resource use: rows scanned, bytes returned, CPU-milliseconds, or a managed service’s own unit. Azure’s example logs Cosmos DB request units per request with the tenant ID. Amazon notes that request counts mislead when operations differ in cost.

Correlate one tenant with everyone’s p99

Put the top 10 tenants’ request rates on the same time axis as everyone else’s p99. If p99 moves with one tenant’s line, as in Figure 1, you have your suspect. Two useful panels, assuming a request counter with a top-N tenant label and cAdvisor’s container metrics:

# Top 10 tenants by request rate (everyone outside the top N is labelled "other")
topk(10, sum by (tenant) (rate(api_requests_total{tenant!="other"}[5m])))

# Share of CPU periods in which each pod was throttled at its own CPU limit
sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
  /
sum by (namespace, pod) (rate(container_cpu_cfs_periods_total[5m]))

Read the throttling ratio carefully. A noisy pod that is throttled is contained; a quiet pod that is throttled has a limit set too low. A noisy pod with no limit never shows up here, so also compare each pod’s CPU use with its request.

Look below the application

• VMs. Run mpstat -P ALL 5 and watch %steal. Steal that rises while your own load is steady means a busy neighbour or an oversubscribed host.
• Databases. PostgreSQL’s pg_stat_statements keys its statistics by role, database and query, so it splits load by tenant only if tenants (or tiers) connect as different roles. In a shared-role pool, set application_name per tenant and sample pg_stat_activity:

-- Who is running right now? The app sets application_name = 'tenant:<id>' per transaction.
SELECT application_name, count(*) AS active, max(now() - query_start) AS longest
FROM pg_stat_activity
WHERE state = 'active' AND application_name LIKE 'tenant:%'
GROUP BY application_name
ORDER BY active DESC
LIMIT 10;

-- If big tenants have their own roles: execution time and disk reads by role.
SELECT r.rolname, round(sum(s.total_exec_time) / 1000) AS exec_seconds,
       sum(s.shared_blks_read) AS blocks_read
FROM pg_stat_statements s JOIN pg_roles r ON r.oid = s.userid
GROUP BY r.rolname
ORDER BY exec_seconds DESC
LIMIT 10;

• DynamoDB. CloudWatch Contributor Insights for DynamoDB reports the most-accessed and most-throttled partition keys, and you can alarm on them. Its throttled-keys mode processes events only when throttling happens, so it is cheap to leave on.
• Queues. Track the oldest job’s age per tenant: a FIFO hides a small tenant’s five-minute wait inside a healthy average.

Telling it apart from look-alikes

They overlap. A busy front end is a noisy neighbour you wrote into your own process, and a retry storm can start when a noisy tenant’s throttled clients retry without backoff.

How to fix it

Ask three questions in order. Can one tenant take more than its share? Limit it. Does one tenant’s work wait in line ahead of others? Schedule fairly. Is the tenant simply too big for the pool? Isolate it.

Put a token bucket per tenant at the gateway

A token bucket refills at a steady rate up to a burst size; each request takes a token, and an empty bucket means rejection (rate limiting compares the algorithms). The anti-pattern version has one bucket for the whole API:

# BEFORE: one bucket for everyone. Whoever sends the most gets the most.
api_bucket = TokenBucket(rate=8000, burst=8000)

def admit(request):
    return None if api_bucket.take() else (503, {})

The fix checks a bucket per tenant, sized by plan, before the global one, so a noisy tenant cannot drain the shared tokens:

# AFTER: a bucket per tenant inside a global bucket. Single-threaded sketch.
import time

PLANS = {"free": (50, 100), "pro": (500, 1_000), "enterprise": (2_000, 4_000)}  # (rate/s, burst)

class TokenBucket:
    def __init__(self, rate, burst):
        self.rate, self.burst, self.tokens = rate, burst, burst
        self.stamp = time.monotonic()

    def take(self, cost=1.0):
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.stamp) * self.rate)
        self.stamp = now
        if self.tokens < cost:
            return False
        self.tokens -= cost
        return True

tenant_buckets = {}
pool_bucket = TokenBucket(rate=7_200, burst=7_200)     # 90% of the pool's 8,000 req/s

def admit(tenant_id, plan, cost=1.0):                  # cost > 1 for expensive calls
    bucket = tenant_buckets.get(tenant_id)
    if bucket is None:
        bucket = tenant_buckets[tenant_id] = TokenBucket(*PLANS[plan])
    if not bucket.take(cost):
        return 429, {"Retry-After": "1"}               # you are over your quota
    if not pool_bucket.take(cost):
        return 503, {"Retry-After": "2"}               # we are full: shed, for everyone
    return None                                        # admitted

Rerun the sale. Merchant 417 is on the pro plan, so 500 of its 5,000 requests a second get through and 4,500 get a 429. The pool carries 3,950 + 500 = 4,450 req/s, 56% of capacity, and the other 999 tenants never notice.

What you pay:

• False throttling. Merchant 417’s sale was real business; Amazon notes that quotas raise the service’s availability while lowering what that tenant sees. Let tenants burst into spare capacity while the pool is quiet (hierarchical buckets, in noisy neighbours and fairness), publish usage so tenants can alarm early, and let them raise quotas before a planned sale.
• Tiers and billing. Quotas become product: plans, upgrades and metering.
• Distributed counting. This bucket lives in one process. Across 10 gateway pods, divide each quota by 10 (assuming even spread) or count in a shared store; design a rate limiter covers both.

Replace one FIFO with fair queues

Say a webhook or export service has 200 workers that finish 200 jobs a second. Tenant A enqueues 60,000 jobs, and tenant B enqueues one right after. In a single FIFO, B waits behind all of A’s: 60,000 ÷ 200 = 300 seconds.

Give each tenant its own queue (or a virtual queue keyed by tenant) and let the scheduler take one job from each active tenant in turn. B now waits for at most one job per other active tenant: with 10 active tenants, about 10 ÷ 200 = 50 ms. A still gets every worker nobody else needs, so its import finishes about as fast as before.

Variants, from simplest to most capable:

• A per-tenant concurrency cap: no tenant may hold more than 20 of the 200 workers. Easy, but B still waits behind A’s jobs in the queue.
• Round-robin or weighted fair queuing: one turn per active tenant, or more turns for higher plans.
• Shuffle-sharded queues: hash each tenant onto a few of many queues, as the Kubernetes API server does (bulkheads describes it).

You pay with more moving parts (a thousand tenants means a thousand queues, real or virtual), no global arrival order, and weights someone must choose.

Give every container a request and a ceiling

On a shared node, the fix is to tell the scheduler and the kernel what each pod needs and how far it may go. Here an export job with a 4-CPU request and no limits shares an 8-vCPU node with a checkout pod that has no requests at all:

# BEFORE: checkout has no requests or limits, so it is BestEffort.
containers:
  - name: checkout
    image: shop/checkout:4.2

# AFTER: checkout is Guaranteed (requests equal limits); the export is capped and low priority.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-low
value: -100                       # below the default of 0, so online pods win
description: "Exports, reports and reindexing"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-export
spec:
  template:
    spec:
      priorityClassName: batch-low
      restartPolicy: Never
      containers:
        - name: export
          image: shop/export:1.9
          resources:
            requests: { cpu: "2", memory: 8Gi }
            limits: { cpu: "4", memory: 8Gi }   # throttled above 4 CPU; killed alone above 8 GiB
# checkout's container gets: requests and limits both { cpu: "2", memory: 4Gi }

Three things changed. Checkout’s request gives it real weight, and because its requests equal its limits it is Guaranteed, the strictest QoS class. The export’s limits make the kernel throttle it at 4 CPU and kill it, not its neighbours, past 8 GiB. And its low priority lets the scheduler preempt it when online pods need room. Add a LimitRange so nobody ships a pod without requests, and a ResourceQuota per tenant namespace (bulkheads shows both). Outside Kubernetes, the knobs are cgroup v2’s cpu.weight, cpu.max, io.weight and io.max.

You pay with reserved headroom (requests often sit idle), throttling even on an idle node (a CPU limit is a hard ceiling), OOM kills when a memory limit is too tight, and tuning. At the VM layer, the equivalents are dedicated hosts for workloads that can’t tolerate steal and, for managed services, reserved throughput or an isolated tier, both on Azure’s list of mitigations.

Design partition keys for your biggest tenant

DynamoDB caps each partition at 3,000 read units and 1,000 write units a second. A write unit is one write of up to 1 KB per second; a read unit is one strongly consistent read of up to 4 KB.

Say your events table uses tenant_id as the partition key and a timestamp as the sort key. Your largest tenant writes 2,500 events a second, each under 1 KB: 2,500 write units on one key against a maximum of 1,000, so about 1,500 writes a second are throttled, and smaller tenants whose keys share that partition can be throttled too. Adaptive capacity can boost a hot partition or isolate a hot item, but never past the maximum, and it won’t split an item collection whose sort key only ever increases.

The fix is write sharding for big tenants: spread writes across tenant_417#0 to tenant_417#4, so each key takes 2,500 ÷ 5 = 500 write units, half the maximum. You pay on reads, which must query all five keys and merge, so shard only the tenants that need it. Reads hit the same cliff: a 20 KB item costs 5 read units per strongly consistent read, so one item tops out at 3,000 ÷ 5 = 600 such reads a second; cache it (hot keys). In sharded SQL, the move is placing and rebalancing hot tenants onto their own shard (database sharding).

Shrink the blast radius: bulkheads, shuffle shards and cells

Limits stop a tenant taking too much; isolation decides how many others share its fate when something still goes wrong.

• Bulkheads per tier. Separate pools, queues or connection pools for free, pro and enterprise traffic, so a free-tier flood can’t reach paying tenants.
• Shuffle sharding. Give each tenant a random pair out of 8 workers. There are 8 × 7 ÷ 2 = 28 pairs, so another tenant shares both of a noisy tenant’s workers about 1 time in 28 (3.6%), against 1 in 4 with four fixed shards of two. Clients must retry on the other worker.
• Cells. Complete copies of the stack, each serving a slice of tenants behind a thin router; with 12 cells a runaway tenant reaches about 1/12 ≈ 8% of customers (cell-based architecture).

You pay with stranded capacity in every compartment, a router to protect, and more to deploy and watch.

Move the whales to their own capacity

Some tenants are simply too big or too spiky for the pool. AWS’s SaaS Lens describes the spectrum: pool shares everything, silo gives a tenant dedicated resources, and bridge mixes the two per service, with noisy-neighbour behaviour named as one reason to silo a service. Most products end up in the middle: pool the long tail, give the largest tenants or premium tiers their own database, cell or fleet.

Silos cost the most: a minimum footprint per tenant with idle headroom, more stacks to patch and deploy (the SaaS Lens stresses that silos must still be onboarded, managed and deployed through one shared system), and a migration each time a tenant outgrows its home. Price dedicated capacity as a premium plan; see routing requests to tenants.

Compare the fixes

Verify the fix

Replay the incident as a load test: hold 999 synthetic tenants at their normal rates and drive one to 10× its quota. Pass if the others’ p99 stays within about 10% of baseline and every 429 goes to the noisy tenant. Repeat with a batch job on a shared node and check that online pods keep at least their requested CPU.

When it’s fine

• Overcommitment is the business model, as long as it has limits: Amazon lets workloads stretch past their allocation while there is spare capacity and holds them to it when there isn’t.
• Best-effort tiers: batch, analytics or free-tier work that is explicitly preemptible and slowed first.
• Internal clusters with one owner and little contention, where a phone call beats a quota system.
• Many small, similar tenants. If no tenant can exceed, say, 0.5% of capacity, none can move your p99 alone; correlated peaks still need capacity planning.

The test: your largest tenant’s or job’s realistic peak fits in the pool’s headroom, and a limit stops anyone who tries to exceed it.

Cheat sheet

How it shows up in interviews

Noisy neighbours hide in any shared design: a multi-tenant SaaS platform, an API gateway, a webhook delivery service (one merchant’s slow endpoint), a notification system (one customer’s 10-million push campaign), a job scheduler, an LLM gateway (one customer’s giant prompts) and a rate limiter.

The probes: “One customer is 40% of your traffic and runs a sale. What happens to everyone else?” “A tenant uploads a million jobs. When does my one job run?” “How would you know which tenant is causing this?” Answer with the arithmetic, then limit, schedule and isolate, naming the costs.

How to say it in the interview

“In a pooled system, one tenant can take a disproportionate share of something everyone needs, and the others suffer although their traffic is flat. If one merchant goes from 50 to 5,000 requests a second on a pool sized for 8,000, all 1,000 tenants slow down. So I’d put a token bucket per tenant at the gateway, sized by plan, checked before a global limit: the merchant gets 429s at 500 a second and the pool stays at 56%. Background jobs go into per-tenant queues served in turn, pods get requests and limits with batch at low priority, and the largest tenants get their own cells or databases. I’d watch per-tenant cost and top-N dashboards. The price is idle headroom and some false throttling, so tenants can see their usage and raise quotas before a sale.”

Why not just add capacity? Capacity raises the ceiling but not the fairness. Without a per-tenant limit, the next big tenant or buggy loop grows into the new headroom, and you are paying for one tenant’s worst day on every server.

How do you choose a tenant’s quota? Start from the plan and the tenant’s observed peak with headroom, and check that no single quota is a large share of the pool: 500 on 8,000 is about 6%. Allow bursts into spare capacity when the pool is quiet, and alarm when tenants approach their limits.

Is it a noisy neighbour or just under-provisioning? Check the top-N. One tenant far above its normal while everyone else is flat needs a limit; everyone growing together needs capacity.

One tenant is 40% of all traffic. Now what? Give it dedicated capacity, a cell or a silo, with a negotiated limit and a capacity plan. Then its sale is its own event, and its bad day can’t touch the pool.

Where to go next

• Noisy neighbours and fairness: hierarchical quotas and fair scheduling in depth.
• Tenancy models: silo, pool and bridge: where each tenant should live.
• Bulkheads: compartments for threads, pools, queues and nodes.
• Cell-based architecture: cells, shuffle sharding and moving tenants.
• Rate limiting and design a rate limiter: the machinery behind per-tenant quotas.
• Multi-tenancy and isolation: the security side of keeping tenants apart.
• Backpressure and load shedding: when everyone peaks at once.
• Busy front end, monolithic persistence and retry storm: the look-alikes.

References and further reading

1. Noisy Neighbor antipattern (Azure Architecture Center)learn.microsoft.comOne tenant’s peak versus many tenants peaking together, client and provider mitigations, and tracking resource use per tenant by putting the tenant ID in telemetry.
2. Fairness in multi-tenant systems (Amazon Builders’ Library)aws.amazon.comPer-tenant rate-based quotas, 429 versus 503, token buckets and bursting into unused capacity, cost-weighted requests and heavy-hitter tracking for high-cardinality keys.
3. Workload isolation using shuffle-sharding (Amazon Builders’ Library)aws.amazon.comEight workers split into four shards versus shuffle shards of two, and why a noisy or poisonous customer fully overlaps very few others.
4. SaaS Lens: Silo, pool, and bridge models (AWS Well-Architected)docs.aws.amazon.comDedicated versus shared tenant resources, and noisy-neighbour behaviour as a reason to silo some services while pooling others.
5. Kubernetes documentation: Resource Management for Pods and Containerskubernetes.ioRequests for scheduling and as a CPU weighting under contention, CPU limits enforced by throttling and memory limits by OOM kills.
6. Kubernetes documentation: Pod Quality of Service Classeskubernetes.ioHow requests and limits make a pod Guaranteed, Burstable or BestEffort, what a BestEffort pod may consume, and when a pod becomes an eviction candidate.
7. Amazon DynamoDB Developer Guide: Burst and adaptive capacitydocs.aws.amazon.comThrottling when one partition gets more than 3,000 reads or 1,000 writes per second, and how adaptive capacity boosts or isolates hot items, but never past that maximum.
8. Amazon DynamoDB Developer Guide: Best practices for designing partition keysdocs.aws.amazon.comRead and write unit sizes (4 KB and 1 KB), the per-partition maximum, and the 20 KB item example.
9. Amazon DynamoDB Developer Guide: CloudWatch Contributor Insights for DynamoDBdocs.aws.amazon.comMost-accessed and most-throttled partition keys, the low-cost throttled-keys mode, and alarms on throttle counts.
10. mpstat(1) Linux manual pageman7.orgThe definition of %steal: involuntary wait by a virtual CPU while the hypervisor serves another virtual processor.