Backpressure and load shedding

Backpressure and load shedding are the two honest answers to work arriving faster than you can finish it. Backpressure sends the problem upstream: the slow stage tells the fast one how much more it can take, and the fast one waits. Load shedding deals with it on the spot: an overloaded service deliberately turns some requests away, quickly, so that the ones it keeps finish in time. Anything else, such as an endless queue, only hides the overload until memory or patience runs out.

Picture it first: the dosa counter at a wedding

At a big wedding the dosa counter is always the bottleneck: one cook, about six dosas a minute, two hundred hungry guests.

A sensible caterer does two things. The cook keeps only eight clean plates on the counter, and you may join the line only by picking one up; no plate, and you wait at your table. That is backpressure: the cook’s real speed travels upstream as plates. And when too many guests are waiting for plates, someone announces “Dosa is 30 minutes, the pulao counter has no wait” and waves the grandparents to the front. That is load shedding, with degradation (pulao) and priority (elders first).

The caterer who does neither gets a line of sixty. Orders are shouted to the cook as guests join, so the griddle is 100% busy, but guests at the back give up and leave, and their dosas go cold. Busy cook, fewer people fed: that gap is what this article is about.

Map it back: the cook is your worker pool, the plates are credits (a window, a prefetch count, a concurrency limit), the line is a queue, the person at the counter is admission control, pulao is a degraded response, and cold dosas are replies to clients that already timed out.

Two answers, and their neighbours

When offered load exceeds what a component can do, the excess must go somewhere: wait in a buffer, slow the sender, or be dropped.

Backpressure is flow control. The receiver tells the sender how much it may send, and excess work waits at the source. You gain bounded memory at every hop and lose nothing. You pay with a pipeline that runs at the speed of its slowest stage, and it only works if the source can wait.

Load shedding is admission control. The overloaded component rejects some work on purpose so the work it accepts finishes in time. You gain steady latency and goodput. You pay with errors for some callers, and you must choose which.

Their neighbours solve different problems:

• Buffering (a message queue) absorbs a burst whose average fits capacity. If the average doesn’t fit, it only delays the failure.
• Rate limiting caps each client at an agreed rate, however busy the server is: fairness, not protection. The SRE book notes it often ignores service health, so in-quota traffic can still overload you (design a rate limiter).
• Autoscaling adds capacity, minutes later, if the bottleneck can scale.
• A circuit breaker is a caller refusing to call a failing dependency: all-or-nothing shedding by the client.
• Graceful degradation sends a cheaper answer instead of none.

The Reactive Streams specification shows where the two meet: a source whose rate can’t be influenced, such as clock ticks, must buffer or drop to respect the consumer’s limits. Users are that kind of source. So use backpressure inside the system, where producers can wait, and shedding at the boundary, where they can’t.

Why overload turns into an outage

Waiting time explodes as you near 100%

Utilisation, ρ (rho), is the fraction of time a server is busy. The classic single-server queue model (M/M/1: random arrivals and service times) gives a brutal result. If each request needs S seconds of work, the average wait in the queue is S × ρ ÷ (1 − ρ), and response time adds S. With S = 50 ms:

Going from 90% to 99% busy, a 10% rise in traffic, multiplies response time by ten. A pool of many workers pushes the knee further right, but the wall at 100% is the same for everyone, and past it there is no steady state at all.

Unbounded queues hide the overload

Above 100%, a queue grows by (arrival rate − service rate) every second, forever. An unbounded queue turns that into rising latency and memory while CPU looks normal and health checks pass, until the process runs out of memory or thrashes in garbage collection and dies, pushing its traffic onto its neighbours.

These queues are often invisible: executor queues inside web frameworks, kernel socket buffers. Amazon’s Builders’ Library advises assuming there are queues you haven’t found yet. The SRE book shows the cost: with a queue ten times the thread count and 100 ms per request, a request joining a full queue takes 1.1 seconds, almost all of it waiting.

Throughput stays high while goodput collapses

Throughput is everything the server processes. Goodput is the part that finishes without error and soon enough for the client to use. Under overload they split apart: the server stays 100% busy, but more and more replies arrive after the client gave up. When the median latency equals the client timeout, half of all requests fail, though the server never stops working.

Retries close the loop

Timed-out clients retry, so offered load rises exactly when capacity is shortest, and retries multiply through layers. In the SRE book’s example, if browser code, frontend and backend each make 4 attempts, one user action can become 4 × 4 × 4 = 64 attempts on an overloaded database. Wasted work, retries and slower servers feed each other until overload is the steady state: congestive collapse, or in services, a retry storm.

It spreads, too. In the SRE book, one cluster fails and its neighbour goes from 1,000 to 1,200 queries per second, past its capacity, and falls over as well. Recovery is harder than prevention: a service that starts crashing at 11,000 QPS won’t recover at 9,000, because only the servers still standing take the load. If 10% survive, traffic must fall to about 1,000 QPS.

A worked example: a tatkal-style spike

A booking service has 200 workers, each request taking 50 ms, so capacity is 200 ÷ 0.05 s = 4,000 requests per second. Normal traffic is 3,000 per second. At 10:00 the tatkal window opens and traffic jumps to 6,000 per second for 10 seconds. The app times out after 2 seconds.

Option 1: one unbounded FIFO queue. The queue grows at 6,000 − 4,000 = 2,000 per second, so a request arriving at second t waits 2,000 × t ÷ 4,000 = t ÷ 2 seconds.

• Requests arriving after second 4 wait over 2 s; their clients are gone before a worker picks them up. From second 6 the workers process only abandoned requests.
• At second 10 the queue holds 2,000 × 10 = 20,000 requests, a 5-second wait.
• Back at 3,000 per second, the queue drains by only 1,000 per second. Waits fall under 2 s once it is below 2 × 4,000 = 8,000: (20,000 − 8,000) ÷ 1,000 = 12 more seconds.

So every request arriving between seconds 4 and 22 fails: 6,000 × 6 + 3,000 × 12 = 72,000 timeouts, half after the spike ended, with the workers 100% busy throughout. If each failed client retries once, the queue never drains.

Option 2: a bounded queue plus shedding. Allow at most 250 ms of queueing: by Little’s law, 4,000 per second × 0.25 s = 1,000 queued requests. When it is full, reject at once with a 503 and Retry-After.

• The queue fills in 1,000 ÷ 2,000 = 0.5 s; then about 2,000 requests per second are turned away, each in well under a millisecond.
• Every accepted request finishes in at most about 300 ms, far inside the timeout.
• At second 10, the 1,000 queued requests drain in 1,000 ÷ 1,000 = 1 second.

The 19,000 is 60,000 arrivals − 40,000 served − 1,000 still queued at second 10. A 50% overload for 10 seconds caused 18 seconds of outage with the unbounded queue and none for accepted users with shedding; rejected users retry with backoff and jitter once there is room.

Backpressure, from the wire up

Every backpressure mechanism has the same shape: the receiver grants credit, the sender spends it, and the sender stops when it runs out. Only the unit changes.

TCP: the receive window

Every TCP segment carries a window field: how many more bytes its sender will accept (RFC 9293). When an application stops reading its socket, the kernel buffer fills, the advertised window falls to zero, and the other side must stop sending, apart from occasional zero-window probes, spaced ever wider, to learn when it reopens. The RFC lets a receiver keep its window closed indefinitely; the sender must keep the connection open as long as probes are acknowledged.

So a slow reader at one end of a TCP connection eventually blocks the writer’s write() call. It is free backpressure, but coarse: per connection, in bytes, blind to which request matters.

HTTP/2: credit per stream and per connection

HTTP/2 multiplexes many streams over one connection, so one slow stream must not stall the rest. RFC 9113 adds credit at two levels, each stream and the whole connection, both starting at 65,535 bytes. Only DATA frames spend credit, so control frames never block, and the receiver tops credit up with WINDOW_UPDATE frames. It is hop by hop: a proxy keeps separate windows on each side.

Inside a process: bounded buffers and request(n)

Between threads, the simplest backpressure is a bounded blocking queue: when full, the producer’s put() blocks. The Kafka producer works this way: records wait in buffer.memory (32 MiB by default), and when it is full, send() blocks for up to max.block.ms (60 seconds by default), then throws (producer configs).

Blocking threads is costly in asynchronous code, so Reactive Streams (Java 9’s Flow interfaces are equivalent) makes demand explicit. The subscriber calls request(n), and rule 1.1 says the publisher must never send more items than requested. Having requested P items and processed N, a subscriber can receive at most P − N more, so its buffer is bounded by construction. The spec calls this a “dynamic push-pull” stream: push while demand is outstanding, effectively pull when it isn’t.

Brokers: pull, prefetch and blocked publishers

Kafka consumers pull. Kafka’s design notes say push tends to overwhelm a consumer that falls behind, while with pull “the consumer simply falls behind and catches up when it can.” The backlog becomes consumer lag on the broker’s disk, bounded by retention, so alarm on lag well before old segments are deleted. Brokers also throttle clients over quota: they compute how long the client must wait, return that delay at once, and stop reading from its connection until it passes.

RabbitMQ pushes, bounded by prefetch. A consumer’s prefetch count (basic.qos) caps its unacknowledged messages; at the cap, deliveries pause. Publishers get automatic flow control, blocked and unblocked several times a second when they outrun their queues. If a node’s memory passes the high watermark (by default about 60% of RAM) or free disk falls below its limit (50 MB by default), a resource alarm blocks every publishing connection in the cluster while consumers keep draining. Publishers just hang, so use publisher confirms and timeouts, and separate publishing and consuming connections.

Backpressure stops at the user

Backpressure needs a sender that can wait. A pipeline stage or a Kafka consumer can; a person on a phone during a sale can’t. In a synchronous call chain, waiting just moves the queue upstream: while C is slow, B holds a thread and a connection per waiting request, and so does A, filling thread pools and connection pools.

So in request-response systems, turn backpressure into fast rejection at each boundary:

• Bound every queue, in time as well as length.
• Cap concurrency per service. Little’s law gives the number: 4,000 per second × 50 ms = 200 in flight. The 201st request gets a 503, not a place in line.
• Cap outstanding calls per dependency in the caller. In Maurer’s “Fail at Scale”, clients count in-flight requests per service and fail new ones immediately past a limit, so one slow service can’t take all the caller’s threads: a bulkhead.
• Say why. An explicit “overloaded” error lets callers back off or degrade instead of waiting for a timeout.

Load shedding, done well

Shedding is easy to do badly: at random, after most of the work is done, with clients retrying at once. Each technique below fixes one of those mistakes.

Reject early and cheaply

Check load before parsing the body, calling auth or touching a database, so a rejection costs a tiny fraction of a real request. Return 503 Service Unavailable for overload (RFC 9110 defines it for “temporary overload”) and 429 Too Many Requests for a client over quota, both with Retry-After. Amazon warns that a stray log line can make dropping surprisingly expensive, so load-test the rejection path too.

Shed by priority, not at random

Google’s SRE book tags every request with one of four criticality levels, set at the frontend and propagated with every downstream call:

• CRITICAL_PLUS: failure causes serious user-visible harm.
• CRITICAL: the default for production traffic; capacity is planned for it.
• SHEDDABLE_PLUS: partial unavailability expected; the default for batch jobs that can retry hours later.
• SHEDDABLE: frequent partial and occasional full unavailability expected.

An overloaded task rejects lower criticalities first, at lower utilisation thresholds, and per-customer quotas make global overload hurt the customers exceeding their share. Amazon adds two rules: the load balancer’s health check matters most, and requests that finish work beat requests that start it, so checkout() beats addToCart() and page 5 of a listing beats page 1.

Drop work whose deadline has passed

Send a deadline with every request and pass the remainder down: given 30 seconds, A spends 7 and calls B with 23; B spends 4 and calls C with 19. Each server checks the time left after dequeuing and before each expensive stage, and drops what can’t finish. In the worked example, workers would skip abandoned requests in microseconds instead of spending 50 ms on each.

Bound time in the queue: CoDel and adaptive LIFO

“Fail at Scale” describes a queue that sets its own timeout, a variant of the CoDel algorithm from network routers. If the queue was empty at some point in the last N = 100 ms, it is absorbing a burst, and requests may wait 100 ms. If not, a standing queue has formed, and requests may wait only M = 5 ms. The paper reports these values working across many services without tuning. Adaptive LIFO pairs with it: once a queue forms, serve the newest request first, because the oldest one’s user has probably given up.

M, N = 0.005, 0.100            # 5 ms when overloaded, 100 ms when calm

def admit(req, queue, now):
    overloaded = queue.last_empty_time < now - N
    req.expires = now + (M if overloaded else N)
    if overloaded:
        queue.push_front(req)  # adaptive LIFO: newest is served next
    else:
        queue.push_back(req)   # normal FIFO

def next_request(queue, now):
    while queue:
        req = queue.pop_front()
        if now < req.expires and now < req.client_deadline:
            return req
        req.reject(503)        # stale: cheap to drop, pointless to serve
    return None

A real queue also needs a hard length cap and a sweep for expired entries at the back.

Adaptive concurrency limits

A fixed limit goes stale as code, hardware and dependencies change. Netflix’s concurrency-limits library treats the limit like a TCP congestion window, tuned by measured latency. Its Vegas-style limiter estimates the queue as limit × (1 − minRTT ÷ sampleRTT): with a limit of 100, a best-case latency of 50 ms and a current 55 ms, 100 × (1 − 50 ÷ 55) ≈ 9 queued. Above a threshold (typically 4 to 6) the limit drops by one; below another (2 to 3) it grows by one. It settles where latency starts rising, and the excess is rejected at once (UNAVAILABLE in gRPC, 429 over HTTP). Limits can be partitioned, say 90% guaranteed to live traffic and 10% to batch.

Client-side throttling and retry budgets

Rejection is never free: when a request is a cheap RAM lookup, refusing it costs nearly as much as serving it, and a server can overload itself just saying no. So stop traffic at the client. In Google’s adaptive throttling, each client counts, over two minutes, requests (attempts by the application) and accepts (accepted by the backend), and rejects new requests locally with probability:

max(0, (requests − K × accepts) ÷ (requests + 1))

With K = 2, a client that tried 3,000 requests while 1,000 were accepted drops (3,000 − 2,000) ÷ 3,001 ≈ 33% locally. With K = 2 the backend still receives about one request it rejects for every one it accepts, so up to half its effort can go on saying no. Where rejecting costs nearly as much as serving, lower K: at 1.1, the SRE book notes, only one request is rejected for every 10 accepted. A higher K throttles less and notices recovery sooner.

Add retry budgets: at most 3 attempts per request, and retries capped at 10% of each client’s requests. In the SRE book’s worst case, the per-request cap alone lets load grow to almost 3×; with the 10% cap, about 1.1×. Retry only at the layer directly above the failure, and return “overloaded; don’t retry” when retrying is hopeless. More in delivery guarantees.

Degrade before you drop

A cheaper answer often beats an error. During a Big Billion Days-style sale, switch off the personalised recommendations carousel, serve category pages from cache, and show “in stock” instead of live counts. Put each degradation behind a flag and exercise it regularly: as the SRE book notes, the code path you never use is often the one that doesn’t work.

Choosing the right tool

Going deeper: running it in production

Where to shed

Protect in layers. The edge rejects most cheaply but knows least; the service knows most but pays more to reject. Amazon keeps proxy connection limits high, lets the server shed with local knowledge, and relies on the layers in front to stop volumes a server can’t afford even to refuse.

What to measure

• Queue age (how long the oldest request has waited) and in-flight concurrency: they move before CPU does.
• Goodput beside throughput, and rejections by reason, client and criticality. Amazon aims for zero rejections while capacity is spare.
• Latency of successful requests only. A service shedding 60% of traffic can show a great median while its real requests are slow.

Pitfalls

• Shedding health checks. If the load balancer’s ping times out, the node is marked dead, the rest get its traffic, and the fleet shrinks mid-overload (health checks done right).
• Blinding the autoscaler. If shedding holds CPU at the autoscaler’s target, it never scales out. Scale on demand signals, including rejections.
• Hiding missing headroom. A shedding fleet can look calm while too close to its limit to survive losing a zone.
• Rejecting late, after TLS, auth and parsing, or retrying at every layer without a budget.

Test past the knee

Load-test far beyond the point where goodput stops rising. The SRE book’s bar: a task keeps serving its provisioned rate, without significant extra latency, however much excess arrives, up to 2× or even 10× its capacity. Amazon also holds load steady while removing servers, raising per-host load without a bigger load generator.

Cheat sheet

How to say it in the interview

“When load exceeds capacity, I make the excess go somewhere explicit. Inside the pipeline that’s backpressure: Kafka consumers pull, so a slow consumer just builds lag, which I alarm on against retention. At the API, users can’t wait, so I shed. Each service has a concurrency limit from Little’s law, 4,000 requests a second times 50 milliseconds is 200 in flight, and a queue bounded at about 250 milliseconds of wait. Beyond that, requests get an instant 503 with Retry-After. Requests carry a deadline and a criticality, so expired and sheddable work goes first and checkout goes last. Clients have retry budgets and adaptive throttling, so rejections don’t become a retry storm. And before a sale I degrade: recommendations off, category pages from cache.”

Isn’t load shedding just rate limiting? No. A rate limit caps each client at an agreed rate whether or not the server is busy. Shedding reacts to the server’s actual load and protects it from everyone at once, including in-quota traffic.

Why not just make the queue bigger? A bigger queue adds waiting, not capacity. Past the client timeout every queued request is wasted work; in the tatkal example, 10 seconds of overload became 18 seconds of outage.

How do you choose the concurrency limit? Start from Little’s law with throughput and latency measured at the knee, then let an adaptive limiter track it, because the right value moves with every deploy and every slow dependency.

What stops rejected clients from making it worse? Retry-After, backoff with jitter, at most 3 attempts, a 10% retry budget, retries at one layer only, and client-side throttling so most excess never reaches the network.

When is backpressure the wrong answer? When the producer is a person or a system that won’t wait, or when waiting would pin threads and connections all the way up a synchronous chain. Then reject fast at the boundary.

Where to go next

• Message queues: load levelling, Little’s law sizing and scaling on queue age.
• Delivery guarantees and DLQs: retries without a storm.
• Rate limiting and design a rate limiter: per-client caps and 429s.
• Timeouts, retries and backoff with jitter: deadlines and polite clients.
• Circuit breaker and bulkheads: containing a slow dependency.
• Retry storm: how retries sink a recovering service.
• Autoscaling done right: adding capacity without fighting your shedder.
• Noisy neighbours and fairness: per-tenant shares when capacity is scarce.
• Design a flash sale: all of this under a single, predictable spike.
• Kafka: consumer lag, retention and quotas.

References and further reading

1. Handling Overload (Google SRE book, chapter 21)sre.googlePer-customer limits, client-side adaptive throttling and its rejection formula, the four criticality levels, utilisation signals and per-request and per-client retry budgets.
2. Addressing Cascading Failures (Google SRE book, chapter 22)sre.googleHow overload cascades, queue sizing, load shedding and graceful degradation, LIFO and CoDel, retry amplification and deadline propagation.
3. Using load shedding to avoid overload (Amazon Builders’ Library)aws.amazon.comGoodput versus throughput, the overload feedback loop, the cost of rejecting, timeout hints, bounding time in queues, layered protection and testing to breakage.
4. Fail at Scale (Ben Maurer, ACM Queue, 2015)queue.acm.orgA large social network’s controlled-delay queue (5 ms and 100 ms), adaptive LIFO and per-service client concurrency caps.
5. Netflix concurrency-limits (GitHub README)github.comAdaptive concurrency limits borrowed from TCP congestion control, Little’s law, the Vegas queue estimate and live-versus-batch partitions.
6. Reactive Streams for the JVM, specification 1.0.4github.comDemand signalled with request(n), the rule that a publisher never sends more than requested, and buffer-or-drop for sources that cannot slow down.
7. RFC 9113: HTTP/2, Section 5.2 Flow Controlrfc-editor.orgCredit-based, hop-by-hop flow control per stream and per connection with WINDOW_UPDATE; 65,535-byte initial windows; only DATA frames count.
8. RFC 9293: Transmission Control Protocolrfc-editor.orgThe receive window field, why receivers should not shrink it, and zero-window probing that keeps a paused connection open.
9. RabbitMQ documentation: Memory and disk alarmsrabbitmq.comResource alarms block publishing connections cluster-wide while consumers keep draining; links to flow control and the memory threshold guide.
10. Apache Kafka 4.2 documentation: Designkafka.apache.orgWhy Kafka consumers pull (a slow consumer just falls behind) and how brokers throttle clients over quota by delaying responses and muting the channel.