Retry storm

A retry storm happens when many callers retry failed requests at once, so a dependency that is already struggling gets several times its normal traffic exactly when it can least afford it. Each retry looks harmless, but multiplied across layers and synchronised across clients, retries can keep a service down long after whatever broke it has been fixed. The cure is to treat retries as a scarce resource: one layer owns them, they spend from a budget, they back off with jitter, and servers shed load before they drown.

Picture it first: the tatkal refresh

Tatkal booking opens at 10 am and thousands of people wait on the booking page. The site slows: a page that loads in one second now takes eight. Nobody waits eight seconds. They hit refresh, then refresh again.

Each refresh is a new request, and the server cannot tell the old one was abandoned, so it keeps building pages for tabs that no longer exist. With three requests per person, two-thirds of its work is wasted, and pages get slower still, so people refresh more.

By 10:05 the rush is over, yet the site still crawls. The rush started the problem; the refresh loop sustains it. Map it back:

• Refresh is a retry, often automatic and at several layers at once.
• The abandoned tab is a request whose caller timed out but whose work the server still finishes.
• The 10 am rush is the trigger; the refresh loop is the sustaining effect. Break the loop, because the next trigger will be something else.

Symptoms you’d see

A storm’s signature is that cause and effect run in the opposite order to a normal overload:

• Dependency traffic rises after errors begin. In a real spike, load rises first and errors follow. In a storm, errors start, then requests to the failing dependency jump 2× or 3× while edge traffic stays flat.
• Errors climb instead of recovering, and the outage outlives its trigger. The blip is over, the failover has finished, and the service is still down.
• Busy CPUs, little useful work. Throughput (requests handled) looks healthy, but goodput (Amazon’s term for successful responses that arrive in time to be used) is near zero. Logs show 200 OKs written after the caller had gone.
• Old queues. The oldest queued request is older than the caller’s timeout, so everything dequeued is already dead.
• Latency piles up at the timeout. Client histograms spike at exactly the timeout (say 500 ms) and its multiples, while server latency looks merely slow.
• Synchronised waves: a sawtooth at the retry interval, or a cliff when thousands of clients reconnect after an outage or a deploy.

Beware fast failures: quick rejections make median latency look excellent, so keep failed requests out of latency metrics.

Why it happens

Retries are a good idea in a healthy system. If 1% of requests fail independently, one retry turns that into 1% × 1% = 0.01%, for about 1% more traffic. The maths breaks when failures are correlated, as when a dependency fails because it is overloaded: then every retry adds load to the thing that is drowning. Amazon’s Builders’ Library calls retries “selfish”.

Retries multiply across layers

Each team adds its own retries, and attempts multiply rather than add. A food-delivery checkout might go mobile app → API gateway → order service → database. If the first three each make up to three attempts, the database sees 3 × 3 × 3 = 27 queries for one tap.

Published examples are worse. Amazon’s Builders’ Library takes a five-deep stack with three tries per layer: 3^5 = 243× the database load. Google’s SRE book counts three layers with 4 attempts each: 4^3 = 64. Bronson et al. describe a system whose added retries and failover destinations produced a worst-case work amplification of over 100×.

Retries fire immediately, and together

A loop with no wait retries in the same millisecond, before the dependency can recover. Backoff helps, but clients that failed together and wait the same fixed delay also retry together: a wave every second instead of a trickle. Jitter flattens the waves; the mechanics live in timeouts, retries and backoff with jitter.

Herds also form without errors: deploys that restart every instance, scaling events that open thousands of connections at once, cron jobs at the top of the minute, phones reconnecting after an outage. If 5 million phones following a cricket match all reconnect after a fixed one second, the edge faces about 5 million TLS handshakes at once; spread over a random 60 seconds, that is 5,000,000 ÷ 60 ≈ 83,000 per second.

Timeouts shorter than the work

If the caller’s timeout is shorter than the server’s processing time under load, each attempt is wasted twice: the caller retries, and the server finishes the original because nobody told it to stop. An order service that normally answers in 120 ms but has a 700 ms median under load, behind a 500 ms timeout, does up to three full units of work per request while goodput heads to zero. Late answers earn no credit.

Retrying errors that can never succeed

A 400, 401, 403, 404 or 422 will fail the same way next time, so retrying it triples load for nothing. Retrying a 429 or 503 immediately ignores the one explicit signal the server sent. Google’s SRE book and Azure’s guidance agree: classify errors, and never retry a malformed request.

Retries you didn’t write

Many retries live in code nobody on the team reads:

• Cloud SDKs. AWS SDKs make 3 attempts by default in standard mode: one request plus two retries.
• RPC frameworks. gRPC transparently retries RPCs that never reached the server’s application code, even with no retry policy configured.
• Meshes and proxies. Istio retries HTTP requests twice by default; an Envoy route with a retry policy defaults to one retry.
• Everything else: load balancers resending to another backend, queues redelivering messages, people pressing refresh.

App code, SDK and sidecar each trying three times is Figure 1’s 27× tree, built by a team that believes it has “three retries”.

Why sensible teams fall into it

Each retry decision is local and reasonable; nobody owns the product. The multipliers are zero on good days, so happy-path load tests miss them, and “add a retry” fixes independent failures, so it is a popular postmortem action. As Bronson et al. note, metastable failures often come from features added to improve reliability.

Metastable failures: when the storm feeds itself

Nathan Bronson, Abutalib Aghayev, Aleksey Charapko and Timothy Zhu named this pattern in “Metastable Failures in Distributed Systems” (HotOS 2021). A system is stable if it recovers by itself once a trigger ends. It is vulnerable if it is healthy but a big enough trigger can tip it over; many systems run here on purpose because it is efficient. It is metastable once it is stuck in a failure state that persists after the trigger is gone, because a sustaining effect, usually work amplification such as retries, holds load above capacity. Only a strong push, such as a deep cut in load, gets it out.

Their retry example is worth memorising. A database answers in under 100 ms below 300 queries per second (QPS), and an order of magnitude slower above that. A web app sends one query per request plus one retry if there is no answer within one second. At 280 QPS, a 10-second network outage causes a surge when connectivity returns, latency passes one second, and every request now costs two queries: 280 × 2 = 560 QPS against a 300 QPS database. Every query times out, goodput is zero, and it stays there.

To recover, load must fall below 150 QPS (150 × 2 = 300) or retries must stay under 20 QPS (280 + 20 = 300). The paper calls 300 the advertised capacity and 150 the hidden capacity, the limit below which the system heals itself.

The paper’s key lesson for postmortems: the trigger is not the root cause. A network blip, a garbage-collection pause, a deploy or a cache flush can each tip a vulnerable system over; fix only the trigger and the next one brings the same outage.

How to detect it

Put these metrics on one dashboard

Most of these already exist. gRPC’s OpenTelemetry metrics count grpc.client.attempt.started per attempt and grpc.client.call.duration per call, so their ratio is the attempt multiplier; Envoy counts upstream_rq_retry and upstream_rq_retry_overflow (retries refused by a budget or breaker). Names below follow the OpenTelemetry Prometheus exporter:

# Attempts per logical call to the order service; alert if above 1.5 for 2 minutes.
sum(rate(grpc_client_attempt_started_total{grpc_method=~"orders.OrderService/.*"}[1m]))
/
sum(rate(grpc_client_call_duration_seconds_count{grpc_method=~"orders.OrderService/.*"}[1m]))

Slice server-side counts by client too: Azure’s guidance shows one client IP sending over 80,000 requests in an hour during a storm. Google’s backends read each request’s attempt number; a surge of second and third attempts says their peers are overloaded too.

What a trace looks like

In a distributed trace, a storm is a staircase: one parent span with several child spans to the same dependency, each ending at exactly the timeout. OpenTelemetry’s HTTP conventions put http.request.resend_count on each repeated attempt, so you can search for it. On the server side, look for spans that finish after the caller’s span ended: work nobody used.

In the database

At the bottom of the stack, a storm looks like many copies of the same statement running at once. In PostgreSQL:

-- Identical statements running concurrently, and the oldest one's age.
SELECT left(query, 60)          AS statement,
       count(*)                 AS running_copies,
       max(now() - query_start) AS oldest
FROM pg_stat_activity
WHERE state = 'active'
GROUP BY 1
ORDER BY running_copies DESC
LIMIT 5;

Hundreds of copies, the oldest older than any caller’s timeout, means the database is working for callers who have left.

Load-test signature

Ordinary load tests never fail anything, so they never find a storm. Add two experiments, using the real SDKs and sidecars. Push offered load to twice capacity: goodput should plateau, not fall towards zero. And at 70–90% of capacity, inject a 10-second fault and watch what happens after it ends: a stable system recovers by itself, a vulnerable one stays down.

Telling it apart from look-alikes

They also combine: a cache flush slows the database and the timeouts trigger retries. Bronson et al. describe a look-aside cache that never refills because the overloaded database cannot answer in time; see hot keys, cache stampede and invalidation.

How to fix it

Two jobs: escape a running storm, then prevent the next one.

Get out of a running storm

Google’s SRE book is blunt: you must dramatically reduce or eliminate load until the retries stop and the backends stabilise. Cuts that leave load above the hidden capacity usually achieve nothing.

1. Cut load hard, at the edge. Shed a large share of traffic at the gateway with 503 and Retry-After, or drain the cluster.
2. Turn retries off with a config flag, if you built one; a code push mid-outage is slow.
3. Drop queued work older than the callers’ timeouts; it is already dead.
4. Ramp back in steps, say 10% → 25% → 50% → 100%, watching goodput and queue age, and warm caches first.

Don’t count on autoscaling: new instances start with cold caches, and stateful systems lose capacity while they rebalance.

Retry at one layer

For each call path, one layer owns retries and every other layer tries once and fails fast. Google retries only at the layer immediately above the one rejecting; Amazon retries at a single point in the stack. When the owner gives up, it returns an explicit “overloaded; don’t retry” error.

The cost is coordination: someone must audit SDK, mesh and proxy defaults. Disabling a mesh retry is one line:

# Istio VirtualService: the order service's caller owns retries, not the mesh.
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: orders
spec:
  hosts: ["orders.shop.svc.cluster.local"]
  http:
    - route:
        - destination:
            host: orders.shop.svc.cluster.local
      timeout: 2s
      retries:
        attempts: 0   # 0 disables retries; Istio's default would retry twice

Cap attempts, back off, add jitter

Cap attempts at two or three, wait exponentially longer between them up to a ceiling, and randomise each wait. The AWS SDKs’ updated retry behaviour (opt-in during 2026) waits random(0, 1) × min(20 s, base × 2^retry), with a 50 ms base for transient errors and 1 s for throttling. Backoff alone only delays a storm: once every client sits at the cap, they all retry at the capped rate. Details are in timeouts, retries and backoff with jitter.

Put retries on a budget

A per-request cap bounds amplification at 3×; a retry budget bounds the whole client by allowing retries only while they are a small share of traffic. Google combines both, 3 attempts per request and retries under 10% of a client’s requests, cutting worst-case growth from just under 3× to about 1.1×. Common forms:

• gRPC retryThrottling. A per-server token count starts at maxTokens; failures subtract 1, successes add tokenRatio, and retries pause at or below half of maxTokens. With 10 and 0.1, retries stop once failures pass about 10% of successes.
• AWS SDK retry quota. A 500-token bucket per client. In the updated behaviour, a transient-error retry costs 14 tokens, a throttling retry 5, a successful retry gets its cost back, and a first-try success adds 1 token; an empty bucket means no retries.
• Envoy retry budgets. Once enabled, a budget caps concurrent retries at 20% of active and pending requests by default, with a minimum of 3. Envoy recommends budgets to avoid retry storms.
• Google’s adaptive throttling. Clients drop requests locally with probability max(0, (requests − K × accepts) / (requests + 1)) over the last two minutes, typically with K = 2.

The same ideas in client code, before and after:

# BEFORE: retries at once, on every error, forever.
def fetch_menu(restaurant_id):
    while True:
        try:
            resp = http.get(f"{MENU_URL}/menus/{restaurant_id}", timeout=0.5)
            if resp.ok:
                return resp.json()
        except Exception:
            pass

# AFTER: capped, jittered, budgeted, deadline-aware, only for retryable errors.
import random, time

RETRYABLE = {429, 502, 503, 504}

class RetryBudget:
    """Token bucket in the style of gRPC retry throttling (not thread-safe as written)."""
    def __init__(self, max_tokens=10, refill=0.1):
        self.max, self.refill, self.tokens = max_tokens, refill, max_tokens
    def success(self):
        self.tokens = min(self.max, self.tokens + self.refill)
    def failure(self):
        self.tokens = max(0, self.tokens - 1)
    def can_retry(self):
        return self.tokens > self.max / 2

budget = RetryBudget()

def fetch_menu(restaurant_id, deadline):
    for attempt in range(3):
        left = deadline - time.monotonic()
        if left < 0.05:
            break
        try:
            resp = http.get(f"{MENU_URL}/menus/{restaurant_id}", timeout=left,
                            headers={"Request-Timeout-Ms": str(int(left * 1000))})
        except (ConnectionError, TimeoutError):
            resp = None
        if resp is not None and resp.ok:
            budget.success()
            return resp.json()
        if resp is not None and resp.status_code not in RETRYABLE:
            raise MenuError(resp.status_code)          # e.g. a 4xx: retrying won't help
        budget.failure()
        if attempt == 2 or not budget.can_retry():
            break
        wait = random.uniform(0, 0.1 * 2 ** attempt)   # full jitter
        wait = max(wait, retry_after_seconds(resp))    # server's hint; 0 if none
        if wait >= deadline - time.monotonic():
            break
        time.sleep(wait)
    raise MenuUnavailable(restaurant_id)               # fail fast, upwards

You pay with some rescuable requests failing fast during wide failures, and each client sees only its own traffic.

Circuit breakers

A circuit breaker stops calling a dependency after a failure threshold, fails instantly for a cool-down period, then lets a few probes through. It stops a crowd from hammering something that is clearly down. Amazon’s Builders’ Library names the cost: modal behaviour that is hard to test and can lengthen recovery. Amazon prefers a local retry token bucket, built into the AWS SDKs since 2016.

Shed load early and cheaply on the server

Old app versions, partners and scripts will retry however they like, so servers must protect themselves:

• Admission control. Cap in-flight requests or queue time, and reject the excess with 503 before doing expensive work.
• Cheap rejection. No stack traces, synchronous logging or DNS lookups on the error path; Bronson et al. list slow error handling as a sustaining effect in its own right.
• Drop doomed work. Keep queues short (Google suggests at most about half the thread-pool size for steady traffic), check the deadline before each stage, and consider LIFO or CoDel-style queues.
• First attempts before retries. If requests carry an attempt number, shed retries first; Bronson et al. show this breaks their example loop.

You pay with deliberate 503s, a reject path to build and test, and tuning: shedding at the CPU level that drives autoscaling hides the autoscaler’s signal. More in backpressure and load shedding.

Honour Retry-After, 429 and 503

RFC 9110 lets a server send Retry-After with a 503, in seconds or as a date, and RFC 6585 allows it on a 429. Clients should wait at least that long. But if every client hears Retry-After: 5, they all return together, so jitter the value on the server or add jitter on the client. gRPC’s equivalent is grpc-retry-pushback-ms, where a negative value means “don’t retry”.

Propagate deadlines so servers drop abandoned work

Set one deadline at the edge and pass the remaining time down. If the gateway allows 2 s and the order service spends 0.5 s before calling payments, payments gets 1.5 s; if the request then waits 1.6 s in a queue, payments drops it unprocessed. gRPC sends remaining time rather than a timestamp, avoiding clock skew, and its Java and Go libraries propagate deadlines by default. Add cancellation so downstream calls stop when the caller gives up; that removes the wasted bar in Figure 3.

Make retries safe with idempotency keys

On writes, retries risk doing things twice, which is why RFC 9110 says clients should not automatically retry non-idempotent requests unless they know it is safe. Send an idempotency key per logical operation, reuse it on every attempt, and return the stored result for keys already seen. That makes a retried payment safe but does nothing for load, so pair it with a budget.

Compare the fixes

Verify the fix worked

Rerun the fault-injection test before and after, for example 2,000 requests per second at the edge with every database query failing for 10 seconds:

• Database attempts during the fault: before, up to 2,000 × 27 = 54,000 per second; after, about 2,000 × 1.1 = 2,200 per second.
• Retry-refusal counters, such as Envoy’s upstream_rq_retry_overflow, should rise: the budget is working.
• Recovery after the fault ends: before, possibly never without an operator; after, within seconds.
• Goodput at 2× overload should plateau near capacity instead of collapsing.

When retries are fine

Retries remain the right tool for independent, transient faults: a dropped connection, one bad host, a leader election. Google even retries immediately when only a few backend tasks are overloaded, because the retry probably lands on an idle task, and relies on its budgets when many are. The rule: a retry is fine when it is bounded, spread out, safe and owned by one layer.

A public example: AWS, December 2021

AWS’s summary of the 7 December 2021 event in its Northern Virginia (us-east-1) region shows a storm’s anatomy. At 7:30 AM PST, an automated activity to scale capacity of a service on the main AWS network triggered unexpected behaviour from a large number of clients on AWS’s internal network. The surge of connection activity overwhelmed the networking devices between the two networks; the delays raised latency and errors, which caused even more connection attempts and retries, and the congestion persisted.

Real-time monitoring was impaired, so operators worked from logs. Congestion improved significantly by 1:34 PM, and the devices had fully recovered by 2:22 PM PST. AWS wrote that its clients’ well-tested back-off behaviour had been defeated by a latent issue in a long-standing code path. It disabled the triggering scaling activity, committed to a fix for the back-off issue, and added network configuration to protect the devices from a similar surge.

The lessons travel well: exercise back-off code with fault injection, keep observability off paths that can congest, and fix the sustaining effect, not only the trigger.

Cheat sheet

How it shows up in interviews

Retry storms arrive as follow-ups to any design with a call chain: a payment flow, a flash sale, a microservice checkout, a chat app with millions of connections, or a rate limiter (“what should clients do with your 429?”). The classic probe: “the database fails over for 30 seconds; walk me through the minute after it returns.” Count the attempts, name the loop, and bound it.

“Retries help with random failures, but in an overload they’re extra load on the thing that’s failing. If the app, gateway and service each try three times, the database sees 27 queries per tap. So one layer owns retries: the order service, at most three attempts with capped, jittered backoff, on a budget that keeps retries under about 10% of traffic. Everything else tries once and fails fast. The gateway and database shed load early with 503 and Retry-After, deadlines propagate, and writes carry idempotency keys. I’d alert on attempts per logical request, and on dependency traffic rising while edge traffic is flat.”

Why not add more retries to improve availability? Retries only help when failures are independent. When a dependency fails because it is overloaded, each retry adds load, so more retries lower availability and can make the outage self-sustaining.

Isn’t exponential backoff with jitter enough? It spreads retries out but does not bound total volume; with every client at the capped delay, load can still sit above capacity. Add a per-request cap, a per-client budget and server-side shedding.

How do you tell a retry storm from a real traffic spike? By the order of events. In a real spike, edge traffic rises first and attempts per request stay near 1. In a storm, errors come first, the edge is flat, and attempts per request and the dependency-to-edge ratio jump.

The trigger is fixed but the service is still down. What now? It is metastable. Cut load far below normal at the edge, turn retries off, drop queued work past its deadline, and ramp back in steps while watching goodput. Then fix the sustaining effect.

Where should retries live in a five-service chain? In one place per call path, usually just above the dependency that fails. Every other layer makes one attempt and passes an “overloaded, don’t retry” error upwards, so attempts never multiply.

Where to go next

• Timeouts, retries and backoff with jitter: timeouts and backoff curves in depth.
• Circuit breaker: states, thresholds and the cost of failing fast.
• Backpressure and load shedding: refusing work early and cheaply.
• Idempotency: keys and stored results that make retries safe.
• Backend failure modes: how dependencies fail and how failures cascade.
• Hot keys, cache stampede and invalidation: the cache-side look-alike that often triggers a storm.
• Observability: logs, metrics and traces: the dashboards and traces this article relies on.

References and further reading

1. Metastable Failures in Distributed Systems (Bronson, Aghayev, Charapko and Zhu, HotOS 2021)sigops.orgDefines stable, vulnerable and metastable states, trigger versus sustaining effect, hidden capacity, and the worked retry example with a 300 QPS database.
2. Google SRE book, chapter 22: Addressing Cascading Failuressre.googleHow naive retries snowball, the 4^3 = 64 multi-layer example, retry budgets, deadline propagation and why recovery needs a large cut in load.
3. Google SRE book, chapter 21: Handling Overloadsre.googlePer-request (3 attempts) and per-client (10%) retry budgets, adaptive client-side throttling, “overloaded; don’t retry” errors and retrying only one layer above the failure.
4. Timeouts, retries, and backoff with jitter (Amazon Builders’ Library)aws.amazon.comRetries as “selfish”, the five-layer 243× amplification example, retrying at a single point in the stack, and the trade-offs of circuit breakers versus token buckets.
5. Using load shedding to avoid overload (Amazon Builders’ Library)aws.amazon.comGoodput versus throughput, overload as a positive feedback loop, dropping doomed work, and keeping fast rejections out of latency metrics.
6. Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, December 2021aws.amazon.comAWS’s own account of a surge of connection attempts and retries that congested internal network devices, and of a latent back-off bug.
7. AWS SDKs and Tools Reference Guide: Retry behaviordocs.aws.amazon.comDefault attempts, the full-jitter backoff formula and the retry quota token bucket (capacity and per-retry costs) in the AWS SDKs.
8. gRPC documentation: Retrygrpc.ioTransparent retries, per-method retry policies, and the retryThrottling token bucket that pauses retries when failures pile up.
9. Retry Storm antipattern (Azure Architecture Center)learn.microsoft.comClient and service symptoms of a retry storm, honouring Retry-After, not retrying 400s, and throttling at a gateway.
10. Envoy documentation: Circuit breaking and retry budgetsenvoyproxy.ioProxy-level limits on active retries and the recommendation to prefer retry budgets so retry volume cannot explode.