Timeouts, retries and backoff with jitter

A timeout caps how long a caller waits for a remote call. A retry sends the call again when it fails. Backoff with jitter spaces those retries out, randomly, so that thousands of clients do not hit a struggling service at the same instant. With limits on all three, brief partial failures become invisible blips; without them, a slow dependency becomes an outage that outlives its cause.

Picture it first: paying at a crowded shop

It is Diwali evening at a busy sweet shop. You scan the QR code, press Pay, and the app spins. Forty other customers are doing the same, because the payment provider has just had a hiccup.

How long do you stare at the spinner before deciding something is wrong? That is a timeout. Do you press Pay again? That is a retry, and it is safe only if pressing twice cannot charge you twice: the money may already have left, and the spinner only tells you the answer has not arrived. A good app checks the status of the same payment instead of starting a new one. That is idempotency.

If all forty customers press Pay again in the same second, the provider chokes again. Waiting longer after each failure is backoff; each person waiting a slightly different, random time is jitter. And your train leaves in ten minutes, after which even a successful payment is useless to you. That is a deadline.

Map it back: you are a client, the provider is a dependency, the spinner is an in-flight request holding a thread and a connection, and the forty customers are your servers, all seeing the same error at once.

Why every remote call needs a timeout

Many clients wait forever unless told otherwise: Python’s Requests library never times out unless you pass a value, and gRPC sets no deadline by default. AWS’s rule is to put a timeout on every remote call, even between processes on the same machine.

Waiting is not free. Each in-flight call holds a thread, a pooled connection and memory, and Little’s law prices it: calls in flight = arrival rate × time per call. Take a service that handles 1,000 requests per second with 200 worker threads and 50 ms per request: 1,000 × 0.05 s = 50 threads are busy. Now a dependency starts hanging on 2% of calls:

• 30-second timeout: the stuck calls hold 20/s × 30 s = 600 threads. The pool has 200, so soon every request fails, not 2%.
• 300 ms timeout: they hold 20/s × 0.3 s = 6 threads, 49 + 6 = 55 in total, and 98% of users notice nothing.

The Google SRE book shows the same effect with a 100-second deadline: 5% of requests that never complete exhaust the frontend’s threads, and about 80% of all requests fail. A timeout keeps a partial failure partial, and turns “slow forever” into an error you can count, alert on and handle.

Connect, read and overall timeouts

“The timeout” is really several timers:

Keep the connect timeout much shorter than the per-attempt one. Opening a connection needs no work from the server’s application, so a dead host should fail in milliseconds. And because many read timeouts measure the gap between bytes rather than the total (the Requests documentation says so explicitly), you need an overall timer on top.

Setup work also hides inside first calls. AWS describes a call with a timeout of about 20 ms that failed only just after deployments: the timer included opening a new TLS connection, which took longer than 20 ms, while later calls reused it. The lasting fix was to open connections at startup, before taking traffic.

Picking the number

A timeout that is too long protects nothing. One that is too short cuts off calls that would have succeeded, and those calls are retried, adding load. AWS starts from data: choose an acceptable false-timeout rate, such as 0.1%, and use the matching percentile of the dependency’s latency, here the p99.9.

Say you call an inventory service 2,000 times a second, with p50 = 8 ms, p99 = 40 ms and p99.9 = 90 ms.

• 90 ms (p99.9): 2,000 × 0.001 = 2 false timeouts a second. Negligible.
• 40 ms (p99): 20 a second on a good day. But if a slow deploy pushes 10% of calls past 40 ms, 2,000 × 0.10 = 200 calls a second time out and retry, against a service that is already slower. This is how, as AWS warns, a small latency increase grows into an outage.

Watch three blind spots. Distant callers: server-side percentiles leave out the network, so add a realistic worst-case delay for clients across the internet. Tight distributions: if p50 is 5 ms and p99.9 is 7 ms, a 2 ms hiccup times out many calls, so pad generously. Mixed operations: a key lookup and a bulk export need different timeouts. And as the SRE book notes, a deadline orders of magnitude above the mean latency is usually a mistake; in its 80% example it was 1,000 times the normal latency.

Deadlines across a call chain

A timeout is local to one call; a deadline belongs to the whole request. Per-hop timeouts add up: if the gateway, checkout and payments each wait 1 s and the bank adapter 2 s, a request meant to take one second can keep machines busy for five.

So set one budget at the edge and hand the remainder down. gRPC builds this in. A deadline (a point in time) and a timeout (a duration) convert into each other, and a server that calls another server can carry the incoming deadline onward: by default in Java and Go, as an opt-in in C++. On the wire it sends the remaining time, not a timestamp, so clock skew between machines does not matter. Over HTTP, do the same with a header that carries the milliseconds left.

Three habits make it work:

• Check the budget before each expensive stage, as the SRE book advises. With 180 ms left and a call that needs 250 ms, fail now; the dashed outline in the figure is the work that would not have fitted.
• Keep a margin for the network and your own work on the response.
• Shrink, never extend the deadline you received.

In Go, the incoming context carries the caller’s deadline, and a derived context keeps whichever deadline is earlier:

func (s *Checkout) Pay(ctx context.Context, req *pb.PayRequest) (*pb.Receipt, error) {
	// Not enough budget left for a typical bank call? Say so now, cheaply.
	if dl, ok := ctx.Deadline(); ok && time.Until(dl) < 250*time.Millisecond {
		return nil, status.Error(codes.DeadlineExceeded, "budget too small to charge")
	}
	// Tighten for this hop if useful; the earlier of the two deadlines wins.
	ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
	defer cancel() // also cancels the bank call if we return early
	return s.bank.Charge(ctx, req)
}

What happens after the caller gives up

When a client times out it stops waiting, but nothing tells the server to stop. Whatever the server finishes is wasted work. Under overload this compounds: the queue fills with requests whose callers have gone, the server spends its scarce capacity on them, and fresh requests time out behind them.

So honour cancellation: a gRPC server cancels the call once the client’s deadline passes, but your handler must check the context and stop the queries and downstream calls it started. Drop expired requests when they come off a queue. And when one branch of a fan-out dooms the whole request, cancel its siblings.

Which failures to retry

Retry transient failures: a dropped connection, a restarting replica, a moment of overload. Do not retry permanent ones, such as a malformed request or a missing permission; they fail the same way every time.

The AWS SDKs draw the same line, with a 50 ms base delay for transient errors and 1 s for throttling. Eventual consistency blurs it a little: a 404 for something created a moment ago may be a 200 a second later.

Servers can help. RFC 9110’s Retry-After header gives a delay in seconds or an HTTP date; with a 503 it says how long the service expects to be unavailable, and 429 responses may carry it too. gRPC’s equivalent is server pushback through the grpc-retry-pushback-ms key, where a negative value means “do not retry”. Treat the server’s wait as a minimum.

Idempotency makes retries safe

RFC 9110 calls a method idempotent if sending it several times has the same intended effect as sending it once. GET, HEAD, PUT and DELETE are idempotent; POST and PATCH are not. So a client may repeat a PUT after a dropped connection, but should not automatically retry a POST unless it knows the repeat is safe, and a proxy must never do so.

To make a write safe to retry, give it an idempotency key: one unique ID per logical operation, created before the first attempt and sent with every attempt. The server stores the key with the result and answers repeats from it. EC2’s RunInstances API takes a client token for exactly this. In a UPI-style flow, a retry with the same payment reference means “finish or report payment X”, never “pay again”. Do not mint a new key per attempt, and insert keys atomically so two concurrent attempts cannot both look new. The idempotency article covers storage and expiry.

Retry amplification

Each layer that retries multiplies the attempts of every layer below it. In AWS’s example, five layers each make up to three attempts, so when the database starts failing under load, one user action becomes 3, 9, 27, 81 and finally 3^5 = 243 attempts at the database. The SRE book’s version, three layers of four attempts, gives 4^3 = 64. Either way, the multiplier peaks exactly when the bottom layer is weakest.

So retry at one layer only, normally the one directly above the failing dependency, as Google’s SRE book puts it. When that layer gives up, it returns an explicit “overloaded; don’t retry” error or a degraded answer, so the layers above fail fast. AWS adds that retrying at the top of the stack wastes the work already done below; its practice for cheap operations is to retry at a single point in the stack.

Retries also hide in libraries: HTTP clients, cloud SDKs (the AWS SDKs make three attempts by default), RPC frameworks (gRPC transparently retries calls that never reached the server’s application code), sidecars and load balancers. List every layer that can retry, then switch off all but one.

Backoff: wait longer each time

Retrying instantly under overload brings the client back while the cause is still there. Exponential backoff doubles the wait after each failure:

sleep = min(cap, base × 2ⁿ) for retry n = 0, 1, 2, …

With a 100 ms base and a 1.6 s cap, that is 100, 200, 400, 800, 1,600, 1,600 ms. Without the cap, the eleventh retry would wait 100 ms × 2^10 ≈ 102 s. Two limits matter more than the cap: a maximum of about three attempts in total (the AWS SDK default and the SRE book’s per-request limit), and the deadline: never start a sleep that ends after the caller has given up.

Why backoff alone still synchronises clients

Backoff spreads one client’s retries over time; it does not spread different clients apart. If 1,000 clients fail together, during a deploy, a network blip or a database failover, they compute the same sleeps and return in waves. Each wave swamps the server, and idle gaps sit in between. In AWS’s simulation, un-jittered exponential backoff barely reduced total work, because the calls stayed clustered.

Jitter: full, equal and decorrelated

Jitter randomises each sleep so that clients which failed together retry apart. Writing v = min(cap, base × 2ⁿ), AWS’s 2015 analysis compared three versions:

• Full jitter: sleep = random(0, v).
• Equal jitter: sleep = v/2 + random(0, v/2), so no retry comes back too early.
• Decorrelated jitter: sleep = min(cap, random(base, previous_sleep × 3)), starting from previous_sleep = base.

In AWS’s simulation every jittered version did far less work than plain backoff. Full and equal jitter made about the same number of calls and decorrelated somewhat more; equal jitter was the slowest to finish; full jitter did the least work but finished slightly after decorrelated. The waves chart above agrees: full jitter finished in about 1.1 s with roughly 3,000 calls, against 5.3 s and 3,850 calls without jitter. gRPC is gentler: it multiplies each backoff by random(0.8, 1.2), which breaks exact lockstep but leaves clients within a ±20% band.

Default to full jitter. The AWS SDKs use it: random(0, 1) × min(20 s, base × 2^retry), with a 50 ms base for transient errors and 1 s for throttling.

Retry budgets: capping the total

Backoff and jitter decide when retries happen, not how many. In a full outage every first attempt fails, so three attempts per request means 3× the load on a dependency that is already down. A retry budget caps retries as a share of real traffic:

• Google: at most three attempts per request, and retries only while they are under 10% of a client’s requests. In the SRE book’s worst case, that cuts the load from just under 3× to about 1.1×.
• gRPC retry throttling: per server, a token count starts at maxTokens. Each failure removes one token, each success adds tokenRatio, and retries and hedges pause while the count is at or below half of maxTokens. With 10 and 0.1, five straight failures pause retries, and they stay off while failures exceed about one per ten successes. Only retryable failures count.
• AWS SDKs: a 500-token quota per client. Each retry spends tokens (in the updated behaviour AWS documents for 2026, opt-in when this was written: 14 after a transient error, 5 after throttling), a successful retry refunds its cost, and a first-try success adds one. In a sustained outage the bucket empties and the SDK fails fast. Adaptive mode adds a client-side rate limiter that can delay first attempts too: right for a batch job hammering one table, wrong as a default, since throttling on one resource then slows every call from that client.

The whole policy in Python:

import random, time

BASE, CAP, MAX_ATTEMPTS = 0.05, 2.0, 3      # seconds, seconds, total tries
RETRYABLE = {429, 502, 503, 504}

class RetryBudget:
    """Retries are allowed only while retryable failures stay rare."""
    def __init__(self, max_tokens=10.0, ratio=0.1):
        self.max, self.ratio, self.tokens = max_tokens, ratio, max_tokens
    def record(self, ok):
        step = self.ratio if ok else -1.0
        self.tokens = min(self.max, max(0.0, self.tokens + step))
    def can_retry(self):
        return self.tokens > self.max / 2

def call(send, req, deadline, budget):
    # req.idempotency_key was created once, before the first attempt.
    for attempt in range(MAX_ATTEMPTS):
        left = deadline - time.monotonic()
        if left <= 0:
            raise TimeoutError("deadline exceeded before attempt")
        resp = send(req, timeout=min(left, 0.3))         # per-attempt timeout
        budget.record(resp.status not in RETRYABLE)
        if resp.status not in RETRYABLE:
            return resp
        if attempt == MAX_ATTEMPTS - 1 or not budget.can_retry():
            return resp                                  # give up, surface it
        sleep = random.uniform(0, min(CAP, BASE * 2 ** attempt))  # full jitter
        sleep = max(sleep, resp.retry_after or 0)        # honour Retry-After
        if time.monotonic() + sleep >= deadline:
            return resp                                  # no time for another try
        time.sleep(sleep)

Connection errors and timeouts take the same path, for idempotent requests only. Share one budget across every call from the process to that dependency; a budget per request limits nothing.

Hedged requests: retrying before a failure

Retries deal with failures; hedged requests deal with slowness. In “The Tail at Scale”, Jeff Dean and Luiz André Barroso take a server that usually answers in 10 ms but has a p99 of one second. A request that needs answers from 100 such servers is slow with probability 1 − 0.99^100 ≈ 0.63. With enough fan-out, the tail becomes the common case.

A hedged request goes to one replica. If no answer comes within a short delay, a copy goes to a second replica; the client takes the first answer and cancels the other. With the delay set at the p95, only the slowest 5% of calls send a copy, so the extra load stays near 5%. In a Google benchmark reading 1,000 keys spread over 100 BigTable servers, hedging after 10 ms cut the p99.9 latency from 1,800 ms to 74 ms for just 2% more requests. It works because delays usually come from interference on one server, such as a garbage-collection pause, which a second replica rarely shares.

Tied requests go a step further: the request is queued at two servers, each told about the other, and whichever starts first cancels its twin. gRPC supports hedging per method, as an alternative to a retry policy: up to maxAttempts copies (capped at 5) sent hedgingDelay apart. The first success wins, the deadline covers every copy, and retry throttling applies.

The price: extra load by design; idempotent calls only, because both replicas may execute the request; no help when the whole cluster is slow from overload, where hedges only add load; and cancellation is required, or the losing copy’s work is wasted.

Retries, circuit breakers and retry storms

Retries assume failures are brief and independent. When a dependency is down or overloaded for minutes, retries become part of the problem. In a retry storm, retry traffic keeps a service overloaded after the original trigger has gone, a self-sustaining state often called a metastable failure; the SRE book warns that escaping can mean cutting load drastically until the retries stop. Retry storm covers symptoms, detection and recovery. The defences on this page are what keep you out of one.

A circuit breaker goes further: after enough failures it stops all calls to the dependency, first attempts included, then lets a few probes through. A budget limits extra attempts; a breaker limits every attempt. AWS notes that breakers add a mode that is hard to test and can slow recovery, and prefers local token-bucket retry limits. Many teams use both: a budget for everyday blips, a breaker for a dependency that is clearly down.

Going deeper: edge cases

Timeouts must nest

Suppose the gateway gives checkout 1 s, and checkout tries payments three times with a 500 ms per-attempt timeout and 100 ms and 200 ms of backoff. The worst case is 3 × 500 + 100 + 200 = 1,800 ms. The gateway gives up during the second attempt, so the third can only be wasted work. Make attempts × per-attempt timeout + total backoff fit the caller’s budget, or make the loop deadline-aware, as in the Python example.

Let servers say “don’t retry”

In the SRE book, clients put an attempt counter in each request’s metadata. A backend that sees many retries arriving infers that other tasks are overloaded too, and answers “overloaded; don’t retry” instead of a plain “busy”. Retry-After, gRPC pushback and AWS’s x-amz-retry-after header carry the same idea; the AWS SDKs clamp that last one between the computed backoff and the backoff plus 5 s, and add no jitter of their own. In your own metrics, count attempts as well as calls: a rising attempts-per-call ratio is the earliest warning sign.

Cheat sheet

How to say it in the interview

“Every network call gets a timeout, set from the dependency’s p99.9 plus padding, so roughly one call in a thousand that would have succeeded is cut. At the edge I set an end-to-end deadline, say one second, and pass the remaining budget down, so a service that receives a request with 50 ms left fails fast. I retry only transient failures, like 503s and connection resets, and only idempotent calls; writes carry an idempotency key. Only the layer directly above the failing dependency retries, because three tries at five layers is 243 calls at the database. Retries use capped exponential backoff with full jitter inside a shared 10% retry budget, so an outage adds a tenth more load instead of tripling it.”

How do you pick the timeout value? From data. Choose an acceptable false-timeout rate, say 0.1%, take the dependency’s matching percentile (p99.9) and add padding, more for tight distributions and distant callers. Set it per operation, and check that all retries fit inside the caller’s deadline.

Why add jitter if you already back off? Clients that failed together compute the same backoff and retry together, in waves that overload the server again. Full jitter, a random sleep between zero and the backoff, spreads them out. In AWS’s simulation every jittered variant did far less work than plain backoff.

What if the operation isn’t idempotent? Then a timeout is ambiguous: the write may have happened. Make it idempotent with a client-generated key stored with the result, or do not retry automatically; query the operation’s status, or surface the error for a reconciliation job.

Retries or a circuit breaker? Both, for different failures. A retry budget absorbs brief, scattered errors. A breaker stops all traffic to a dependency that is clearly down, so callers fail fast and the dependency has room to recover.

When would you hedge instead of retry? When the problem is tail latency, not errors, especially on fan-out reads. Send a copy after the p95, take the first answer and cancel the other: about 5% more load for a much shorter tail. Never hedge non-idempotent writes.

Where to go next

• Retry storm: when retries feed an outage, and how to detect and stop one.
• Circuit breaker: failing fast when a dependency is clearly down.
• Idempotency: keys, their storage and expiry.
• Backpressure and load shedding: rejecting work early and cheaply, which keeps retries safe.
• Rate limiting: where 429 and Retry-After come from.
• Backend failure modes: the slow and partial failures that timeouts catch.
• Bulkheads: separate pools so one slow dependency cannot take everything down.
• Reliability: SLIs, SLOs and error budgets: where the end-to-end latency budget comes from.

References and further reading

1. Timeouts, retries, and backoff with jitter (Amazon Builders’ Library)aws.amazon.comA timeout on every remote call, the p99.9 false-timeout method and its pitfalls, the 3^5 = 243 amplification example, token-bucket retry limits and jitter for scheduled work.
2. Exponential Backoff and Jitter (Marc Brooker, AWS Architecture Blog, 2015)aws.amazon.comCapped exponential backoff, the full, equal and decorrelated jitter formulas, and a simulation comparing their total calls and completion time.
3. Google SRE book, chapter 22: Addressing Cascading Failuressre.googleRandomised backoff, per-request and server-wide retry limits, the 4^3 = 64 example, deadline propagation, wasted work after a deadline, and the thread-exhaustion arithmetic.
4. Google SRE book, chapter 21: Handling Overloadsre.googlePer-request (three attempts) and per-client (10%) retry budgets, the 3x to 1.1x worst case, and retrying only at the layer directly above the one that rejects.
5. gRPC documentation: Deadlinesgrpc.ioNo deadline by default, deadline versus timeout, server-side cancellation, and propagation that sends remaining time to avoid clock skew.
6. gRFC A6: gRPC client retriesgithub.comThe retry and hedging policies, the plus-or-minus 20% backoff jitter formula, the cap of five attempts, retry throttling tokens and server pushback.
7. AWS SDKs and Tools Reference Guide: Retry behaviordocs.aws.amazon.comStandard and adaptive retry modes, three default attempts, the full-jitter formula with 50 ms and 1 s bases and a 20 s cap, and the 500-token retry quota.
8. RFC 9110: HTTP Semanticsrfc-editor.orgWhich methods are idempotent and why that permits automatic retries, the Retry-After field, and 503 Service Unavailable.
9. The Tail at Scale (Dean and Barroso, Communications of the ACM, 2013)barroso.orgWhy fan-out magnifies tail latency, hedged requests sent after the p95 for about 5% extra load, the 1,800 ms to 74 ms benchmark, and tied requests.
10. Requests documentation: Timeoutsrequests.readthedocs.ioNo timeout unless you set one, separate connect and read timeouts, and a read timeout that measures the gap between bytes rather than the whole response.