Latency vs throughput vs bandwidth

Latency is how long one operation takes, throughput is how many operations finish per second, and bandwidth is the most a link or resource could carry per second. They are related but not interchangeable: a fat pipe can still be slow, and a fast server can still drown. Performance questions in a design round are about how the three trade off, and about the slow tail of latency rather than its average.

Picture it first: an expressway and a toll plaza

Picture a four-lane expressway to a cricket stadium on match night, with a four-booth toll plaza just before the ground. Each booth takes about 6 seconds per car.

• Bandwidth is the most the road can carry: four booths at one car every 6 seconds is 4 × 600 = 2,400 cars an hour.
• Throughput is what actually gets through: if 2,000 cars an hour arrive, 2,000 leave, and never more than 2,400.
• Latency is one car’s trip: the driving, plus the wait in the queue at the plaza, plus the 6 seconds of paying.

On a quiet night the trip takes 90 minutes, nearly all of it driving. On match day it takes two hours on the same road: the booths are 2,000 ÷ 2,400 ≈ 83% busy and cars arrive in bunches, so the extra half hour is all queueing.

A fifth lane on an empty road at midnight shortens nobody’s trip: more bandwidth raises the ceiling, not the speed. And fans who reach the stadium after the first ball count as throughput but got nothing from the trip; counting only the trips that arrive in time gives you goodput.

Three words, three units

Latency is the time one operation takes, in milliseconds or microseconds. Every request has its own, so latency is a distribution, summarised with percentiles. Say where the stopwatch sits: a timer inside your handler never sees time spent in a socket backlog or on the network.

Keep two terms apart. Service time is the time a request is actually worked on: CPU, disk, calls to dependencies. Response time is what the caller waits: network both ways, plus queueing, plus service time. Everyday “latency” means response time.

Throughput is completed work per unit of time: requests per second (RPS), transactions or messages per second, MB/s. Count completions, not arrivals: if 12,000 requests arrive each second and 10,000 complete, throughput is 10,000 and a queue grows by 2,000 a second.

Bandwidth is capacity: the most a link or resource can move per second, quoted in bits. A 1 Gbps network card moves at most 10⁹ ÷ 8 = 125 MB a second. Throughput lives under that ceiling, minus headers, retransmissions and every narrower hop on the path. In design rounds it usually means egress: 10 million viewers of a cricket final at 4 Mbps each need 10⁷ × 4 × 10⁶ = 40 Tbps, which is why video goes through a CDN.

Goodput is the useful slice of throughput: in the Amazon Builders’ Library definition, requests handled without errors and fast enough for the client to use. An overloaded server can keep throughput high while goodput collapses, because it is finishing requests whose callers already gave up.

Utilisation is throughput ÷ capacity: 600 Mbps on a 1 Gbps link is 60%.

Where one request’s time goes

Response time is the sum of four delays:

• Propagation: distance ÷ signal speed. Light in fibre travels at about two-thirds of its vacuum speed, 300,000 ÷ 1.5 ≈ 200,000 km a second, so each 100 km adds about 1 ms of round trip. Mumbai to London is about 7,200 km in a straight line: at least 72 ms per round trip. Only moving closer fixes this.
• Transmission: size ÷ bandwidth. 1 MB at 100 Mbps takes 8 × 10⁶ ÷ 10⁸ = 80 ms; at 1 Gbps, 8 ms. This is the part bandwidth buys back.
• Queueing: waiting behind other work in a router buffer, accept queue, thread pool or lock.
• Service: the work itself.

Small messages are dominated by round trips, large transfers by bandwidth: a 2 KB response transmits in 0.016 ms at 1 Gbps but still pays every round trip. Figure 2 adds up one HTTPS call from a phone. A fresh connection pays for DNS, then one round trip for the TCP handshake and another for TLS 1.3 (RFC 8446), before the request even leaves.

That is why connection reuse, keep-alive and terminating TLS close to users often beat any code optimisation: the server dashboard says 50 ms, the user waited 195.

How latency and throughput pull against each other

For one worker handling one request at a time, throughput is 1 ÷ latency: 50 ms per request means 20 a second. Beyond that, the two come apart, and most techniques trade one for the other.

Parallelism raises throughput, not speed. Forty such workers handle 40 × 20 = 800 requests a second, and each request still takes 50 ms. Only splitting one request into parallel parts makes it faster, say ten shards each scanning a tenth of the data, and then you wait for the slowest shard (see the tail section). More in concurrency vs parallelism.

Batching buys throughput with latency. Say forcing a database log to disk takes 2 ms. One flush per commit caps the log at 1 ÷ 2 ms = 500 commits a second. Let commits that arrive during a flush wait and go out together in the next one: at 10,000 commits a second about 20 share each flush, so the same disk sustains 10,000 a second while each commit waits 2–4 ms instead of 2. Kafka producers, bulk inserts and TCP’s Nagle algorithm make the same trade.

Pipelining hides round trips. On a connection with a 0.5 ms round trip, send-and-wait caps you at 1 ÷ 0.5 ms = 2,000 commands a second, however fast the server. Send 50 before reading the replies and the same connection carries up to 100,000 a second, while each command still takes at least 0.5 ms. Redis pipelining and HTTP/2 streams use the same idea.

They also feed each other. With a fixed number of workers, throughput = workers ÷ latency, so halving latency doubles it (Little’s law, below). And pushing a resource towards 100% busy makes latency grow without limit (the hockey stick, below).

Percentiles: why the average lies

A percentile answers “how slow is it for the unluckiest x% of requests?” The p99 is the latency that 99% of requests beat. The p50 (median) describes the typical request, p95 and p99 the tail users notice, and p99.9 matters for back ends called many times per page.

The mean is dragged towards rare, slow outliers. In Figure 3 the median is 40 ms, the mean 59 ms, and 79% of requests are faster than the average; the p99 is 340 ms, 8× the median. Google’s SRE book makes the same point in Monitoring Distributed Systems: a service averaging 100 ms at 1,000 requests a second can easily have 1% of requests taking 5 seconds. Check it: 0.99 × 50 ms + 0.01 × 5,000 ms ≈ 100 ms, a latency almost no request had. And a user whose session makes 100 such requests has a 1 − 0.99¹⁰⁰ ≈ 63% chance of meeting at least one p99.

Rules that keep percentiles honest:

• Report p50 and p99 together, plus p99.9 for high-volume back ends.
• Never average percentiles across servers or time windows; the result is not a percentile of anything. Record histograms (counts per latency bucket) and merge those. The SRE book suggests bucket boundaries growing by about 3× (10, 30, 100 ms…); HdrHistogram keeps a fixed relative precision from microseconds up to an hour.
• Track error latency separately. Fast failures flatter the median: the Builders’ Library notes that a service shedding 60% of its traffic can show an excellent median while successful requests are slow.
• Keep the mean for capacity. As Marc Brooker points out, Little’s law and efficiency work run on means. Use means to size, percentiles to judge experience.

Going deeper: the tail at scale

One slow server is a nuisance; a request that waits for many servers makes it the norm. If each call is slow with probability p, a request that waits for N independent calls is slow whenever any one of them is:

P(at least one slow) = 1 − (1 − p)^N

With every call at its p99 (p = 0.01): 1 call → 1%; 10 calls → 1 − 0.99¹⁰ = 9.6%; 100 calls → 1 − 0.99¹⁰⁰ = 63%. Dean and Barroso’s The Tail at Scale uses exactly this case: servers that usually answer in 10 ms but take 1 second at p99, behind a 100-way fan-out, make 63% of user requests take over a second. Even 1-in-10,000 slowness, spread over 2,000 servers, slows 1 − 0.9999²⁰⁰⁰ ≈ 18% of requests.

Google’s measurements of a real service show the same shape: the p99 for one random leaf server to answer was 10 ms, for all leaves 140 ms, and for 95% of the leaves 70 ms. The slowest 5% of servers caused half the tail.

Variability comes from shared machines, background work such as garbage collection and compaction, power throttling and queues at every layer. You can’t remove it all, so large services tolerate it:

• Hedged requests. Send to one replica; if no reply arrives within the p95 latency, send a copy to another and take the first answer. Waiting for the p95 keeps extra load near 5%. In Google’s benchmark reading 1,000 keys from 100 Bigtable servers, hedging after 10 ms cut the p99.9 from 1,800 ms to 74 ms for 2% more requests.
• Tied requests. Queue the request on two servers, each told about the other; whichever starts it first cancels its twin. With the second copy sent 1 ms later, median latency fell 16% and p99.9 nearly 40%, for under 1% extra disk use.

Caching alone does not fix the tail, the paper notes, unless the whole working set fits in the cache: the misses are your tail (see caching 101).

Little’s law: in flight = rate × time

Little’s law links three quantities in any stable system:

L = λ × W

L is the average number of items inside, λ the average arrival rate, and W the average time each item spends inside. It holds for a thread pool, a queue, a database, a whole region or the expressway: 2,000 cars an hour × 2 hours per trip means about 4,000 cars on the road at any moment. It needs only a stable system (as much leaves as arrives, on average) and long-run averages. As Brooker puts it in Telling Stories About Little’s Law, concurrency is what consumes threads, connections and memory, yet it is hard to observe; rate and latency are easy to measure.

Sizing threads. A food-delivery order API serves 2,000 requests a second at peak with a 150 ms mean latency: 2,000 × 0.15 = 300 in flight, or 30 per instance across 10 instances. To keep each pool under 60% occupied, give each instance about 30 ÷ 0.6 = 50 workers (or the async equivalent).

Sizing a connection pool. Each request holds a database connection for two 4 ms queries: 2,000 × 0.008 = 16 connections busy fleet-wide. Pools of 5 per instance (50 in total) leave plenty of headroom; 100 per instance would hand the database 1,000 mostly idle connections. If queries slow to 40 ms you need 2,000 × 0.08 = 160, three times the pool: requests queue for connections, latency climbs further, and the pool is exhausted (see connection pooling).

Capping throughput. λ = L ÷ W: a payment-status poller with 64 workers and 200 ms bank calls finishes at most 64 ÷ 0.2 = 320 checks a second, and only 80 if the bank slows to 800 ms. A fixed concurrency limit turns a slow dependency into lost throughput.

Queue wait. W = L ÷ λ: a steady backlog of 30,000 messages drained at 1,000 a second means a wait of about 30 seconds, so measure lag in seconds.

The law is exact for averages, but real systems feed back on themselves: latency rises with concurrency through contention, and arrivals rise with latency when clients time out and retry. Brooker’s point is that this produces cliffs, not slopes. Backpressure and load shedding exist to break that loop.

Utilisation and the hockey stick

Utilisation ρ (rho) is arrival rate ÷ service rate: the share of time a server is busy. The simplest queueing model, M/M/1 (random arrivals, random service times, one server), gives a mean wait in the queue of:

wait = ρ ÷ (1 − ρ) × service time

and a mean response time of service time ÷ (1 − ρ). Apply Little’s law and the average number of requests in the system is ρ ÷ (1 − ρ), the curve Brooker plots in Latency Sneaks Up On You. With a 10 ms service time:

Going from 50% to 95% busy buys 1.9× the throughput and costs 10× the response time. That is why you plan servers for 50–70% utilisation at peak, not 90%:

• The knee. Below about 70% delay grows gently; above 80% each extra percent costs more than the last.
• Bursts hide inside averages. Brendan Gregg describes a customer whose monitoring never showed CPU above 80% while latency suffered: the tool averaged over five minutes, and the CPU hit 100% for seconds at a time.
• Failures remove capacity. Three zones at 60% become two zones at 90% when one zone fails.
• Reality is worse. The table shows averages for random arrivals; p99 waits blow up sooner, and cron jobs, sale openings and retry storms are burstier.

Efficiency wins also evaporate: Brooker notes that speeding a service up and then shrinking the fleet brings utilisation, and the p99, right back. High percentiles are a poor efficiency metric but a good early warning of overload; the SRE book suggests watching p99 over windows as short as a minute as an early sign of saturation.

The bandwidth-delay product

To keep a link busy, a sender must have enough data travelling to cover one round trip. That amount is the bandwidth-delay product (BDP):

BDP = bandwidth × round-trip time

A 1 Gbps path with a 100 ms round trip needs 10⁹ × 0.1 = 10⁸ bits = 12.5 MB in flight. It is Little’s law again, counted in bytes: in flight = rate × time.

TCP allows one window of unacknowledged data in flight, so a single connection’s throughput is at most window ÷ RTT. The TCP header’s window field is 16 bits, so without extensions the window tops out at 64 KiB: 65,535 bytes ÷ 0.1 s ≈ 655 KB/s ≈ 5.2 Mbps on that 1 Gbps path, about 0.5% of it. RFC 7323 lifts the limit with the Window Scale option (windows up to 1 GiB) and calls large-BDP paths “long, fat networks”, where packet loss is especially costly.

In practice the cap is your socket buffers, which operating systems auto-tune up to a configured maximum (on Linux, net.ipv4.tcp_rmem and net.ipv4.tcp_wmem). A buffer held at 4 MB on a 100 ms path caps one connection at 4 MB ÷ 0.1 s = 40 MB/s ≈ 320 Mbps, a third of the link. That is why a cross-region backup from Mumbai to Europe crawls on one stream while the same copy inside a data centre flies. Fix it with larger buffers, parallel streams, or by moving the data closer; windows and congestion control are covered in TCP vs UDP.

Going deeper: measuring latency honestly

Coordinated omission

Most load generators send a request, wait for the reply, then send the next. When the server stalls, the tester stalls with it, so the requests that would have been sent during the stall, all of which would have waited, are never recorded. This is coordinated omission: as the wrk2 README puts it, the load generator coordinates with the server to avoid measuring during high-latency periods.

The HdrHistogram documentation gives a clean example. A tester samples every 10 ms. For 100 seconds every response takes 1 ms: 10,000 samples. Then the server freezes for 100 seconds and the tester records one 100-second sample. The raw data says about 99.99% of requests took 1 ms or less, yet a user arriving at a random moment in those 200 seconds had only about a 50% chance of a 1 ms response.

The fixes:

• Generate load at a constant rate (open loop), independent of replies.
• Measure from the intended send time, not the actual one (or let HdrHistogram correct for a known interval). wrk2 does both; in its README, a 1.4 s pause injected into a run made the corrected p99 about 200× the uncorrected one.

What to watch in production

• The four golden signals (Google SRE), per service: latency (successes and errors tracked separately), traffic, errors (including “too slow counts as failed” by policy) and saturation of the most constrained resource.
• USE (Brendan Gregg), per resource: utilisation (time busy), saturation (queued work) and errors; for a network card, utilisation is throughput ÷ bandwidth.

The golden signals tell you that users are hurting; USE tells you which resource is the bottleneck. Both feed the SLIs in reliability: SLIs, SLOs and error budgets and the tooling in observability.

Stating latency and throughput requirements

“It should be fast” is not a requirement. A useful one names the operation, the percentile, the threshold, the load it must hold at, where it is measured, and the window. For example: p99 of successful search requests under 200 ms, measured at the load balancer over 5-minute windows, at up to 20,000 requests a second at peak. Without the load, any service meets its target at 10 requests a second; without the percentile, the mean hides the tail; without “successful”, fast errors flatter the number.

Different features want different shapes (illustrative targets):

Then turn the numbers into design. At the worst allowed latency, 20,000 × 0.2 s = 4,000 requests in flight, so pools and connection limits must allow it. A 200 ms budget leaves room for only a handful of serial 40 ms cross-region round trips. And if search fans out to 50 shards, the user-level p99 holds only if each shard is slow less than 1 − 0.99^(1/50) ≈ 0.02% of the time: each shard must meet 200 ms at about p99.98.

Cheat sheet

How to say it in the interview

“Latency is time per request, which I’ll quote as percentiles; throughput is completed requests per second; bandwidth is the ceiling on a link. For this API I’d target p99 under 200 ms at 20,000 RPS peak. At a typical 100 ms, Little’s law gives 20,000 × 0.1 = 2,000 requests in flight, so I’ll size pools for that and run servers at 50 to 70% busy, because queueing explodes past about 80%. Search fans out to 20 shards, so the tail compounds: 1 − 0.99²⁰ means about 18% of requests would hit some shard’s p99. I’d set shard targets at p99.9 and hedge reads after the p95.”

Why not report average latency? The mean describes almost nobody: a few slow requests drag it up while most are far faster, and the tail stays invisible. Report p50 and p99 (p99.9 for back ends) from merged histograms; keep the mean for capacity maths.

Can you increase throughput without hurting latency? Yes, below saturation: add parallel capacity, cut per-request work or remove a bottleneck, and each request still takes the same time. What you can’t do is run the same servers closer to 100% busy without queueing; batching and pipelining also make individual items wait.

Your p99 doubled but CPU is at 60%. Why? Average CPU is the wrong lens. Candidates: bursts that hit 100% inside the averaging window; another saturated resource (connection pool, disk, a lock, one hot partition or core); a slower dependency, which by Little’s law raises in-flight work until a pool queues; garbage-collection pauses; or a fan-out that now touches more shards. I’d check each resource with USE and break the latency down with traces.

How many workers do you need for 5,000 RPS at 80 ms? Little’s law: 5,000 × 0.08 = 400 in flight on average. At a 60–70% occupancy target that is about 600 workers, say 20 instances × 30, plus spare for losing a zone. Then stress it: if a dependency slows to 400 ms, demand jumps to 2,000 in flight.

What is coordinated omission? A bug in closed-loop testers: each connection waits for a reply before sending again, so during a stall the tester records one slow sample instead of the hundreds of requests that would have waited, and percentiles come out far too good. Fix it with constant-rate load measured from the intended send time (as wrk2 does) or HdrHistogram’s correction.

Where to go next

• Numbers to know: the latency ladder and per-component throughput figures.
• Back-of-envelope estimation: turning user counts into RPS, bandwidth and servers.
• Scalability: when one machine’s throughput is not enough, including Amdahl’s law and the USL.
• Reliability: SLIs, SLOs and error budgets: turning percentiles into objectives.
• Timeouts, retries and backoff with jitter: timeouts from percentiles, without retry storms.
• Backpressure and load shedding: protecting goodput when arrivals exceed capacity.
• Connection pooling: Little’s law applied to database connections.
• Observability: logs, metrics and traces: histograms, tracing and the golden signals in practice.
• Content delivery network: beating propagation delay by moving bytes closer to users.

References and further reading

1. The Tail at Scale (Dean and Barroso, Communications of the ACM, 2013)cacm.acm.orgFan-out amplification (100 servers at a 1-second p99 → 63% of requests slow), Google’s measured leaf-versus-root percentiles, sources of variability, and hedged and tied requests with their measured gains.
2. Monitoring Distributed Systems (Google SRE book, chapter 6)sre.googleThe four golden signals, separating error latency, why the mean misleads (100 ms average with 1% at 5 s), exponential histogram buckets, and p99 over short windows as an early saturation signal.
3. The USE Method (Brendan Gregg)brendangregg.comUtilisation, saturation and errors for every resource; network utilisation as throughput ÷ bandwidth; the five-minute average that hid 100% CPU bursts.
4. Latency Sneaks Up On You (Marc Brooker, 2021)brooker.co.zaThe M/M/1 curve ρ ÷ (1 − ρ), why efficiency gains vanish when fleets shrink, and high-percentile latency as a leading indicator of overload.
5. Telling Stories About Little’s Law (Marc Brooker, 2018)brooker.co.zaL = λW as a capacity tool for threads and connections, its long-run-average assumptions, and the feedback loops (contention, timeouts and retries) that create cliffs.
6. Using load shedding to avoid overload (Amazon Builders’ Library)aws.amazon.comGoodput versus throughput, throughput that degrades under overload, and fast-failing shed requests that flatter the median.
7. HdrHistogram README (Gil Tene)github.comRecording latency with fixed relative precision, and the 100-second-pause example of coordinated omission and its correction.
8. wrk2: a constant-throughput HTTP benchmarking tool (Gil Tene)github.comWhy closed-loop load generators exhibit coordinated omission, measuring from the intended send time, and the 1.4 s pause that made the corrected p99 200× larger.
9. RFC 7323 — TCP Extensions for High Performancerfc-editor.orgLong, fat networks, the 64 KiB limit of the 16-bit window field, and the Window Scale option allowing windows up to 1 GiB.
10. RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3rfc-editor.orgThe one-round-trip full handshake used in the request anatomy figure.