Your server saw 50 ms. The user waited 195.
A fresh HTTPS call pays for DNS and two handshakes before the request leaves. Reuse the connection and it’s 95 ms.
Will a fatter pipe help?
Bad: More bandwidth
At 1 Gbps a 2 KB reply takes 0.016 ms. Round trips still dominate.
Good: Fewer round trips
Reuse connections, move closer. Bandwidth raises the ceiling, not the speed.
79%
of requests are faster than the average
Rare stalls drag the mean to 59 ms while the median is 40 and the p99 is 340. Report p50 and p99 together.
One server’s 1% becomes 63% when a request waits on 100.
A request waiting on N calls is slow if any one is, so 1 − 0.99ᴺ. Ten calls give 9.6%; a hundred, 63%.
A food-delivery API serves 2,000 requests a second at a 150 ms mean. How many are in flight?
Little’s law
In flight = rate × time: 2,000/s × 0.15 s = 300. Size pools from numbers you already measure.
Past 70% busy, a little more load multiplies the wait.
From 50% to 95% busy buys 1.9× the throughput and costs 10× the response time. Plan for 50–70% at peak.
Keep the latency numbers honest
- Merge histograms; never average percentiles
- Track error latency apart from successes
- Watch p99 and saturation, not five-minute CPU
- Load tests: constant rate, timed from the intended send
Remember
Latency is time per request, quoted as p50 and p99; throughput is completions a second; bandwidth is the ceiling. Size pools with Little’s law, run 50–70% busy, and expect fan-out to compound the tail.