Two honest answers. Slow the sender, or turn some away.
Backpressure sends the problem upstream when the producer can wait. Load shedding rejects the excess, fast, when it can’t.
100×
an idle server’s response time, at 99% busy
With 50 ms of work: 250 ms at 80% busy, 500 ms at 90%, 5 s at 99%.
Spike coming. Bigger queue?
Bad: Unbounded queue
20,000 waiting by second 10, every reply too late, 72,000 timeouts.
Good: Bounded to 250 ms
1,000 queued at most. The excess gets an instant 503; accepted users never notice.
Backpressure is credit flowing upstream
Each stage grants credit, a TCP window, an HTTP/2 WINDOW_UPDATE, request(n) or a prefetch count. A full buffer stops granting, so the source waits.
Backpressure needs a sender that can wait. A user on a phone during a sale can’t.
The catch
Waiting only moves the queue upstream, filling thread and connection pools. At the edge, reject fast instead.
Shed load well
- Reject early, before parsing, auth or the database
- Shed by priority, so checkout beats add-to-cart
- Drop requests whose deadline has already passed
- Under overload, serve the newest request first
Retries close the loop: 4 × 4 × 4 = 64
If browser, frontend and backend each make 4 attempts, one action can become 64 database calls. Cap retries at 10% of requests.
Remember
Backpressure inside, where producers can wait; shedding at the edge, where users can’t. Bound every queue in seconds.