L4 reads the envelope. L7 opens it.
L4 sees only addresses and ports, so it decides once per connection, in microseconds. L7 ends TLS, reads every request and decides per request.
Does an L4 balancer spread your requests?
Bad: “It balances requests”
It never sees one. It picks a backend once, when the connection opens.
Good: It balances connections
One connection can carry one request or a million.
gRPC: one connection, 1,000 calls a second, one backend
L4 decides once per connection, so every call lands on one server. Two of four sit idle; servers added at 9 pm get nothing.
Three ways out of the gRPC trap
- An HTTP/2-aware L7 proxy balances every stream
- Clients learn every backend and pick one per call
- Cap connection age, so clients reconnect and land elsewhere
65 ms
mean latency, down from 163 ms
One of four backends stalls at 500 ms. Round robin still feeds it a quarter of requests; counting requests in flight lets its queue push work away.
Twenty proxies see a new backend at zero in flight. All twenty pile on.
The fix
Power of two choices: sample two at random, take the less loaded. The busiest 10% win only 1% of picks.
Most big systems run both
An L4 tier owns the public address and survives a balancer dying. Behind it, L7 proxies end TLS and route each request by path.
Remember
An L4 tier owns the address and survives a balancer dying. Behind it, L7 proxies balance every request, the less loaded of two random picks.