Load balancing: L4 vs L7

A load balancer spreads incoming traffic across a pool of interchangeable servers and stops sending it to the ones that fail. A layer 4 (L4) balancer decides once per connection, using only IP addresses and ports. A layer 7 (L7) balancer ends the connection, reads each HTTP request and chooses a server per request. Most large systems run both: a fast L4 tier at the edge with L7 proxies behind it.

Picture it first: a sorting office in festival week

A city’s main post office in Diwali week gets parcels by the truckload. At the gate, staff read only the label on each sack and wave it to one of ten sorting tables. They never open anything, so they’re fast, and every sack from one truck goes to one table.

Inside, a supervisor opens each sack and reads every letter. Refund claims go to the refunds desk, new connections to another, and within each desk to whichever clerk has the smallest pile. She is slower and must be trusted with the contents, but she can split one giant sack across five clerks and hand a letter to someone else when a clerk goes home sick.

Map it back:

• The gate is L4. The label is the IP address and port; the sealed envelope is TLS; keeping a truck at one table is keeping a TCP connection on one server.
• The supervisor is L7. Opening the sack is terminating TLS; reading each letter is parsing HTTP; the giant sack is one HTTP/2 connection carrying thousands of requests.
• A clerk who keeps one customer’s file on her own desk is a sticky session. It works until she takes leave.

What a load balancer is for, and where it sits

Three jobs justify the extra hop:

• Scale out. Ten modest servers behind one address replace one huge one (see vertical vs horizontal scaling).
• Availability. The balancer health-checks every server and routes around dead ones; clients never notice.
• Safe change. Take servers out one at a time for a rolling deploy, or send 5% of traffic to a canary.

In a typical internet-facing stack, DNS or anycast picks a region or site. An L4 tier owns the public virtual IP (VIP) and spreads connections across an L7 tier of reverse proxies, which terminate TLS and route each request to a service pool. An API gateway often lives in that L7 tier.

Inside the data centre, east-west calls go through an internal balancer or, increasingly, a client library or sidecar proxy that learns healthy instances from service discovery and picks one per request, with no extra hop.

The names come from the OSI model: layer 4 is the transport (TCP or UDP), layer 7 the application protocol (HTTP, gRPC).

Layer 4: balancing connections

An L4 balancer sees the 5-tuple: source IP, source port, destination IP, destination port and protocol. It must send every packet of a TCP connection to the same backend, because a backend that gets a packet for a connection it never opened answers with a reset. As Cloudflare’s Unimog write-up puts it, a misdirected packet is much worse than a dropped one.

So the balancer hashes the flow. AWS’s Network Load Balancer hashes the protocol, addresses, ports and TCP sequence number, keeps each TCP connection on one target for its whole life, and handles millions of requests a second. QUIC is routed by connection ID instead, because a migrating client changes its address (see TCP vs UDP).

Proxy mode or direct server return

Proxy mode terminates the client’s TCP connection and opens a second one to the backend, copying bytes between them. It works on any network and needs nothing special on the backends. You pay twice: every response byte flows back through the balancer, and the backend sees the balancer’s address, not the client’s. The usual fix is the PROXY protocol: one short header, sent before the client’s data, that carries the original addresses (see proxy vs reverse proxy).

Pass-through with direct server return (DSR) forwards the client’s packets unchanged inside a tunnel such as GRE or IP-in-IP. The backend holds the VIP on its loopback interface, unwraps the packet and replies straight to the client from the VIP. Google’s Maglev and Meta’s Katran both work this way, because responses dwarf requests. For a 1 KB request and a 20 KB response, the balancer carries 1 of 21 KB, under 5% of the bytes.

Keeping every packet on one backend

The obvious design is a table per balancer, 5-tuple → backend, filled on the first packet; Maglev keeps one. But big L4 tiers run many balancer machines behind a router that spreads packets across them with ECMP (equal-cost multipath), and when a machine joins or leaves, ECMP reshuffles flows. A connection’s next packet can land on a balancer with no entry for it.

The fix is consistent hashing: every balancer computes the same backend from the same 5-tuple, with no shared state, and a change to the backend set moves as few flows as possible. Maglev fills a lookup table of prime size M (65,537 by default) by letting backends take turns claiming preferred slots, so each gets ⌊M/N⌋ or ⌈M/N⌉ entries; keeping M above 100 × N limits the imbalance to about 1%. With 1,000 backends, classic ring and rendezvous hashing needed 29.7% and 49.5% spare capacity instead. The price is a little more reshuffling when backends change, which the connection table absorbs.

What L4 can’t do

• Route by content. One VIP and port maps to one pool; there is no path, host or cookie to read.
• Retry a request. It never sees requests; if a backend dies mid-connection, the client gets a reset.
• Read TLS. It forwards ciphertext unless it terminates TLS, which is L7 work.
• Balance work. One connection can carry one request or a million.

In return you get microseconds per packet, no private keys, and any protocol on TCP or UDP.

Layer 7: balancing requests

An L7 balancer terminates the client’s connection (usually TLS, see where TLS terminates), parses HTTP, and keeps its own connections to backends. Envoy, NGINX, HAProxy and AWS’s Application Load Balancer (ALB) are common. Because it reads every request, it can:

• Route by host, path, header, method or cookie, so one address fronts dozens of services.
• Balance each request with a load-aware algorithm, spreading the streams of one HTTP/2 connection across many backends.
• Retry a failed idempotent request on another backend before the client notices.
• Rewrite: ALB adds X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Port so the app learns the client’s address and scheme. Proxies also redirect, compress and cache.
• Split traffic by weight for canaries (see deployment strategies).
• Multiplex: ALB carries requests from many client connections over one backend connection.

The costs: every request pays for TLS, parsing and buffering, so an L7 tier needs far more CPU than an L4 tier for the same traffic. It holds your private keys and sees plaintext, so it is a security boundary. It adds a hop, buffers bodies and enforces limits: ALB caps a request’s headers at 64 KB. Authentication and quotas on top of routing belong to the API gateway.

The long-lived connection trap: gRPC, HTTP/2 and WebSockets

HTTP/2 multiplexes many requests over one connection, and gRPC clients hold that connection open for hours (see REST vs GraphQL vs gRPC). An L4 balancer decides once, when the connection opens, so every request on it goes to the same backend.

gRPC’s own connection-management proposal names the problem: L4 balancers balance connections, not calls, so with HTTP/2 they can’t move load off hot servers or onto new ones. Servers that autoscaling adds at 9 pm get nothing from clients connected since 6 pm. Three fixes:

1. An HTTP/2-aware L7 proxy balances each stream. Cleanest when you don’t control the clients.
2. Client-side balancing. The gRPC team’s guidance: trusted clients learn every backend and pick one per call, or ask a lookaside balancer. No extra hop, but every client language needs the logic.
3. Cap connection age. gRPC servers can send GOAWAY after MAX_CONNECTION_AGE (default: infinite, with ±10% jitter), so clients reconnect through L4 and land somewhere new. Cheap, coarse, and the only option if you’re stuck with L4.

WebSockets are pinned even at L7. With least outstanding requests, ALB picks a WebSocket’s target once and sends every later message over that connection; its routing rules stop applying after the upgrade. Cloudflare keeps such connections up for days. Balance them by connection count at open, cap their lifetime, and make clients reconnect with jitter during deploys (see real-time connections).

L4 vs L7 side by side

Choose by the question you need answered:

• Is the traffic HTTP at all? Databases, MQTT, game servers and raw TCP get L4.
• Must encryption reach the backend untouched? L4 pass-through, with no path routing, retries or caching at the balancer.
• Need path routing, retries, canaries or fair gRPC balancing? L7, in a proxy or in the client.
• Very high packet rates, or floods to absorb? L4 in front of L7, as in Figure 1: the L4 tier sheds bad packets cheaply and lets the L7 tier scale out behind one stable address.

Algorithms and affinity: who gets the next request

Why counting in-flight requests beats taking turns

Round robin assumes every request costs the same and every backend is equally fast; break either assumption and queues form. Take four backends sharing 400 requests a second: B1 to B3 answer in 50 ms, and B4, stuck in garbage-collection pauses, takes 500 ms.

Little’s law says requests in flight = arrival rate × time in the system. Under round robin each backend gets 100 a second, so B1 to B3 hold 100 × 0.05 = 5 each and B4 holds 100 × 0.5 = 50. A quarter of all users wait half a second.

Least outstanding requests sends each request to the backend with the fewest in flight, so the counts converge to a common level L. Each fast backend then serves L ÷ 0.05 = 20L a second and B4 serves L ÷ 0.5 = 2L. From 3 × 20L + 2L = 62L = 400, L ≈ 6.5: the fast backends take about 129 a second each and B4 about 13. Mean latency falls from 0.75 × 50 + 0.25 × 500 ≈ 163 ms to (387 × 50 + 13 × 500) ÷ 400 ≈ 65 ms.

The same logic covers uneven request costs: a backend that drew three slow report exports has a longer queue, so it gets less new work. NGINX’s least_conn and L4 balancers count connections, which is equivalent when each carries one request at a time.

Power of two choices: nearly as good, without the herd

“Always pick the least loaded” breaks down with many balancers working from slightly stale counts. If 20 proxies all see a freshly added backend at zero, all 20 send it their next request at once, and keep doing so until their counts catch up. Michael Mitzenmacher’s study of stale load information found exactly this: sending tasks to the apparently least-loaded server can significantly hurt, while the less loaded of two random servers stays effective across a wide range of conditions.

That rule is power of two choices (P2C): sample two backends at random, send to the one with fewer in flight. Mitzenmacher’s 2001 analysis showed two choices give an exponential improvement over one, and a third adds only a constant factor. The intuition: the busiest 10% of backends get a request only when both samples land among them, 0.1 × 0.1 = 1% of the time, against 10% for random. Envoy’s least-request balancer does this by default; NGINX calls it random two least_conn.

Latency-aware balancing

Counting requests in flight reacts only after a queue builds. Finagle’s peak EWMA reacts sooner: it keeps a peak-sensitive moving average of each backend’s round-trip time, weights it by requests in flight, and compares two random backends on that cost. In a Linkerd experiment where one of 11 backends was stuck at 2 seconds for 30 seconds, callers with a 1-second timeout saw about 95% success under round robin, 99% under least loaded and 99.9% under peak EWMA. The trap: a backend that fails instantly looks like your fastest server, so pair it with outlier detection.

Hashing for affinity

Hashing sends the same key to the same backend: the 5-tuple for L4 flows, or a user ID or cache key at L7 so each node’s cache stays warm. With Envoy’s ring hash, removing one of N hosts moves about 1/N of the keys; its Maglev table builds and picks several times faster but moves about twice as many keys. Either way, a hot key still lands on one node.

Sticky sessions, and why to avoid them

Sticky sessions pin a user to one server with a cookie (ALB’s AWSALB, valid for a day by default) or a source-IP hash, bypassing the algorithm after the first pick. You pay four ways:

• Uneven load. Heavy users stay put, and source-IP hashing sends every phone behind one carrier-grade NAT address to the same backend.
• Scale-out does little. New servers get only new users.
• Failover loses state. Users of a dead server lose their sessions mid-checkout.
• Deploys wait for sessions to drain.

Instead, make the service stateless and keep sessions in a shared store such as Redis, or in a signed token. Soft affinity through consistent hashing is fine when it only improves cache hits and any server could still answer.

Health checks, slow start and draining

A balancer is only as good as its picture of which backends are healthy, and it needs two sources for it. What the endpoint should test is covered in health checks done right.

Active checks: probing on a timer

The balancer calls an endpoint such as /healthz on a timer. ALB’s defaults: every 30 s, a 5 s timeout, 2 consecutive failures to mark a target unhealthy, 5 passes to bring it back.

Do the arithmetic. A server that hangs just after a passing check fails the checks at 30 s and 60 s, each declared at its 5 s timeout, so it’s removed about 65 s after it died: up to 65,000 requests at 1,000 a second. Coming back takes 5 × 30 = 150 s. At a 10 s interval, removal takes about 15 to 25 s.

Passive checks: watching real traffic

Outlier detection watches real responses. Envoy’s defaults eject a host after 5 consecutive 5xx responses (for TCP, failed connections), for 30 s × the number of times it has been ejected, capped at 300 s, and never eject more than 10% of the cluster. At 1,000 requests a second, five failures take 5 ms. Active checks still find servers that get no traffic and confirm recovery. An L4 proxy only sees connection failures; an L7 proxy also sees status codes.

Slow start and draining

A fresh instance has cold caches, an unwarmed JIT and empty connection pools, yet under least outstanding requests it looks idle and gets flooded. Slow start ramps its share up linearly: ALB allows 30 to 900 s, though not with least outstanding requests.

Draining is the mirror image. A deregistered ALB target gets no new requests, but ALB waits up to 300 s by default (0 to 3,600) for in-flight ones; if the process exits early, those clients get 5xx errors. Set the delay a little above your longest normal request, and keep serving for a few seconds after removal, because the balancer’s view lags.

Keeping the balancer itself up

Everything passes through the balancer, so it must not become a single point of failure itself.

Active-passive pairs. The classic setup is two boxes sharing a floating IP through VRRP. The standby takes over after missing the active box’s advertisements for 3 × the interval plus a small skew: about 3 to 4 s at RFC 5798’s 1 s default (see failover). Maglev started this way and moved on: half the machines sat idle, no VIP could exceed one machine’s capacity, and coordinating the pair was complex.

Active-active with ECMP. Every balancer announces the same VIP to the router over BGP at equal cost, and the router hashes flows across all of them. A sick machine withdraws its route. That gives N+1 redundancy, consistent hashing keeps connections on their backends while the balancer set changes, and capacity is machines × per-machine throughput. One Maglev machine saturated a 10 Gbps link with small packets: a 100-byte packet takes 138 bytes on Ethernet with framing, preamble and gap, so 10¹⁰ ÷ (138 × 8) ≈ 9 million packets a second, at about 350 ns each. Unimog makes every edge server a balancer, at under 1% of CPU.

Managed cloud balancers do this for you. An ELB runs a node in each Availability Zone you enable and publishes their addresses in DNS with a 60 s TTL; an ALB needs at least two zones. Across regions, the front door is GeoDNS, which answers per lookup and is cached for the TTL so failover takes minutes, or anycast, where every site announces one address and a failed site withdraws its route. The DNS lesson compares them.

Going deeper: what breaks in production

DSR means an asymmetric path

Requests go client → balancer → backend; responses go backend → client. A stateful firewall that sees only one direction drops the flow, and the balancer gets no passive health signal or latency data. Tunnelling adds bytes: Unimog’s 1,500-byte packets become 1,536 bytes, so Cloudflare runs jumbo frames inside its data centres.

Zones that don’t match

In AWS’s example, one zone has 2 targets and another 8, and each zone’s balancer node gets half the traffic. With cross-zone balancing off, each small-zone target gets 50% ÷ 2 = 25% of all traffic and each large-zone target 50% ÷ 8 = 6.25%, a 4× imbalance. NLB ships with cross-zone off; ALB keeps it on. Keep zones even, or enable cross-zone and pay for that traffic.

Retries that multiply load

L7 retries hide blips but multiply load exactly when a service struggles. If each of five layers makes 3 attempts, one click becomes 3⁵ = 243 calls at the bottom. Retry only idempotent calls, at one layer, within a budget, with backoff and jitter (see retry storm and timeouts, retries and backoff).

Idle timeouts and the 502 race

Each hop closes idle connections on its own timer: ALB after 60 s by default, NLB TCP flows after 350 s. If a backend closes a pooled keep-alive connection just as the balancer sends a request on it, ALB returns 502. Make each server’s keep-alive timeout longer than that of the balancer in front of it, so the balancer always closes first. More traps live in HTTP API implementation gotchas.

Cheat sheet

How to say it in the interview

“Clients hit one VIP. An L4 tier announces it over BGP from several machines, the router spreads flows by ECMP, and each machine picks a backend by consistent-hashing the 5-tuple, so a balancer can die without breaking connections. Behind it, L7 proxies terminate TLS, route by path and pick a backend per request with least outstanding requests over two random choices, which copes with uneven costs and gRPC’s long-lived connections. Health checks plus outlier detection remove bad instances; deploys drain, then slow-start. Services are stateless, so no sticky sessions.”

Isn’t the load balancer a single point of failure? Not if it’s a tier rather than a box. Several balancers announce the VIP over BGP and the router spreads flows by ECMP; a dead one withdraws its route, and consistent hashing keeps connections on their backends. Smaller setups use a VRRP pair with about 3 seconds of failover.

Why does gRPC need L7 or client-side balancing? Every call rides one long-lived HTTP/2 connection and L4 decides once per connection, so a client’s calls all hit one backend and new backends get nothing. Balance per call in an HTTP/2-aware proxy or the client; capping connection age is the fallback.

Round robin or least connections? Round robin only for identical servers and uniform requests. Otherwise count requests in flight, so a slow server’s queue earns it less traffic. With many balancers, take the less loaded of two random picks, so stale counts don’t make them all pile onto one server.

How do you deploy without dropping requests? Deregister first so new requests stop, drain a little longer than your slowest normal request, then stop the process. The new version must pass health checks and then gets a slow-start ramp.

Should we use sticky sessions? Avoid them: they skew load, blunt scale-out and lose sessions on failover. Keep servers stateless with sessions in Redis or a signed token.

Where to go next

• Proxy vs reverse proxy: what the L7 tier is, in general terms.
• API gateway: authentication, quotas and transformations on top of L7 routing.
• Consistent hashing: rings, virtual nodes, and why only 1/N of keys move.
• Health checks done right: liveness versus readiness, and what an endpoint should test.
• Stateful vs stateless design: the design that makes sticky sessions unnecessary.
• Domain Name System: GeoDNS and anycast, the layer above the balancer.
• Sidecar, ambassador and service mesh: client-side L7 balancing for east-west traffic.
• Multi-region and active-active: balancing across regions.

References and further reading

1. Maglev: A Fast and Reliable Software Network Load Balancer (Eisenbud et al., NSDI 2016)usenix.orgGoogle’s L4 balancer: ECMP in front, GRE and direct server return, connection tracking, Maglev hashing, packet rates and the move from active-passive pairs to N+1.
2. Unimog: Cloudflare’s edge load balancer (Cloudflare, 2020)blog.cloudflare.comWhy a misdirected packet breaks a connection, why ECMP alone is not enough, forwarding tables with 100× more buckets than servers, encapsulation overhead and long-lived connections.
3. How Elastic Load Balancing works (AWS)docs.aws.amazon.comNLB flow hashing versus ALB per-request routing, the cross-zone imbalance example, connection multiplexing, X-Forwarded-* headers and header size limits.
4. Edit target group attributes for your Application Load Balancer (AWS)docs.aws.amazon.comRound robin, least outstanding requests and weighted random; slow start; the 300-second deregistration delay; sticky-session cookies; fail-open behaviour.
5. Health checks for Application Load Balancer target groups (AWS)docs.aws.amazon.comDefault interval, timeout and healthy and unhealthy thresholds, and failing open when every target is unhealthy.
6. Envoy: supported load balancersenvoyproxy.ioWeighted round robin, P2C least request and its resistance to herding, random, ring hash (1/N disruption) and Maglev (faster, about twice the key movement).
7. Envoy: outlier detectionenvoyproxy.ioPassive health checking: consecutive 5xx and gateway errors, success-rate ejection, growing ejection times and the maximum ejection percentage.
8. The Power of Two Choices in Randomized Load Balancing (Mitzenmacher, IEEE TPDS, 2001)eecs.harvard.eduThe supermarket model: two random choices give an exponential improvement over one, and a third choice only a constant factor more.
9. How Useful Is Old Information? (Mitzenmacher, IEEE TPDS, 2000)eecs.harvard.eduWith stale load data, sending every task to the apparently least-loaded server hurts, while the less loaded of two random servers stays effective.
10. gRPC Load Balancing (gRPC blog, 2017)grpc.ioProxy versus client-side and lookaside balancing, and why an L7 balancer can spread one client’s HTTP/2 streams across many backends.