TCP vs UDP

TCP and UDP are the two transport protocols on top of IP that carry almost all application traffic. TCP gives you a reliable, ordered byte stream and charges a handshake round trip, per-connection state and stalls while lost data is resent. UDP hands you individual datagrams with no promises, so it is cheap and immediate but leaves loss, ordering and pacing to you. The real question is not “which is faster?” but “what should happen when a packet goes missing?”

Picture it first: a courier and a cricket commentator

You send a 40-page contract to a lawyer by courier. First the courier phones to check someone is there to sign. Every page is numbered, the lawyer signs for each batch, and if page 17 goes missing the courier sees the gap in the receipts and sends it again. The lawyer reads strictly in order, so she waits for page 17 with pages 18 to 25 already on her desk.

Now picture ball-by-ball cricket commentary on the radio. Each ball is described once. If static swallows ball three, the commentator does not go back: ball four is already being bowled, and the next score update tells you what matters.

Map it back:

• The courier is TCP: a setup call (handshake), numbered pages (sequence numbers), receipts (acknowledgments), resends (retransmission) and reading in order. The lawyer stuck on page 17 is head-of-line blocking.
• The commentary is UDP: self-contained datagrams, no setup, no receipts, no repeats. Your application decides whether a gap matters.
• Real apps use both: the courier for a UPI-style payment, the commentary for the live score beside it.

Two contracts on top of IP

IP moves packets between hosts on a best-effort basis: they can be dropped, duplicated, delayed or reordered, and nobody tells the sender. A transport protocol adds ports, so data reaches one process rather than one machine, plus whatever guarantees it offers. The layering is covered in the OSI model, addressing in IP addresses.

Both protocols carry 16-bit source and destination ports and a checksum for corrupted bits. That is where the similarity ends:

• UDP (RFC 768) adds a length field and stops. Its header is always 8 bytes.
• TCP (RFC 9293) adds sequence and acknowledgment numbers, control flags, a receive window and options. Its 4-bit data offset counts 32-bit words, so the header is 5 × 4 = 20 bytes at minimum and 15 × 4 = 60 bytes at most.

A TCP connection is identified by its 4-tuple: source IP and port, destination IP and port. It comes back for TIME_WAIT, load balancers and QUIC.

TCP: a handshake before the first byte

TCP is connection-oriented: both ends agree on starting sequence numbers before any data flows. The three-way handshake does that:

1. The client sends SYN carrying its initial sequence number.
2. The server replies SYN-ACK: its own initial sequence number, plus an acknowledgment of the client’s.
3. The client sends ACK, and can put the first request bytes in the same flight.

RFC 9293’s main reason for three messages is to stop an old, duplicate SYN being mistaken for a new connection. The price is one round trip (RTT) before the request leaves, so the first response byte arrives no earlier than 2 RTTs after the client started.

Round trips add up. Say a phone in Delhi talks to a server in Singapore over an 80 ms RTT (illustrative). A fresh HTTPS request pays 80 ms for the TCP handshake, 80 ms for TLS 1.3 and 80 ms for the request: 240 ms to the first response byte, against 80 ms on an open connection. TLS is covered in HTTP and HTTPS; the transport lesson is that setup is paid in round trips, not bandwidth, so a faster link doesn’t make it cheaper (see latency vs throughput vs bandwidth).

So clients and proxies reuse connections: HTTP keep-alive, long-lived gRPC channels, database connection pools. Reuse also skips slow start.

TCP: numbered bytes, acknowledgments and resends

A byte stream, not messages. TCP numbers bytes, not writes. The reader gets the same bytes in the same order, but in chunks that need not match the writes: two 100-byte sends can arrive as one 200-byte read. Every protocol on TCP must frame its own messages; HTTP uses Content-Length or chunked encoding, and most binary protocols prefix each message with its length.

def send_msg(sock, payload: bytes):
    sock.sendall(len(payload).to_bytes(4, "big") + payload)

def recv_exact(sock, n: int) -> bytes:
    buf = b""
    while len(buf) < n:  # recv() may return fewer bytes than asked for
        chunk = sock.recv(n - len(buf))
        if not chunk:
            raise ConnectionError("peer closed the connection")
        buf += chunk
    return buf

def recv_msg(sock) -> bytes:
    size = int.from_bytes(recv_exact(sock, 4), "big")
    return recv_exact(sock, size)

Cumulative acknowledgments. An ACK number means “I have every byte before this one.” RFC 9293 recommends delayed ACKs: at least one ACK per two full-sized segments, never held for 0.5 seconds or more. Linux holds an ACK for 40 to 200 ms hoping to piggyback it on a reply.

Retransmission. The sender keeps every unacknowledged byte and resends it in one of two ways:

• On a retransmission timeout (RTO), computed from measured RTTs. RFC 6298 sets a 1-second floor (Linux uses 200 ms) and doubles the timer after each failed retry, so a timeout leaves the connection silent for a long time.
• On three duplicate ACKs. When one segment is lost but later ones arrive, the receiver repeats its last ACK for each. RFC 5681 treats three duplicates as a loss and resends at once (fast retransmit).

In-order delivery. Segments that arrive after a gap are buffered, not handed to the application, until the gap is filled. That makes TCP easy to program against, and it causes head-of-line blocking.

Flow control and congestion control

Flow control protects the receiver. Every segment advertises a receive window (rwnd): how many more bytes the receiver can buffer. The 16-bit field is stretched by a window scale option to as much as 1 GiB (RFC 7323). A slow consumer shrinks its window and the sender pauses: backpressure built into the transport.

Congestion control protects the network. The sender also keeps a congestion window (cwnd), its estimate of what the path can hold without overflowing a router queue. Unacknowledged data may never exceed min(cwnd, rwnd). RFC 5681’s classic rules:

• Slow start. cwnd grows by up to one segment per ACK, roughly doubling every RTT. “Slow” describes the start, not the growth.
• Congestion avoidance. Past a threshold (ssthresh), cwnd grows about one segment per RTT.
• Three duplicate ACKs. ssthresh drops to half the data in flight and sending continues from there (fast recovery).
• A timeout. cwnd collapses to one segment and slow start begins again.

The initial window. RFC 6928 raised the starting cwnd to 10 segments, capped at 14,600 bytes. On standard Ethernet a full segment carries 1,500 − 20 (IP header) − 20 (TCP header) = 1,460 bytes, so the first flight is 10 × 1,460 = 14.6 KB. Without loss, each round trip doubles it:

So a 100 KB API response on a brand-new connection needs 3 RTTs of data after the TCP handshake, plus one more for TLS 1.3: 5 × 80 ms = 400 ms on the Delhi–Singapore path, against 80 ms on a warm connection whose window is already large. It is also why front-end engineers try to fit critical HTML into the first ~14 KB. Google’s experiments in RFC 6928 found that going from 3 to 10 initial segments cut average web search latency by 11.7% in one data centre.

Two refinements. A connection idle for over one retransmission timeout may restart from min(initial window, cwnd) (RFC 5681, section 4.1), so pools help most when busy. And CUBIC, the default in Linux, Windows and Apple stacks per RFC 9438, cuts the window to 70% rather than half and regrows along a cubic curve: same probe-and-back-off shape.

Head-of-line blocking at the transport layer

Because TCP hands bytes over strictly in order, one lost segment holds back everything behind it, including data that has already arrived. That data waits in the receive buffer for the resend: at least one more RTT with fast retransmit, a full RTO if the timer fires.

For one file download that barely matters. It matters when unrelated things share a connection. HTTP/2 multiplexes many requests over one TCP connection, so a lost packet from an image also stalls the JSON queued behind it, as Cloudflare’s The Road to QUIC explains. For live media it is worse: an audio frame that arrives 200 ms late is no longer worth playing.

QUIC orders each stream separately, so a loss only blocks the streams whose data was in the lost packet (RFC 9000, section 13). If one packet carried frames from several streams, all of them wait.

UDP: datagrams and nothing else

UDP is barely more than an IP packet with ports. You get message boundaries (one send is one datagram, delivered whole or not at all), no setup or connection state (the first datagram carries data; the server keeps nothing per client unless the app does) and an 8-byte header.

You don’t get delivery, ordering, duplicate suppression, flow control or congestion control. There is also a size rule: stay inside the path MTU, because a fragmented datagram is lost if any fragment is. A 1,500-byte Ethernet MTU over IPv4 leaves 1,500 − 20 − 8 = 1,472 bytes of payload, and QUIC assumes only 1,200 bytes get through everywhere.

RFC 8085, the IETF’s UDP guidelines, lists what your application now owns:

• Congestion control. A low-volume sender should send at most one datagram per RTT, or one every 3 seconds without an RTT estimate. Bulk senders should behave like TCP.
• Loss, duplication and reordering. Assume all three, and survive datagrams delayed by up to 2 minutes.
• Middleboxes. NATs often forget idle UDP flows in under the 2 minutes the NAT requirements ask for. Keep-alives, if needed, no more often than every 15 seconds.
• Abuse. Source addresses are trivially spoofed. Randomise source ports, and don’t answer small unauthenticated requests with large responses.

Here is “the application handles it” for a game client: keep the newest snapshot, drop older ones, never wait for a resend.

last_tick = -1

def on_datagram(data: bytes):
    global last_tick
    tick = int.from_bytes(data[:4], "big")  # the server stamps every snapshot
    if tick <= last_tick:
        return  # late or duplicate: a newer state already won
    last_tick = tick
    apply_snapshot(data[4:])  # a gap just means we skipped a tick

What runs on UDP, and why

• DNS. Question and answer each fit in one datagram, so a lookup costs one round trip and a lost query is just retried. Answers too big for a datagram are truncated and retried over TCP (RFC 7766). More in the domain name system.
• Voice and video calls. Media travels as RTP over UDP; the receiver’s jitter buffer smooths uneven arrival and conceals a lost packet instead of waiting. Discord keeps signalling on WebSockets and sends voice over UDP to its media servers, with clients pinging to keep firewalls open (Discord engineering).
• Real-time game state. Each snapshot supersedes the last, as in the code above.
• Metrics. StatsD clients fire lines such as api.latency:320|ms as datagrams and never wait, so a slow metrics server can’t slow the app (StatsD). Some samples vanish under load.
• QUIC, the transport under HTTP/3.

QUIC: TCP’s guarantees, rebuilt on UDP

QUIC (RFC 9000) is a reliable, encrypted, multiplexed transport. Google’s SIGCOMM 2017 paper gives two reasons it runs over UDP: middleboxes pass UDP but often drop unfamiliar protocols, and a user-space transport ships with the app instead of waiting for operating systems to update. Compared with TCP plus TLS:

• Faster setup. Transport and TLS 1.3 handshakes are combined: 1 RTT before the request instead of 2. A returning client can send its request in the first flight (0-RTT), but 0-RTT data can be replayed, so allow it only for idempotent requests.
• Independent streams. Loss on one stream doesn’t stall the others (Fig. 3).
• Connection IDs. Packets map to a connection by ID, not 4-tuple, so it survives a NAT rebinding or a phone moving from Wi-Fi to mobile data. In QUIC version 1 only clients migrate.
• Encrypted headers. Nearly everything is encrypted and authenticated, so middleboxes can’t come to depend on (and freeze) its internals.
• Anti-amplification. Until a client’s address is validated, a server may send at most 3× the bytes it received; client first packets are padded to 1,200 bytes.

The paper reports the payoff and the bill. Search latency fell 8.0% on desktop and 3.6% on mobile, YouTube rebuffering fell 18.0% and 15.3%, and QUIC carried an estimated 7% of internet traffic by 2017. But it first cost about 3.5× the server CPU of TLS over TCP (about 2× after optimisation), and 4.4% of clients couldn’t use it, mostly behind corporate firewalls blocking UDP, so Chrome races QUIC against TCP and falls back. HTTP/3 itself is covered in HTTP and HTTPS.

TCP, UDP and QUIC compared

The load-balancing row hides real work: an L4 balancer hashing the 4-tuple sends a migrated QUIC connection to the wrong server unless it routes on the connection ID, and long TCP connections pin clients to one backend, which complicates draining. See load balancing L4 vs L7.

Choosing a transport in an interview

Ask two questions about each data flow, not about the whole product:

1. Is a late packet still useful? If a resent packet would arrive after it mattered (a voice frame, a game tick), reliability hurts. Use UDP and handle loss in the app.
2. Must every byte arrive, in order? If loss or reordering corrupts the result (a payment, a chat message, a file), use TCP, or QUIC where HTTP/3 is available.

Why do “real-time” web features still ride TCP? A WebSocket starts as an HTTP request and takes over the same TCP connection, inheriting its ordering and head-of-line blocking. For chat, live scores and dashboards that is the right trade: messages are small and each one matters, in order. In browsers, unreliable UDP-based delivery comes mainly through WebRTC, which video calls use. See real-time connections.

Going deeper: what breaks in production

Packet loss caps TCP throughput

Loss-based congestion control reads every drop as a reason to slow down, so random loss caps a TCP flow’s rate however fat the pipe. Mathis, Semke, Mahdavi and Ott’s 1997 model estimates it for a long-running flow:

throughput ≈ (MSS / RTT) × (C / √p)      C ≈ 1.22, p = packet loss rate

With MSS = 1,460 bytes and a 20 ms RTT, MSS / RTT = 73,000 B/s. At 1% loss, C / √p = 1.22 / 0.1 = 12.2, so throughput ≈ 73,000 × 12.2 ≈ 890 KB/s ≈ 7.1 Mbit/s, even on a gigabit link.

Throughput falls with RTT and with the square root of loss: 200 ms at 0.01% loss tops out where 20 ms at 1% does. Treat these as Reno-style intuition (the model assumes no timeouts; CUBIC and other algorithms differ), but the lessons hold: shorten the RTT by terminating connections near users, and don’t expect a fatter pipe to fix a lossy one.

TIME_WAIT and ephemeral port exhaustion

The side that closes a TCP connection first keeps its 4-tuple in TIME_WAIT for 2 × MSL, so stray segments from the old connection can’t be mistaken for a new one on the same ports. RFC 9293 takes the MSL as 2 minutes (so 4 minutes); Linux hard-codes 60 seconds.

That bites proxies and gateways calling one backend at high rates. Each outgoing connection takes a source port from the ephemeral range, on Linux 32,768 to 60,999: 60,999 − 32,768 + 1 = 28,232 ports. If a proxy opens a fresh connection per request to one backend IP and port and closes it first, each port sits in TIME_WAIT for 60 s, capping it at 28,232 / 60 ≈ 470 new connections per second. Beyond that, connect() fails with “Cannot assign requested address”. Cloudflare hit a variant: binding a source IP before connecting made Linux reserve a port per connection across all destinations, until they set IP_BIND_ADDRESS_NO_PORT.

The fixes, best first:

• Reuse connections. A pool of 200 keep-alive connections replaces thousands of short ones and skips handshakes and slow start too.
• Spread the 4-tuple. Each extra destination IP or port, or source IP, gets its own 28,232 ports.
• Let the client close first, so TIME_WAIT lands on many clients, not one proxy.
• Tune with care. A wider port range or tcp_tw_reuse for outgoing connections helps; Linux’s docs advise changing the latter only on expert advice.

Nagle’s algorithm meets delayed ACK

Nagle’s algorithm, which RFC 9293 says stacks should implement, holds back small writes while earlier data is unacknowledged, so chatty apps don’t flood the network with tiny packets. Delayed ACK makes the receiver wait (40 to 200 ms on Linux) to piggyback its ACK. Now a client writes a request as headers, then body: Nagle holds the body until the headers are acknowledged, and the server delays that ACK until the body arrives. Every request stalls for up to the delayed-ACK timer.

The fix is one write per request, or TCP_NODELAY, which RFC 9293 requires every stack to offer per connection; many latency-sensitive RPC and database clients set it.

Keep-alives: TCP’s own are too slow

Per RFC 9293, TCP keep-alives default to off and, when on, wait at least two hours of idleness before probing. NATs, firewalls and load balancers usually forget idle connections far sooner, and the next write fails or hangs. Long-lived connections (WebSockets, database pools, gRPC channels) need application heartbeats or keep-alive timers below the shortest idle timeout on the path, and pools should validate idle connections before use.

UDP amplification

With no handshake, an attacker can send a small request with the victim’s address forged as the source, and the server replies to the victim. If the reply is much bigger than the request, the attacker’s bandwidth is multiplied. CISA lists amplification factors of 28 to 54× for DNS, 556.9× for NTP and 10,000 to 51,000× for memcached. A UDP protocol you design should never answer a small unauthenticated request with a large response, and should rate-limit replies; QUIC bakes this in with its 3× limit.

Middleboxes and blocked UDP

Google’s 2016 measurements found QUIC worked for 95.3% of video clients, failed for 4.4% (UDP blocked or the path MTU too small, mostly corporate networks) and was throttled for 0.3%. So every UDP-based feature needs a TCP fallback, from HTTP/3 dropping to HTTP/2 to call media relayed over TCP port 443. Expect it to be slower.

Cheat sheet

How to say it in the interview

“TCP gives me a reliable, ordered byte stream: a one-round-trip handshake, ACKs and resends, and congestion control that starts at about 14 KB per round trip and doubles. The cost is setup latency and head-of-line blocking. UDP gives me bare datagrams, so I use it only where late data is useless (call media, game state, metrics) and add sequence numbers, pacing and a TCP fallback. APIs, chat and payments go on TCP, or QUIC via HTTP/3, which sets up in one round trip and isolates loss per stream. And I reuse connections, because handshakes and slow start cost round trips.”

Why not build your own reliability on UDP? You would be rebuilding TCP: ACKs, resends, ordering, congestion control, security. Do it only for partial reliability, such as dropping stale frames; otherwise use QUIC.

Does HTTP/2 fix head-of-line blocking? Only at the HTTP layer. Its streams still share one TCP byte stream, so a lost packet stalls all of them. HTTP/3 over QUIC fixes it at the transport.

Why does a new connection feel slow on a fast network? Round trips, not bandwidth: one RTT for TCP, one for TLS, then slow start sends 14.6, 29.2 and 58.4 KB per round trip. A 100 KB response takes 5 RTTs on a new connection and 1 on a warm one.

Your proxy logs “cannot assign requested address”. What’s happening? Ephemeral port exhaustion. Each closed outgoing connection holds its port in TIME_WAIT for 60 s on Linux, so fresh connections to one backend run out at about 28,232 / 60 ≈ 470 per second. Pool connections and spread across more IPs.

Would you use UDP for a chat app? Not for messages, which must arrive once and in order: a WebSocket over TCP fits. UDP fits only the app’s voice and video calls.

Where to go next

• HTTP and HTTPS: TLS, and HTTP/1.1 vs HTTP/2 vs HTTP/3 on these transports.
• The OSI model: where the transport layer sits.
• Real-time connections: long polling, SSE and WebSockets, all on TCP.
• Load balancing L4 vs L7: balancing by 4-tuple or by request.
• Latency vs throughput vs bandwidth: why round trips dominate small requests.
• Connection pooling: reusing TCP connections to databases.
• Timeouts, retries and backoff with jitter: what to do when a connection stalls.
• Design Zoom: TCP signalling and UDP media in one design.

References and further reading

1. RFC 9293 — Transmission Control Protocol (TCP)rfc-editor.orgThe consolidated 2022 TCP spec: header layout, three-way handshake, TIME-WAIT for 2 × MSL, Nagle, delayed ACKs and the two-hour keep-alive default.
2. RFC 768 — User Datagram Protocolrfc-editor.orgThe whole of UDP on three pages: the 8-byte header and the absence of delivery or duplicate protection.
3. RFC 5681 — TCP Congestion Controlrfc-editor.orgSlow start, congestion avoidance, fast retransmit after three duplicate ACKs, and restarting the window after idle periods.
4. RFC 6928 — Increasing TCP’s Initial Windowrfc-editor.orgThe 10-segment (14,600-byte) initial window and Google’s measurements of its latency benefit.
5. RFC 8085 — UDP Usage Guidelinesrfc-editor.orgWhat an application on UDP must handle itself: congestion control, loss, reordering, MTU, middlebox keep-alives and amplification.
6. RFC 9000 — QUIC: A UDP-Based Multiplexed and Secure Transportrfc-editor.orgStreams, per-stream head-of-line behaviour, 0-RTT, connection IDs and migration, the 1,200-byte minimum and the 3× anti-amplification limit.
7. The QUIC Transport Protocol: Design and Internet-Scale Deployment (Langley et al., SIGCOMM 2017)research.googleWhy QUIC runs on UDP, its latency and rebuffering gains, its CPU cost, and how often UDP was blocked or throttled.
8. The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm (Mathis et al., 1997)ccr.sigcomm.orgThe throughput ≈ (MSS / RTT) × C / √p model for loss-limited TCP and its assumptions.
9. How to stop running out of ephemeral ports and start to love long-lived connections (Cloudflare, 2022)blog.cloudflare.comThe Linux ephemeral port range, the 4-tuple, and how a busy proxy ran out of source ports.
10. UDP-Based Amplification Attacks (CISA alert TA14-017A)cisa.govHow spoofed UDP requests are reflected and amplified, with bandwidth amplification factors for DNS, NTP, memcached and others.