HTTP API implementation gotchas

An HTTP API can pass every unit test and still fail in production, because most of its bugs live between the hops: in timeouts nobody set, in connections two machines close at different moments, and in status codes, redirects and cache headers that clients, proxies, CDNs and browsers act on without asking you. This field guide collects the gotchas that bite real APIs, explains each one at the protocol level, and gives you the symptom to look for and the fix.

Picture it first: the rider at the society gate

A delivery rider brings orders to a gated housing society. The food is rarely the problem; the conventions between rider, gate and residents are:

• No deadline. The guard steps away, and the rider waits indefinitely while ten other orders go cold.
• The side gate. The guard locks it after 5 idle minutes, but the rider’s app assumes it stays open for an hour. Now and then he walks into a locked gate.
• The second parcel. The intercom crackles, he can’t tell whether the parcel was accepted, so he hands over another. The resident pays twice.
• The “moved” note. A note says flat 4B moved to 12B. He walks to 12B to ask what was ordered, leaving the parcel on the bike.
• The cheerful receipt. A refused parcel is logged “Delivered (refused)”, so every dashboard shows 100% success.

Map it back: the rider is your HTTP client, the gate is the load balancer and proxies, the society’s rules are HTTP’s defaults, and the resident is your database. The five stories are a missing timeout, a keep-alive race, a retried POST, a redirect that turns POST into GET, and a 200 that carries an error.

The habit to take into an interview: walk one request hop by hop and ask at each what the timeout is, what gets reused, what gets cached and what gets retried.

Timeouts and retries

No timeout means waiting forever

Many HTTP clients ship with no timeout at all. The Requests documentation is blunt: requests do not time out unless you set a timeout explicitly, and without one your code may hang for minutes or more.

A hung call holds a thread, a connection and memory. Say a service has 200 worker threads, takes 1,000 requests a second, and a downstream starts hanging on 5% of calls. That strands 1,000 × 5% = 50 threads a second, so all 200 are gone in 200 ÷ 50 = 4 seconds. The service is down with its CPU idle.

Connect, read and total are different timers

“Set a timeout” hides three questions:

• Connect timeout: how long to wait for the TCP (and usually TLS) handshake. Inside a region a healthy server answers in milliseconds, so keep it short, but check what it covers: the Amazon Builders’ Library describes a 20 ms timeout that fired only after deployments, because it included setting up fresh TLS connections.
• Read timeout: in Requests and most socket-based clients this is the longest gap between bytes, not the time for the whole response. A server that sends one byte every 20 seconds never trips a 30-second read timeout.
• Total deadline: wall-clock time for the whole call, including DNS, connecting, retries and the body. Only this one protects your caller, and you often build it yourself around the client.

How long should the deadline be? Amazon’s approach is to pick an acceptable rate of false timeouts, such as 0.1%, and set the timeout near the downstream’s matching latency percentile (p99.9), plus an allowance for network distance when the caller is on the internet.

Budgets shrink inward

Down a call chain, each caller must wait longer than the callee: say app 10 s, gateway 8 s, orders service 3 s, database 1 s. Invert that and the gateway returns 504 while the work carries on, so a user who taps “Pay” again may pay twice. Better still, pass the remaining deadline downstream (gRPC does this natively) so inner calls stop once nobody is waiting. The API gateway article draws both layouts.

Retry only what is safe, at one layer

Retries turn blips into successes, and four rules keep them from causing outages:

• Only idempotent or keyed requests. RFC 9110 says a client should not automatically retry a non-idempotent request unless it knows the request is safe to repeat. A timed-out POST may have committed, so send an idempotency key and let the server deduplicate.
• One layer, not every layer. Retries hide in SDKs, HTTP client libraries, service meshes and load balancers. If the app, the gateway and a service each make 3 attempts, one tap becomes 3 × 3 × 3 = 27 database calls just when the database is struggling. The Builders’ Library’s five-layer version reaches 3⁵ = 243×, so Amazon retries at a single point in the stack.
• Backoff with jitter. Without randomness, 10,000 clients that failed in the same second retry in the same second, in synchronised waves. Sleep for a random time up to min(cap, base × 2^attempt).
• A retry budget. Cap retries with a token bucket, for example at 10% of requests, so a full outage costs at most 1.1× normal traffic instead of 3×. Honour Retry-After on 429 and 503; it carries either seconds or a date.

More in timeouts, retries and backoff with jitter and retry storm.

Connections: keep-alive, pools and the 502 race

The idle-timeout race

A load balancer keeps warm keep-alive connections to each backend and reuses them. Both ends run their own idle timer, and both may close an idle connection whenever theirs fires. An Amazon ALB’s idle timeout is 60 seconds by default. A Node.js server’s keepAliveTimeout defaults to 5 seconds. So the backend closes first, and in the brief window before its close reaches the balancer, the balancer can send a new request into a socket the backend has already closed.

The symptoms are distinctive: a small, steady trickle of 502s, often worse at quiet times (connections sit idle longer), with no matching error in application logs. AWS’s documentation describes exactly this and recommends making the application’s idle timeout larger than the balancer’s. For Node behind an ALB:

const server = http.createServer(app);
server.keepAliveTimeout = 65_000; // must outlive the ALB's 60 s idle timeout

The rule generalises to every hop: each server’s keep-alive must outlive the idle timeout of the client in front of it. NGINX’s default keepalive_timeout of 75 s already beats a 60 s balancer. Your own outbound clients hit the mirror image: a pooled connection the server closed fails with ECONNRESET, and while a GET can be retried on a fresh connection, a POST cannot be retried blindly. Keep client pool idle timeouts shorter than the server’s keep-alive.

One client, shared and bounded

Bad: a new client per request. Each call pays a fresh TCP and TLS handshake (the handshake tax), and the side that closes first parks the port in TIME_WAIT. A client opening fresh connections to one backend tops out near 470 new connections a second before connect() fails (the arithmetic is in TCP vs UDP). Create one client per process and reuse it; in Python that means a requests.Session, which pools connections. This is the improper instantiation anti-pattern.

Good: a bounded pool sized by Little’s law. In-flight requests = rate × latency, so 2,000 requests a second at 50 ms needs about 2,000 × 0.05 = 100 connections. Too small and requests queue for a connection: latency climbs while the downstream looks healthy, so export the pool’s wait time as a metric. Unbounded and a slowdown to 1 s asks for 2,000 connections against a struggling service. A bounded pool with a short acquire timeout is a bulkhead; see connection pooling.

Great: also cap connection age. A pooled connection keeps talking to the IP address it opened, whatever DNS says now. After a DNS-based failover or blue-green switch, long-lived connections keep hitting the old target until they die. Recycle connections after a few minutes so traffic follows DNS and spreads onto new instances.

Status codes that tell the truth

200 with an error inside

Returning 200 OK with a body like {"status": "error"} breaks every layer that reads status codes. Caches may store the failure as a success, retry logic skips the transient error, passive health checks see a healthy backend, and your 5xx alert never fires while users fail. Use the real code and put details in a problem details body.

500 for a caller’s mistake

The opposite bug: a validation failure escapes as an unhandled exception and becomes a 500. Clients and proxies retry a request that can never succeed, and on-call gets paged for bad input. Map errors deliberately, because the class decides what generic code does next:

404 or 403 for things the caller can’t see

RFC 9110 lets a server answer 404 to hide that a forbidden resource exists. Use that for object IDs: if /v1/orders/1002 belongs to someone else, “not found” stops callers learning which IDs exist. Keep 403 for actions on things the caller can already see, such as an agent issuing a refund without permission. Note that 404 is heuristically cacheable, which matters below.

Redirects that rewrite your request

RFC 9110 carries a historical wart: on a 301 or 302, a user agent may change POST to GET for the follow-up request, and when it does, it drops the body. 307 and 308 were added to forbid that: the client must not change the method. (303 changes to GET on purpose, for the post-then-show-result pattern.)

The usual culprits:

• Trailing-slash and canonical-URL redirects. Web frameworks happily redirect /v1/orders to /v1/orders/. For a POST, the order silently isn’t placed. Accept both forms or return 404; never redirect a write.
• HTTP-to-HTTPS redirects on an API. By the time the redirect arrives, the token has already crossed the network in clear text, and the POST body may not survive the hop. Don’t serve plain HTTP for APIs at all; browsers get HSTS.
• Credentials across hosts. RFC 9110 tells clients to consider stripping Authorization and Cookie when following a redirect, and many do when the host changes. A redirect to another domain then turns into a puzzling 401.
• Sticky permanent redirects. 301 and 308 are heuristically cacheable, so browsers can remember a wrong permanent redirect long after you fix it. Use 307 while you experiment.

Caching headers: say it explicitly

No Cache-Control header does not mean “not cached”. RFC 9111 lets caches assign heuristic freshness to responses without explicit expiry when the status is heuristically cacheable (200, 301, 404 and others), and suggests about 10% of the time since Last-Modified. A /v1/menu response last modified 10 days ago can be served stale for 10 days × 10% = 1 day by a browser or proxy you don’t control. Every response, errors included, should state its policy:

Vary tells a cache which request headers change the response: Vary: Accept-Encoding for compression, Vary: Origin when CORS echoes an allowlisted origin. Forget Vary: Origin and a cache serves one site’s CORS header to another, giving CORS errors that come and go. Varying on Authorization or Cookie makes one cache entry per user, which is effectively uncacheable, so prefer private. For revalidation with ETag, If-None-Match and 304, see HTTP and HTTPS.

Lost updates and other races

Two editors, one record

Two managers of a cloud kitchen open the same dish in a Swiggy-style restaurant dashboard. A cuts the price; B renames the dish and saves the whole form, including the old price. Plain PUT replaces the whole resource, so B’s save quietly undoes A’s change. Both got 200.

The fix is optimistic concurrency with HTTP’s own validators. The GET returns an ETag; the client echoes it in If-Match; the server performs the write only if the version still matches, and otherwise answers 412 Precondition Failed, as RFC 9110 intends for the lost-update problem:

PUT /v1/restaurants/r_88/dishes/42 HTTP/1.1
If-Match: "v7"
Content-Type: application/json

{"name": "Paneer roll (large)", "price_paise": 24900}

HTTP/1.1 412 Precondition Failed

UPDATE dishes SET name = $1, price_paise = $2, version = version + 1
WHERE id = 42 AND version = 7;  -- 0 rows updated means someone got there first: 412

The check and the write must be one atomic statement, or two requests can both pass the check. Derive the ETag from a version counter, not a one-second timestamp, so two writes in the same second still differ. PATCH with only the changed fields shrinks the clash but doesn’t remove it for the same field. See optimistic vs pessimistic locking.

Read-your-writes after a create

The app POSTs an order, gets 201, then immediately GETs /v1/orders/123. The read lands on a lagging replica or a cached 404, so the user sees “order not found” and taps again. Return the full resource in the 201 body with a Location header so no read is needed; route that user’s reads to the primary for a few seconds after a write; and never cache 404s on such routes.

Double submit

During an IRCTC-style tatkal rush, users tap “Book” twice, the app retries after a timeout, or two tabs submit together. Disabling the button can’t cover retries or tabs. Generate an idempotency key once per checkout and reuse it on every retry; the server replays the first result. See idempotency.

CORS: the browser asks first

When a page on app.example calls api.example, the browser enforces CORS. Only “simple” requests go straight out: GET, HEAD or POST with safelisted headers, and a Content-Type of application/x-www-form-urlencoded, multipart/form-data or text/plain. An application/json body or an Authorization header makes the request non-simple, so the browser first sends an OPTIONS preflight and waits for permission. Nearly every JSON API call across origins pays one extra round trip.

The gotchas:

• Short or missing Access-Control-Max-Age. Without it, the browser caches a preflight for just 5 seconds (MDN); browsers cap longer values at 2 hours (Chromium) or 24 hours (Firefox). And because the Fetch Standard keys the cache by URL, /v1/orders/101 and /v1/orders/102 each pay their own preflight. Assuming an 80 ms mobile round trip, that adds 80 ms before the first call to every new URL. Serving the API from the page’s own origin (app.example/api/… through a reverse proxy) removes CORS entirely.
• Auth middleware in front of OPTIONS. Preflights never carry cookies or tokens. A gateway that demands a token on every request rejects the preflight with 401, and the browser reports only a vague CORS error. Answer preflights before authentication.
• Wildcard with credentials. Access-Control-Allow-Origin: * is not allowed for credentialed requests; the browser blocks the response. The tempting fix is worse.

Limits, framing and headers along the path

Size limits differ per hop. NGINX’s client_max_body_size defaults to 1 MB and answers 413; a header line over 8 KB gets a 400. A delivery app’s 3 MB proof-of-delivery photo works against the app server on a laptop and fails in production behind NGINX. Decide limits per route, set them at every hop, document them, and send large files straight to object storage with presigned URLs.

Slow clients hold connections. A client that sends headers a byte at a time (slowloris) ties up a connection as long as the server allows. Node answers 408 once headers exceed headersTimeout (the smaller of requestTimeout and 60 s) or the request exceeds requestTimeout (300 s). NGINX’s client_body_timeout of 60 s is measured between reads, the same trickle loophole as a client read timeout. A buffering reverse proxy in front absorbs slow clients.

Big responses. Building a 200 MB JSON array in memory per request runs a server out of memory under modest concurrency. Paginate from day one: GET /v1/orders works until one merchant has 400,000 orders. For exports, stream newline-delimited JSON, and remember proxies may buffer a stream.

Compression. Compressing a response that holds both a secret (say a CSRF token) and attacker-reflected input leaks the secret through response length: the BREACH attack. Keep secrets out of such responses or mask them per response.

Headers you must not trust or mangle.

• Never trust client-supplied X-Forwarded-For or Host; read them as the proxy article describes, which also covers request smuggling, where a front end and back end disagree on Content-Length versus Transfer-Encoding.
• Header names are case-insensitive (RFC 9110), and HTTP/2 sends them in lowercase. Code that looks up X-Request-Id in an exact-case map breaks the day a proxy starts speaking HTTP/2.
• HTTP/2 also forbids connection-specific headers such as Connection, Keep-Alive and Transfer-Encoding, so an app that sets them by hand can produce malformed messages after an upgrade. These hop-by-hop headers belong to the connection, not the request.

Payload traps

• Timestamps without a zone. 2026-10-27 16:52:42 is ambiguous. Send RFC 3339 timestamps in UTC: 16:52:42 in India (UTC+05:30) is 2026-10-27T11:22:42Z. Convert to local time only for display.
• Money as a float. Binary floating point can’t represent 0.1 exactly, so 0.1 + 0.2 gives 0.30000000000000004. Send integer minor units with a currency code (₹1,620.50 is 162050 paise, INR) or a decimal string.
• Big IDs. JavaScript numbers lose precision above 2^53, so a 64-bit ID parsed as a number can silently change. Send IDs as strings; see serialization and schema evolution.
• Enums grow. A client with an exhaustive switch crashes on a new order status such as refund_pending. Document that unknown values may appear, and map them to a safe default.
• null, missing and empty. In a PATCH, missing should mean “leave alone”, null “clear it” and "" “set to empty” (JSON Merge Patch uses null to delete). Write the rule down; clients will each guess differently.

Going deeper: from symptom to fix

In production you meet these bugs as symptoms, not causes. The table runs backwards from what you see.

Cheat sheet

How to say it in the interview

“Most HTTP API failures live between hops, so I walk the request. The client has a short connect timeout and a total deadline, budgets shrink down the chain, and only idempotent or keyed calls are retried, at one layer, with jittered backoff and a budget. Clients share one bounded pool, and every server’s keep-alive outlives the idle timeout of whatever sits in front of it, or we get random 502s. Status codes are honest, 429 and 503 carry Retry-After, and every response states its caching, with per-user data private. Creates take an idempotency key, updates take If-Match, writes never redirect, and CORS uses an exact allowlist.”

Clients see occasional 502s behind the load balancer. What do you check? First, whether the backend’s keep-alive is shorter than the balancer’s idle timeout (Node’s 5 s against an ALB’s 60 s is the classic pair): the balancer reuses a connection the backend just closed, so the 502s show in balancer logs but not app logs. Raise the backend to 65 s, then check deploy draining and crashing instances.

How do you stop two admins overwriting each other’s edits? Return an ETag on reads and require If-Match on updates. The server checks the version and writes in one atomic statement, answers 412 on a mismatch, and the client re-reads, merges and resends.

Why not return 200 with an error field? Generic layers act on the status code: caches store the failure as success, retry logic skips it, and health checks and alerts count it as healthy. Use the right code and a problem-details body.

What timeouts would you set on this call chain? A connect timeout of tens to hundreds of milliseconds inside a region, and a total deadline near the downstream’s p99.9 plus margin, shrinking inward: say client 10 s, gateway 8 s, service 3 s, database 1 s. Pass the remaining deadline down, and retry at one layer only.

Why is my POST becoming a GET? Something answered 301 or 302, often a trailing-slash or HTTP-to-HTTPS redirect, and RFC 9110 lets clients switch POST to GET on those. Remove redirects from write endpoints, or use 307 or 308, which keep the method and body.

Where to go next

• HTTP and HTTPS: methods, status codes, keep-alive and caching basics.
• Idempotency: idempotency keys for every retried write.
• Timeouts, retries and backoff with jitter: choosing values and retry policies.
• Retry storm: what layered retries do to a struggling service.
• Proxy vs reverse proxy: X-Forwarded-For, buffering, header limits and request smuggling.
• Load balancing: L4 vs L7: idle timeouts, health checks and draining.
• Connection pooling: pool sizing and timeouts at every layer.
• Rate limiting: 429, Retry-After and well-behaved clients.
• What is an API?: status classes, problem details and versioning.
• Optimistic vs pessimistic locking: the database side of If-Match.

References and further reading

1. RFC 9110 — HTTP Semanticsrfc-editor.orgIdempotent methods and automatic retries (9.2.2), field names are case-insensitive (5.1), If-Match and 412 (13.1.1), 404 to hide a forbidden resource (15.5.4), and 301/302 versus 307/308 (15.4).
2. RFC 9111 — HTTP Cachingrfc-editor.orgHeuristic freshness and the typical 10% of time since Last-Modified (4.2.2), Vary (4.1), private (5.2.2.7) and the rule for requests carrying Authorization (3.5).
3. Amazon ALB: edit load balancer attributes (connection idle timeout)docs.aws.amazon.comThe 60-second default idle timeout, and the 502 the balancer returns when a target closes a keep-alive connection first.
4. Node.js documentation: HTTP server timeoutsnodejs.orgserver.keepAliveTimeout defaults to 5 s, headersTimeout to the smaller of requestTimeout and 60 s, requestTimeout to 300 s.
5. NGINX ngx_http_core_modulenginx.orgclient_max_body_size defaults to 1m and returns 413; keepalive_timeout 75s; client_header_timeout and client_body_timeout 60s, measured between reads.
6. Requests documentation: timeoutsrequests.readthedocs.ioRequests never time out unless you pass a timeout; the read timeout is the gap between bytes, not the whole download.
7. Timeouts, retries and backoff with jitter (Amazon Builders’ Library)aws.amazon.comChoosing timeouts from downstream latency percentiles, the 243× retry amplification example, retrying at a single layer, and token-bucket retry limits.
8. MDN: Cross-Origin Resource Sharing (CORS)developer.mozilla.orgWhich requests skip the preflight, why preflights carry no credentials, the wildcard ban for credentialed requests, and Vary: Origin.
9. MDN: Access-Control-Max-Agedeveloper.mozilla.orgPreflight cache lifetime: 5 s by default, capped at 2 hours in Chromium and 24 hours in Firefox.
10. WHATWG Fetch Standard: CORS-preflight cachefetch.spec.whatwg.orgPreflight cache entries are keyed by origin, URL and credentials mode, so every distinct URL needs its own preflight.