API gateway

An API gateway is a reverse proxy that acts as the single front door for client-facing API traffic. It authenticates callers, enforces rate limits and other shared policy, and routes each request to the right service, so services don’t each re-implement the same edge concerns. The skill is deciding what belongs in it, and keeping the rest out.

Picture it first: the main gate of a housing society

A large housing society has 12 towers and one main gate. The guard checks who you are, rings the flat, writes you into the register, and hands you a slip: “B-1204, delivery, 7:42 pm”. Nobody in the towers checks your ID again; they trust the slip. The gate also enforces society-wide rules: deliveries use the service lift, and a flat that has had 15 parcels today collects the rest at the gate. The guard never decides what the family is cooking.

Two failure stories every society knows: the Diwali-rush queue at the one gate backs up onto the road, and one badly worded circular (“no visitors till further notice”) locks out every tower at once.

Map it back:

• The gate is the API gateway; checking ID and ringing the flat are authentication and coarse authorization.
• The slip is the identity header. It is trustworthy only because there is no gap in the back wall: services accept traffic only from the gateway.
• The register is the access log, the service lift is routing, 15 parcels a day is a quota, and the cooking is business logic.
• The Diwali queue is capacity; the bad circular is a bad config push.

The problem a gateway solves

Picture a food-delivery app with six services (users, catalogue, orders, payments, search, reviews) called by a web app, a mobile app and partner restaurants’ systems. Without a gateway, every client knows every service: 3 × 6 = 18 client-to-service links. Each service also terminates TLS, validates tokens, limits callers and writes access logs in its own code: 6 services × 4 concerns = 24 implementations of rules that should behave identically, and won’t.

Microsoft’s microservices guidance adds the other costs of direct calls: clients coupled to how you split services, extra round trips per screen, protocols browsers can’t speak, and more endpoints to harden.

Technically, an API gateway is a specialised reverse proxy: it ends the client’s connection and opens its own to a service. What makes it a gateway rather than a plain L7 load balancer is that it understands API concepts (consumers, keys, scopes, routes, versions, quotas) and applies policy per route and per caller. It also hides service boundaries: clients call /v1/orders forever, while you move that path from a monolith to a new service by changing one route, as in a strangler fig migration.

When you don’t need one

One service with one client gains little over framework middleware and a load balancer, and internal batch jobs and service-to-service calls shouldn’t detour through the public gateway. Draw one once you have several services behind a public API, several client types, or outside developers with keys.

The jobs, and what each one costs

Authentication offload and trusted identity headers

The gateway verifies the caller once, at the edge. For a JWT that means checking the signature against the identity provider’s public keys (its JWKS document), then issuer, audience and expiry. Envoy’s JWT filter caches the keys for a set duration (300 s in its example), so a valid token costs a local signature check, not a network call. Opaque tokens need introspection, a call to the identity provider, so cache the answer briefly or every request pays a round trip.

The gateway then passes identity on as plain headers:

GET /v2/orders/9876 HTTP/1.1
Host: orders.internal
X-User-Id: u_42
X-Tenant-Id: t_7
X-Scopes: orders:read
traceparent: 00-5f2c8a1e9b3d4c7f8a6e2d1b0c9f7e3a-3a7c9e1f5b2d4086-01

Those headers are only as trustworthy as the path they arrived on:

1. Strip before you set. Delete any X-User-Id or X-Scopes the client sent, then write your own; otherwise a caller simply sends X-User-Id: admin.
2. No back door. Services accept these headers only from the gateway, through network policy and ideally mTLS. Microsoft’s gateway offloading guidance is blunt: block direct access to backends, and don’t treat unverified forwarded headers as proof of identity.

API keys are not authentication. AWS’s usage plan docs warn that a key valid for one API in a plan works for every API in it, and keys end up in logs. Use keys to identify and meter apps; authenticate with tokens, signed requests or mTLS (see authentication, sessions, tokens and JWT).

Coarse authorization at the edge, fine-grained in the service

The gateway can answer “may a token with scope orders:read call GET /v2/orders/*?” from the token and route alone. It cannot answer “may user 42 see order 9876?” without the order’s owner, which lives in the orders database, so ownership checks stay in the service. For richer central policy, Envoy’s external authorization filter asks a decision service on every request, returns 403 on a deny and by default fails closed: one policy engine for every route, paid for with a network hop per request.

Rate limits, quotas and usage plans

After authentication the gateway knows the caller, and a rejection costs no backend work, so limits live here. A throttle is a rate plus a burst (a token bucket) protecting capacity second by second; a quota is a budget per day or month per key, a product and billing concept. AWS usage plans expose both per API key, as best-effort targets rather than hard ceilings. Algorithms are in rate limiting; the Redis-backed rate-limit service the gateway calls, and what happens when Redis is slow, are in design a rate limiter.

Transformation, translation and caching

Rewriting headers is cheap; rewriting bodies is not, because the gateway must hold the whole body first. Kong’s plugin docs note that a plugin modifying the full response switches the proxy into buffered mode, which doesn’t work for HTTP/2 or gRPC upstreams. Protocol translation is the cheaper win: Envoy’s gRPC-JSON transcoder lets JSON clients call gRPC services via HTTP mappings in the .proto file, and gRPC-Web lets browsers reach gRPC (see REST vs GraphQL vs gRPC).

Response caching saves a hop for popular reads; AWS REST API caches default to a 300 s TTL (0 to 3,600 s) and 1 MB per response. A cache keyed by path alone serves one user’s /me to the next caller, so key personal data by user or tenant.

What must stay out of the gateway

Many organisations learned this with enterprise service buses: a central layer that began as a router slowly collected transformation, orchestration and business rules, with every change queued behind one team. Gateways drift the same way, one reasonable request at a time (“just map this field for mobile”, “just check the coupon”), until you have a god gateway: your slowest-moving, most coupled, widest-blast-radius component.

Keep these out:

• Business rules such as coupon validity or pricing. Microsoft’s offloading pattern: never offload business logic to the gateway.
• Per-service data reshaping, which ties gateway releases to service releases, the coupling Microsoft says makes offloading a poor fit.
• Per-request state: sessions, carts, workflows. Stateless gateways scale and get replaced freely.
• Orchestrating writes across services. Read fan-out is borderline; put it in a client team’s backend-for-frontend (see API composition and BFFs).

The test: could the platform team run this gateway knowing nothing about food delivery? Ownership keeps the answer yes: the platform team owns the gateway and global policy, and each service team owns its routes as declarative config, checked by automation. An illustrative route, not any product’s syntax:

# routes/orders.yaml, owned by the orders team
route: /v2/orders/*
upstream: orders.internal:8443
auth: { jwt: required, scopes: [orders:read] }
rate_limit: { per: api_key, rate: 50/s, burst: 100 }
timeout: { per_try: 2s, total: 4.5s, retries: 1, retry_on: [connect-failure] }
max_body: 64KB

The pipeline rejects a route with no auth block unless it is marked public.

A request’s path through the filter chain

Inside, a gateway runs a chain of small steps for every request: Zuul and Envoy call them filters, and Kong plugins hook into phases (rewrite, access, header filter, body filter, log). One principle sets the order: the cheapest rejection runs first, so junk dies before it costs a network call.

Step by step:

1. TLS termination and connection limits. A failed handshake never becomes a request. Envoy’s edge guide warns its defaults suit a mesh, not the internet, and caps edge buffers at 32 KiB per connection. See HTTP and HTTPS.
2. IP filter, WAF and size cap. No identity needed and microseconds to run, so floods and oversized bodies (413) stop here, along with a coarse per-IP limit that shields step 4 from junk tokens.
3. Match route. The route decides which policies apply; no match is a 404.
4. Authenticate and authorize. Bad token: 401. Missing scope: 403. No network call with cached keys.
5. Rate limit and quota. Needs identity, so it follows step 4: one call to shared counters, about 0.6 ms at the median in-zone by the rate limiter design’s estimate, or 429 with Retry-After.
6. Validate and transform. Schema check (400), identity and trace headers, protocol translation.
7. Call upstream with a per-try timeout and a tightly limited retry (next section): 502, 503 or 504 on failure.
8. Response filters and logging run on the way back, for rejects too. Logging every 401 and 429 is how you see an attack.

What the hop costs

Kong publishes benchmarks per release. For version 3.16, on Kubernetes on AWS over HTTPS, a bare proxied route served about 144,000 requests per second at 2.9 ms p95 and 4.5 ms p99; with key authentication and rate limiting, about 102,000 requests per second at 4.4 ms p95 and 8.5 ms p99.

So a gateway hop costs a few milliseconds even under heavy load, and every policy costs capacity: (144,274 − 102,089) ÷ 144,274 ≈ 29% fewer requests per second for two plugins. Against a 300 ms p99 target, 8.5 ms is about 3%, paid on every call. The expensive steps leave the machine (introspection, external authorization, remote counters): cache them and give each a deadline.

Timeouts and retries: the gateway owns the outer budget

The rule for timeouts: budgets shrink as you go inward. The client waits longest, the gateway gives up before the client, each service before the gateway, each dependency before its caller. Better, pass the remaining time downstream as a deadline (gRPC does this natively) so inner calls stop once nobody is waiting.

Inverted, the gateway returns 504 while the work carries on: wasted capacity during the overload that caused the slowness, and for writes a hazard, because the user retries a payment that is about to succeed. Protect such endpoints with an idempotency key.

AWS REST APIs cap integration timeouts at 29 s by default, so slower work such as a report export should return 202 Accepted with a job ID; see long-running jobs.

Retries multiply across layers. The Amazon Builders’ Library works the example: five layers with three tries each put 3⁵ = 243× the load on a database that is already failing, so Amazon retries at a single point in the stack. At the gateway:

• Retry only idempotent methods or requests carrying an idempotency key.
• Retry at most once, on another instance, and only when the request clearly never ran (a refused connection).
• Cap retries with a budget, say a token bucket allowing retries on at most 10% of requests.

Count the client too: an SDK making 3 tries through a gateway making 2 turns one tap into 3 × 2 = 6 requests. See timeouts, retries and backoff with jitter and retry storm.

Deployment patterns

“The gateway” is often several. The three common shapes differ in who owns what.

Chris Richardson’s API gateway pattern cites the Netflix API gateway running client-specific adapter code per device, and names a gateway per client type as a variation. Large systems stack layers (CDN or global load balancer, edge gateway, BFFs or domain gateways, services), so decide which layer owns each check. Verifying a JWT twice is cheap; counting one request against two rate limits confuses everyone.

North–south vs east–west

Traffic entering from users and partners is north–south: it crosses the trust boundary carrying end-user identity, keys and quotas. Traffic between your own services is east–west, and the questions differ.

Products overlap (Envoy can play every role), but the jobs don’t:

A mesh’s ingress gateway can be your API gateway, but Microsoft notes such gateways are usually weaker at WAF, API products and transformation, so they are often paired with a dedicated one. See sidecar, ambassador and service mesh.

Managed or self-hosted

Managed: AWS API Gateway as the example

A managed gateway scales and patches itself; you pay per request and live within its quotas. AWS’s published quotas come up in interviews:

• Throttle per account, per Region: 10,000 requests per second across all your HTTP, REST and WebSocket APIs, with bursts absorbed by a token bucket of 5,000. Some newer Regions, including Asia Pacific (Hyderabad), start at 2,500 RPS and 1,250. The rate can be raised; the burst is set by AWS.
• Integration timeout: 50 ms to 29 s on REST APIs by default (raisable for Regional and private APIs, possibly for a lower throttle); 30 s maximum on HTTP APIs.
• Other REST API limits: 10 MB payload, 20,480 bytes of headers, 10,000 API keys per Region.

Work the numbers for a tatkal-style sale at 10:00 in Hyderabad: 50,000 users tapping “Book” within 10 seconds is 50,000 ÷ 10 = 5,000 RPS against a 2,500 RPS refill. The 1,250-token bucket drains at the 2,500/s difference, so it is empty after 1,250 ÷ 2,500 = 0.5 s, and from then on about half of all requests get 429. Raise the limit before launch and put a waiting room in front.

Self-hosted: Envoy, Kong, NGINX, APISIX

Envoy is assembled from filters (JWT, external authorization, rate limit, transcoding). Kong runs Lua plugins on NGINX (OpenResty), with plugin kits in Go, Python and JavaScript. Plain NGINX and Apache APISIX are other common choices.

Netflix’s Zuul 2: what an async gateway buys

Zuul is Netflix’s front door, built from filters. Zuul 1 used blocking servlets with a thread per connection; when a backend slowed or devices retried, threads piled up until nodes entered what Netflix called a death spiral. Zuul 2 moved to Netty’s event loop, roughly one thread per core, so a connection costs a file descriptor and a listener instead of a thread.

The results correct some async hype. The main win was connection scaling: persistent connections to devices, enabling push. The CPU-heavy cluster fronting Netflix’s API (metrics, logging, decryption, compression) gained no efficiency; the light logging cluster gained about 25% throughput with 25% less CPU; and the async code was much harder to debug. Choose async for connection counts, and keep filters cheap.

Going deeper: how gateways fail

The same bad config on every instance

Instances are stateless, so run N + 1 across three zones behind an L4 load balancer, size for peak, and drain connections on deploy, as Microsoft’s offloading guidance advises. But redundancy doesn’t help when every instance loads the same bad rule.

That is how Google’s API front door failed on 12 June 2025. Service Control, which checks policy and quota on Google Cloud API requests, had gained a quota-check code path two weeks earlier with no error handling and no feature flag. A policy with unintended blank fields was then written and, like all quota metadata, replicated globally within seconds; the new path hit a null pointer and the binary crash-looped in every region. The kill switch was out within about 40 minutes, but in us-central1 restarting tasks without randomized backoff overwhelmed the Spanner table they depended on, and recovery took about 2 hours 40 minutes. Cloudflare’s 18 November 2025 outage had the same shape: a generated bot-management file doubled in size, reached every machine, and broke a limit in the core proxy.

Google’s fixes are the playbook for any gateway:

• Ship config like code: validate it, roll it out region by region, keep a last-known-good.
• Put new code paths behind flags, off by default.
• Fail open on non-security checks such as quota lookups; never on authentication.
• Restart with randomized exponential backoff.

Auth caches, key rotation and revocation

Cached keys create two timing problems. Rotation: if the identity provider signs with a new key before gateways fetch it, valid tokens get 401. Publish new keys at least one cache period before using them, and re-fetch (rate-limited) on an unknown key ID. Revocation: a JWT stays valid until it expires, so keep access tokens short-lived and use introspection or a deny list only where instant logout matters.

Cached decisions are trickier. On AWS, a Lambda authorizer’s cached policy is keyed by the token and reused on every route until the TTL expires, which is why AWS says it must cover every method in the API. A narrow policy causes surprise 403s elsewhere; a wildcard allows routes nobody evaluated.

Noisy tenants

AWS’s 10,000 RPS account throttle is one bucket for every API in the account and Region, so a runaway reporting client can push login and payments into 429s. Set per-API and per-key throttles below the account cap, and isolate critical APIs (noisy neighbours and fairness).

Long-lived connections

WebSockets (real-time connections) pin a client to one gateway instance for hours. Envoy’s edge guide notes its 300-second stream idle and request timeouts must be disabled for streaming requests, and a restart drops every client on the instance at once, so drain slowly and reconnect with jitter.

AWS WebSocket APIs close connections after 2 hours or 10 idle minutes and accept 500 new connections per second per account per Region by default. With 1 million users, the 2-hour cap alone forces 1,000,000 ÷ 7,200 s ≈ 139 reconnects per second; after a full disconnect, readmitting everyone takes 1,000,000 ÷ 500 = 2,000 s, about 33 minutes.

Cheat sheet

How to say it in the interview

“I’d put an API gateway behind the regional load balancer as the single entry point for client traffic. It terminates TLS, validates the JWT against cached keys, enforces per-key rate limits and quotas against shared counters, caps body sizes and routes by path and version. It strips client-sent identity headers and sets its own, and services accept traffic only from the gateway, over mTLS. Business logic stays in services or a BFF. The gateway is stateless across three zones, holds the outermost timeout, retries only idempotent calls once, and its config ships like code, canaried by region. Service-to-service calls use the mesh, not the gateway.”

Isn’t the gateway a single point of failure? Its instances aren’t: they’re stateless, N + 1 across three zones. What fails everywhere at once is config, so it gets validation, staged rollout and instant rollback.

Why not validate the JWT in every service instead? Sensitive services can, as cheap defence in depth. The gateway is still where bad traffic dies before it costs anything, and where per-caller limits and one consistent policy live.

Gateway or service mesh? Both: the gateway for north–south users, keys and quotas; the mesh for east–west service identity, retries and per-hop metrics. A mesh ingress gateway can double as the API gateway if you need no API products, heavy transformation or WAF.

Where does the rate-limit state live? Not in gateway memory: across 50 evenly loaded pods, local counters could let a client through up to 50 times its limit. Counters live in Redis behind a small rate-limit service, with a millisecond-scale deadline and fail-open for most rules.

How do you stop it becoming a monolith? Keep it to policy that is the same for every service, give teams self-service route config with automated checks, and push client-specific shaping into BFFs. If a service release needs a gateway release, logic has leaked.

Where to go next

• Proxy vs reverse proxy: the building block under every gateway, and forwarded headers.
• Load balancing: L4 vs L7: what sits in front of the gateway.
• Rate limiting and design a rate limiter: the algorithms, and the service the gateway calls.
• API composition, BFF and GraphQL federation: where aggregation belongs.
• Sidecar, ambassador and service mesh: the east–west half of the picture.
• Authentication, sessions, tokens and JWT: what the gateway verifies.
• Timeouts, retries and backoff with jitter and idempotency: making retries safe.

References and further reading

1. Amazon API Gateway quotasdocs.aws.amazon.comThe account-level throttle: 10,000 requests per second with a 5,000-request token bucket per account per Region, and 2,500 and 1,250 in newer Regions such as Hyderabad.
2. Quotas for configuring and running a REST API in API Gatewaydocs.aws.amazon.comThe 50 ms to 29 s integration timeout, the 10 MB payload and 20,480-byte header caps, cache TTL and cached-response size, and 10,000 API keys per Region.
3. Usage plans and API keys for REST APIs in API Gatewaydocs.aws.amazon.comThrottles versus quotas per API key, best-effort enforcement, and why API keys must not be used for authentication or authorization.
4. Zuul 2: The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix Tech Blog, 2016)netflixtechblog.comZuul as Netflix’s front door, filters, the blocking-gateway death spiral, and measured results: connection scaling, 25% more throughput on a light cluster, no gain on the CPU-heavy API cluster.
5. Gateway Offloading pattern (Azure Architecture Center)learn.microsoft.comWhat to offload, never business logic, preventing gateway bypass, not trusting unverified forwarded headers, and when the pattern does not fit.
6. API gateways (Azure Architecture Center, microservices guide)learn.microsoft.comThe costs of direct client-to-service calls, gateway routing, aggregation and offloading, and mesh ingress gateways compared with dedicated API gateways.
7. Envoy documentation: JWT Authentication filterenvoyproxy.ioRemote JWKS with a cache duration, claims copied into headers, and first-match rules where a request matching no rule needs no JWT.
8. Kong Gateway performance testing benchmarksdeveloper.konghq.comPublished throughput and p95 and p99 latency for a bare proxy route versus key authentication plus rate limiting.
9. Google Cloud incident report: Service Control outage, 12 June 2025status.cloud.google.comA globally replicated quota policy with blank fields crashed the API policy-check binary in every region; the timeline, the restart herd and the fixes.
10. Timeouts, retries and backoff with jitter (Amazon Builders’ Library)aws.amazon.comWhy retries at every layer multiply load 243× across a five-deep stack, and the practice of retrying at a single point.