Busy front end

A busy front end is a web or API tier that does heavy work inside the same request handlers that serve everyone else: resizing photos, rendering PDF statements, parsing a 50 MB bulk import, running a model, or sending ten thousand emails. That work shares cores, threads, memory and the event loop with the cheap requests, so a login that needs 2 ms of CPU waits behind a resize that needs 400. The fix is to keep the request path for work the user must wait for, and hand everything else to a queue and a worker tier built for it.

Picture it first: the counter and the travel agent

A railway reservation counter serves one person at a time, and most need a minute. At 10 am a travel agent arrives with 40 group-booking forms for a wedding party. The clerk works through them at the window for 40 minutes, and the 30 people behind, each wanting one ticket, wait 40 minutes too.

The station’s fix is a back office. The agent drops the forms in a tray, takes a numbered token, and comes back later or waits for an SMS. The windows go back to one-minute jobs, and the back office works through the tray with as many clerks as the pile needs.

Map it back:

• The window clerk is a web worker: a thread, a process, or the event loop.
• The one-ticket passengers are cheap requests: login, browse, health checks.
• The group forms are heavy work: photo resizing, PDF statements, bulk imports.
• The tray is a queue, the token is a 202 Accepted with a status URL, the SMS is a webhook or push notification, and the back office is a worker tier.

Where the time goes: three budgets

Every request spends from budgets the whole pod shares: CPU, request slots and memory. Heavy work spends far more of each, and when any one runs out, every route on the pod pays.

CPU: seconds of work per second

Take a Swiggy-style platform where the customer app and the restaurant partner app share one API deployment of 8-vCPU pods. Browsing costs 2 ms of CPU per request, at 1,500 a second per pod. Around 11 am restaurants refresh their menus, and each dish photo upload decodes a 12-megapixel image and writes three WebP sizes: 400 ms of CPU in a profile.

browse only:       1,500 req/s × 2 ms   =  3 CPU-s/s  →  38% of 8 vCPUs
+ 10 uploads/s:       10 × 400 ms       =  4 CPU-s/s  →  7 in total, 88%
+ 20 uploads/s:       20 × 400 ms       =  8 CPU-s/s  → 11 in total, 138% ✗
pod saturates at:  (8 − 3) ÷ 0.4 s      = 12.5 uploads/s

Twenty uploads a second fill the pod on their own, and 12.5 tip it over at today’s browse traffic, though uploads are only about 1% of requests.

Cheap requests slow down well before 100%, because the operating system time-slices runnable threads across the cores. A rough processor-sharing estimate is time ≈ CPU needed ÷ (1 − utilisation): the 2 ms browse request takes about 3.2 ms at 38%, 16 ms at 88% and 80 ms at 97.5%; above 100% the run queue grows until clients time out. The busy database article draws that curve.

Memory comes along too: a 12-megapixel photo decoded to RGB is 12,000,000 × 3 bytes = 36 MB before resizing starts.

Threads and workers: Little’s law

Most servers have a fixed number of request slots: a thread pool, pre-forked worker processes that each handle one request at a time, or a connection cap. Little’s law says how many are busy:

slots in use = arrival rate × time each request holds a slot

Browse at 1,500 requests a second, taking 10 ms including its database call, holds 1,500 × 0.010 = 15 threads. Let the CPU saturate so it takes 150 ms, and it holds 1,500 × 0.15 = 225, more than a 200-thread pool. A slow CPU becomes pool exhaustion, and new requests wait in the accept queue.

Long work gets there without high rates. A “download all invoices” export that runs for 30 seconds holds its slot for all 30. A 200-thread pool fills at 200 ÷ 30 ≈ 6.7 exports a second, easy for a GST-filing app in the last week of the month. If the exports are CPU-bound the cores give out sooner; either way, login has nowhere to run.

Memory: bytes × concurrency

The same law sizes memory: bytes held = arrival rate × duration × bytes per request. An export that builds a 400 MB spreadsheet in memory over 30 seconds, started once every 3 seconds, keeps 0.33 × 30 = 10 in flight: 4 GB. On a pod with a 4 GB memory limit, the runtime’s own baseline tips it over, the process is killed, and every request on it fails, not just the exports.

The event loop: a pool of one

Node.js runs all your JavaScript on one thread, the event loop, plus a small worker pool for file system, DNS, zlib and some crypto calls. The Node.js guide puts it simply: Node is fast when the work for each client at any moment is small. While a callback runs, no other client gets a turn.

A bulk-import endpoint that calls JSON.parse on a 50 MB catalogue is enough: the guide measured about 1.3 s to parse a 50 MB JSON string. For those 1.3 s the process serves nobody, not even the health check. zlib.deflateSync, crypto.pbkdf2Sync and other *Sync calls do the same, and Python’s asyncio servers share the shape: CPU work in a coroutine stalls every request on that process.

Symptoms you’d see

The signature is that cheap routes get slow in step with one expensive route, while total traffic barely moves.

• Unrelated endpoints slow down when the heavy one is busy. In Azure’s load test, a cheap user-profile GET served about 500 requests a second alone and fell to about 150 once 100 users started posting to a CPU-heavy endpoint on the same servers.
• Web-tier CPU spikes that track a route, not traffic. Requests per second are flat, CPU is at 100%, and the spikes line up with uploads, exports or the 9 am import window.
• Request queue time climbs. Requests wait longer between the load balancer and your handler; thread pools sit at their maximum and the server’s HTTP queue grows.
• 502s, 504s and client timeouts, mostly on routes that did nothing wrong.
• Health checks fail under load, so the orchestrator restarts pods that weren’t broken and the survivors inherit their load, as in Figure 2. See health checks done right.
• Autoscaling helps briefly. New pods get their share of heavy requests and fill up too; Azure’s guidance says plainly that scaling the front end is not the fix.
• Event-loop lag in Node.js: timers fire hundreds of milliseconds late, and p99 jumps for every route at once.
• Connections held for minutes, which stall deploys and get cut by load balancer idle timeouts, so the user’s retry starts the work again.
• Out-of-memory kills that coincide with exports or large uploads, and dependencies throttling with 429s as the heavy route hammers storage or the database.

Why it happens

Nobody sets out to build this. It accumulates from reasonable shortcuts.

The controller is the easiest place for code

The upload arrives in a handler, and the image library is one import away. Resizing inline is ten lines and works perfectly on a laptop. A queue, a worker deployment and a status endpoint feel like ceremony for a minimum viable product.

“It’s only for admins” and “it’s only a few a day”

The bulk import is for internal ops, the PDF statement is a monthly button. Then a Flipkart-style sale starts, 5,000 sellers upload catalogues at 9 am, and “a few a day” is 20 a second.

One process, no line between “must wait” and “can wait”

Many frameworks deploy one process type for every route, and Azure notes the pattern is typical of monoliths where business logic shares a tier with the presentation layer. A user tapping “Publish menu” must learn that the photo was accepted, not wait for three resized variants; when nobody draws that line, everything lands on the synchronous path.

Serverless doesn’t remove the wait

Moving the heavy handler into a function behind an API gateway still keeps the client waiting on one HTTP call. Amazon API Gateway REST APIs have a 29-second default integration timeout and a 10 MB payload limit, while a Lambda function can run for up to 15 minutes. A 90-second report returns a 504 at 29 seconds, the function keeps going, and the user’s retry starts a second copy.

Inputs grow

A CSV import tested with 1,000 rows meets a seller with 400,000. The Node.js guide adds a sharper case: some regular expressions take exponential time on crafted input, so a malicious request can block the loop on purpose.

How to detect it

You’re looking for one route whose cost lands on everyone else. Measure cost per route, then prove the link.

Rank routes by CPU, not by latency

Most APM tools and continuous profilers report CPU time per route. Rank by total CPU = calls per second × CPU per call, not by p99: a route with 0.5% of the traffic using 60% of web-tier CPU is your suspect even if its own latency looks fine. Note its duration and memory per call too.

Measure queue time and pool saturation

Time spent waiting for a slot never shows in your handler’s timer. Stamp the time in a header at the load balancer or proxy and subtract it when the handler starts; many APM agents call this request queue time. Graph it next to active threads or workers against the maximum. A busy front end pins the pool during heavy-route bursts.

Correlate the heavy route with a cheap one

Put a cheap route’s p99 and the heavy route’s rate on one panel. With OpenTelemetry HTTP metrics exported to Prometheus:

# Panel 1: p99 of a cheap route
histogram_quantile(0.99,
  sum by (le) (rate(http_server_request_duration_seconds_bucket{http_route="/login"}[5m])))

# Panel 2, same time axis: requests per second to the heavy route
sum(rate(http_server_request_duration_seconds_count{http_route="/menu-items/{id}/photo"}[5m]))

If login’s p99 rises and falls with upload traffic while login traffic is flat, the two share a resource they shouldn’t.

Watch the event loop

In Node.js, measure how late the loop runs, not just CPU. monitorEventLoopDelay samples the delay into a histogram in nanoseconds, and eventLoopUtilization reports the share of time the loop was busy:

import { monitorEventLoopDelay, performance } from 'node:perf_hooks';

const RES_MS = 10;
const delay = monitorEventLoopDelay({ resolution: RES_MS }); // timer-based sampling
delay.enable();
let last = performance.eventLoopUtilization();

setInterval(() => {
  const now = performance.eventLoopUtilization();
  const elu = performance.eventLoopUtilization(now, last); // busy share since last tick
  last = now;
  // each timer-based sample includes the 10 ms interval itself, so subtract it
  metrics.gauge('eventloop.lag.p99_ms', delay.percentile(99) / 1e6 - RES_MS);
  metrics.gauge('eventloop.lag.max_ms', delay.max / 1e6 - RES_MS);
  metrics.gauge('eventloop.utilization', elu.utilization);
  delay.reset();
}, 10_000);

A healthy API server shows a lag of a few milliseconds and utilisation well below 1. A max near 1,300 ms right after a bulk import is Figure 3 on your dashboard.

Profile under load

A CPU flame graph taken during the slowdown names the culprit: image codecs, PDF or headless-browser renderers, spreadsheet writers, zlib, JSON parsing or model runtimes, sitting under your request handlers. If the wide frames are socket reads or lock waits instead, it’s a different problem.

Prove it with a step load test

Azure’s method: run a steady load of the cheap route, add a stepped load of the suspect route, and watch the cheap route’s throughput and p99. Repeat without the suspect route. If the cheap route recovers, you’ve found it.

Telling it apart from look-alikes

How to fix it

The principle: the request path does only what the user must wait for. Everything else is accepted quickly, recorded durably, and done by a tier sized and scaled for that work.

Before: the handler does the work

# BEFORE: the web worker decodes and resizes a 12 MP photo itself.
# storage, jobs and queue are thin wrappers over your object store, database and broker.
from io import BytesIO
from flask import Flask, request, jsonify
from PIL import Image

app = Flask(__name__)
SIZES = {"thumb": 200, "card": 600, "full": 1600}

def write_sizes(original, item_id):
    for name, width in SIZES.items():
        img = original.copy()
        img.thumbnail((width, width))              # resample: CPU-heavy
        buf = BytesIO()
        img.save(buf, "WEBP", quality=80)          # encode: CPU-heavy
        storage.put(f"menu/{item_id}/{name}.webp", buf.getvalue())

@app.post("/menu-items/<item_id>/photo")
def upload_photo(item_id):
    original = Image.open(request.files["photo"].stream)  # decodes ~36 MB of pixels
    write_sizes(original, item_id)                         # ~400 ms of CPU, here
    return jsonify(status="done"), 201

After: validate, enqueue, answer 202

This is the Queue-Based Load Leveling pattern: a queue sits between a bursty producer and a consumer that works at its own pace.

# AFTER: the handler stores bytes and enqueues; it never decodes the image.
MAX_BYTES = 20_000_000

@app.post("/menu-items/<item_id>/photo")
def upload_photo(item_id):
    photo = request.files["photo"]
    if (request.content_length or 0) > MAX_BYTES:
        return jsonify(error="photo must be under 20 MB"), 413
    if photo.mimetype not in {"image/jpeg", "image/png"}:
        return jsonify(error="send a JPEG or PNG"), 415
    key = request.headers.get("Idempotency-Key")
    job = jobs.find(current_user.id, key) if key else None
    if job is None:                                    # first time we see this upload
        job = jobs.create(current_user.id, key, item_id=item_id, status="queued")
        storage.put(f"originals/{job.id}", photo.stream)   # streams bytes, no decoding
        queue.send({"job_id": job.id})
    return "", 202, {"Location": f"/jobs/{job.id}", "Retry-After": "2"}

@app.get("/jobs/<job_id>")
def job_status(job_id):
    job = jobs.get(current_user.id, job_id)
    if job is None:
        return jsonify(error="no such job"), 404
    if job.status == "succeeded":
        return "", 303, {"Location": f"/menu-items/{job.item_id}/photo"}
    return jsonify(status=job.status, updated_at=job.updated_at.isoformat()), 200

# worker.py: its own deployment, scaled on queue backlog, serves no HTTP.
def handle(msg):
    job = jobs.get_by_id(msg["job_id"])
    if job.status == "succeeded":          # at-least-once delivery: skip duplicates
        return
    original = Image.open(BytesIO(storage.get(f"originals/{job.id}")))
    write_sizes(original, job.item_id)     # the same CPU work, on a worker
    jobs.update(job.id, status="succeeded")

for msg in queue.receive(visibility_timeout=120):
    try:
        handle(msg.body)
        msg.ack()
    except Exception:
        if msg.delivery_count >= 5:        # poison message: park it, alert a human
            dlq.send(msg.body)
            msg.ack()
        # otherwise don't ack: the queue redelivers after the timeout

The web pod now spends a millisecond or so of CPU per upload on validation and bookkeeping, plus I/O to stream the bytes. A burst of 2,000 uploads becomes a backlog that workers drain, not 2,000 requests fighting login for eight cores.

You pay with eventual completion: the photo appears seconds later, so the UI needs a “processing” state. You run a queue and a worker deployment, workers must be idempotent because delivery is at-least-once, and messages that always fail need a dead-letter queue with an alarm. A crash between creating the job row and sending the message strands the job; a transactional outbox closes that gap. And debugging crosses a queue, so carry a request ID in every message.

Tell the client where to look: asynchronous request-reply

202 Accepted is HTTP for “got it, not done”. RFC 9110 calls it intentionally noncommittal: the work might still be refused, and the response should point to a status monitor. The Asynchronous Request-Reply pattern fills in the details:

POST /reports/gst-summary HTTP/1.1
Content-Type: application/json
Idempotency-Key: 3f6c0e1a-7b2d-4c55-9a7e-2f1d8b6c4e90

{"gstin": "29ABCDE1234F1Z5", "month": "2026-09"}

HTTP/1.1 202 Accepted
Location: /jobs/81f3
Retry-After: 10

GET /jobs/81f3 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json

{"status": "running", "percentComplete": 40}

GET /jobs/81f3 HTTP/1.1

HTTP/1.1 303 See Other
Location: /reports/81f3.pdf

• Validate before you accept: a bad request gets a 400 now, not a failed job later.
• Location and Retry-After tell the client where to poll and how often.
• The status resource answers 200 while work is pending, with a state such as queued, running or failed, then 303 See Other to the result. A 303 makes the client fetch with GET; a 302 doesn’t guarantee that, and a client that replays the POST duplicates the job.
• Failures live at the status URL as structured errors; offer DELETE to cancel, and expire old status records.

Polling isn’t the only way back: a server-to-server client can register a webhook, a browser can listen on a WebSocket or server-sent events, and a phone can get a push notification. Each removes polling load and adds a channel to run.

Keep big uploads off the web tier: presigned URLs

Even streaming bytes holds a web connection and buffers for as long as a slow phone takes to send 12 MB. Let the client upload straight to object storage instead; the API only signs a time-limited URL:

import uuid

@app.post("/menu-items/<item_id>/photo-upload-url")
def photo_upload_url(item_id):
    key = f"originals/{item_id}/{uuid.uuid4()}.jpg"
    url = s3.generate_presigned_url(
        "put_object",
        Params={"Bucket": UPLOAD_BUCKET, "Key": key, "ContentType": "image/jpeg"},
        ExpiresIn=300,                     # valid for five minutes
    )
    return jsonify(upload_url=url, key=key), 200

The phone sends a PUT to that URL, and a bucket notification puts a “new object” event on the queue that feeds the workers. The S3 guide covers the mechanics; Azure calls the general idea the valet key pattern.

You pay with validation after the fact: the worker must check type, size and pixel count, because the client chose the bytes. Abandoned uploads leave orphan objects that need a lifecycle rule, and a leaked URL works until it expires, so keep expiry short. Large file uploads covers multipart and resumable uploads.

Size the worker tier on backlog, not CPU

Scale workers on backlog per worker against the delay you promise; autoscaling done right has the formula. The arrival rate gives the floor: 20 uploads a second × 0.4 s keeps 8 vCPUs busy, so at a 70% target you need 8 ÷ 0.7 ≈ 12 vCPUs of workers.

Queues hide trouble while they grow, so watch message age, not just depth. Amazon’s Builders’ Library article on queue backlogs spells out the recovery arithmetic: after an hour-long outage you need double your normal capacity for another hour to catch up, and a spike that queues 10 times what consumers are scaled for, noticed after 30 minutes, takes 300 minutes to drain.

• Separate queues by urgency. A seller watching a spinner for one thumbnail shouldn’t wait behind a bulk re-render of 2 million images; give interactive jobs their own queue and workers.
• Bound consumer concurrency. Azure warns that autoscaling consumers without limiting their combined rate just moves the overload to the database or storage behind them.
• Shed at the door when the backlog is hopeless. Return 503 with Retry-After rather than accept work you’ll finish too late; see backpressure and load shedding.

Bulkhead what must stay synchronous

Sometimes the user really must wait: a price quote from a model, a preview render. Give that route its own compartment: a separate deployment behind path-based routing, or at least its own small pool or semaphore. In Node.js, run the CPU work in worker_threads sized to the cores. The heavy route can then exhaust only its own capacity. You pay with reserved capacity and routing config, and those users still wait; see bulkheads.

Functions for bursty jobs

Event-driven serverless functions suit bursty, independent jobs: one invocation per uploaded object, scaling with the burst and idle at 3 am. Trigger them from the queue or bucket event, never from the user’s synchronous request. You pay with the 15-minute Lambda cap, cold starts, a per-invocation bill that overtakes always-on workers at steady volume, and a fleet that can scale faster than your database.

Push it to the edge or the client

Image CDNs resize and re-encode on request from URL parameters and cache each variant at the edge, and the app can shrink a 12 MP photo to 1,600 pixels before uploading. You pay a per-transformation bill, and you still validate everything the client sends.

Compare the fixes

Verify the fix

Rerun the step load test: steady browse plus a rising upload rate. Login and browse p99 should stay flat, web CPU should track cheap traffic only, and the new curve to watch is queue age against your promise, say thumbnails within 10 seconds. If queue age rises while workers idle, look for a poison message or a throttled dependency.

When it’s fine

• Small, bounded CPU work that is the response. Serialising JSON, rendering a template, verifying a token or totalling a cart usually costs well under 10 ms. Cap input sizes so “small” stays small.
• Work the user must wait for, with its own limits. Login must hash the password, deliberately slowly, and a UPI-style payment must verify signatures. Keep it synchronous but bounded: async APIs (Node’s crypto.scrypt runs on the worker pool; scryptSync blocks the loop), a concurrency limit, and capacity for the peak.
• Low-traffic internal tools, such as an admin report run ten times a day on its own deployment.
• Early products, with a tripwire. Inline resizing at 50 uploads a day is fine if you alert on per-route CPU and know where the queue will go.

The test: at peak, the heavy route’s CPU, slots and memory fit in the pod’s headroom, and a load test of that route leaves cheap routes’ p99 flat.

Cheat sheet

How it shows up in interviews

It hides in any prompt where users hand you something heavy: photo sharing (who makes the thumbnails?), YouTube-style upload and transcoding, Dropbox-style sync, UPI statement PDFs, seller catalogue imports during a sale, a notification system sending a million pushes, and LLM inference.

The probes: “Where does the resizing happen?” “Five thousand sellers upload catalogues at 9 am; what happens to checkout latency?” “The report takes 90 seconds; what does the API return?” Answer with the budget arithmetic, then the queue, the 202 contract and how workers scale.

How to say it in the interview

“I keep the request path for what the user must wait for. Upload, report and import endpoints validate, store the input, enqueue a job and return 202 with a status URL, deduplicated by an idempotency key. A separate worker tier, scaled on queue backlog and age, does the CPU work; the status endpoint answers 303 to the result, or we push a notification. Photos go straight to object storage with presigned URLs. Otherwise 20 uploads a second at 400 ms of CPU is a whole 8-core pod, and login waits behind it. The price is eventual completion, idempotent workers for at-least-once delivery, a dead-letter queue, and more moving parts.”

Why not just add more web servers? Each new pod gets its share of heavy requests and fills the same way, so you pay for heavy-route peaks on every pod and cheap-route latency still depends on the upload mix. Azure’s guidance says this shouldn’t be fixed by scaling the front end.

Isn’t async or a background thread enough? No. Async frees a thread while it waits for I/O; it doesn’t make computation cheaper. A background thread returns the response sooner, but the CPU work still runs on the same pod, now with no limit on how many run at once.

What if the user taps upload twice, or the worker crashes halfway? The idempotency key makes a repeat return the same job. The queue redelivers unacknowledged messages, so workers check job state and write results under deterministic keys; messages that keep failing go to a dead-letter queue with an alarm.

How do you size the worker tier? From arrival rate × CPU per job: 20 uploads a second × 0.4 s keeps 8 vCPUs busy, so about 12 at a 70% target. Autoscale on backlog per worker against the promised delay, alarm on message age, and cap concurrency so workers can’t flood the database.

Where to go next

• Synchronous I/O: the look-alike where threads wait instead of compute.
• Busy database: the same mistake on the tier that’s hardest to scale.
• Noisy neighbour: when the heavy work belongs to someone else.
• Message queues: the buffer between the web tier and the workers.
• Long-running jobs: status tracking, progress and cancellation in depth.
• Large file uploads: presigned, multipart and resumable uploads.
• Idempotency: making repeated submissions and redeliveries safe.
• Bulkheads: isolating heavy routes that must stay synchronous.
• Synchronous vs asynchronous communication: the broader trade-off behind the 202.

References and further reading

1. Busy Front End antipattern (Azure Architecture Center)learn.microsoft.comWhy background threads in the web process still starve requests, the load test where a cheap GET fell from about 500 to 150 requests per second, and why scaling the front end is not the fix.
2. Queue-Based Load Leveling pattern (Azure Architecture Center)learn.microsoft.comA queue as a buffer between bursty producers and a steady consumer, plus at-least-once delivery, dead-letter queues and bounding consumers so autoscaling doesn’t just move the overload.
3. Asynchronous Request-Reply pattern (Azure Architecture Center)learn.microsoft.com202 Accepted with Location and Retry-After, a status endpoint that redirects with 303, idempotency keys, cancellation and retention of status resources.
4. RFC 9110: HTTP Semantics, sections 15.3.3 (202 Accepted) and 15.4.4 (303 See Other)rfc-editor.org202 is deliberately noncommittal and should point to a status monitor; 303 sends the client to a different resource with a GET.
5. Don’t Block the Event Loop (or the Worker Pool) (Node.js documentation)nodejs.orgOne event loop plus a small worker pool, synchronous APIs to avoid in servers, the 50 MB JSON example (about 1.3 s to parse), partitioning versus offloading.
6. Node.js documentation: Performance measurement APIs (perf_hooks)nodejs.orgmonitorEventLoopDelay (a histogram in nanoseconds, sampled every 10 ms by default) and eventLoopUtilization.
7. Avoiding insurmountable queue backlogs (Amazon Builders’ Library)aws.amazon.comWhy backlogs make queues bimodal, the recovery arithmetic after an outage, measuring message age, and separate or spillover queues per workload.
8. Amazon S3 User Guide: Uploading objects with presigned URLsdocs.aws.amazon.comLetting a client upload straight to a bucket with a time-limited signed PUT, without AWS credentials.
9. Amazon API Gateway quotas for REST APIsdocs.aws.amazon.comThe 29-second default integration timeout and the 10 MB payload limit.
10. AWS Lambda Developer Guide: Configure function timeoutdocs.aws.amazon.comFunction timeout defaults to 3 seconds and can be raised to 900 seconds (15 minutes).