Synchronous I/O

Synchronous I/O is a request thread, or an event loop, sitting blocked while a database query, HTTP call, file read or DNS lookup completes. The thread does nothing useful while it waits, but nobody else can use it either, so a server runs out of threads long before it runs out of CPU. Throughput is capped at pool size ÷ latency, and when one dependency slows down, every thread ends up waiting on it and unrelated pages fail too.

Picture it first: the waiter who stands at the kitchen

A busy restaurant has 10 waiters. Each takes an order, walks it to the kitchen, then stands at the counter until the dish is ready, about 12 minutes, before carrying it out. Taking the order and serving take a minute; the other 11 are spent standing still.

Ten waiters tied up for 12 minutes per order can handle at most 10 × 60 ÷ 12 = 50 orders an hour, though they spend over 90% of their time doing nothing. On Saturday the tandoor slows and dishes take 30 minutes. The same 10 waiters now manage 10 × 60 ÷ 30 = 20 orders an hour, and the family at table 4 who only want the bill can’t get anyone’s attention: every waiter is at the kitchen.

The manager can hire more waiters. That helps for a while, but wages climb, the floor gets crowded, and the tandoor cooks no faster. Or the manager can change the rule: leave the ticket at the kitchen, go back to the tables, and return when the kitchen rings a bell. One waiter can now have a dozen orders cooking at once.

Map it back:

• Waiters are threads in a thread-per-request server, or the single event loop in Node.js.
• Standing at the counter is a blocking call: a JDBC query, requests.get, fs.readFileSync, .Result on a task.
• The tandoor is the dependency: a database, a payment API, a blob store.
• The bill-only table is an unrelated endpoint, such as the cart or the health check, that fails only because no thread is free.
• The bell is non-blocking I/O: the operating system says the reply has arrived, and any thread picks the request up again.

Where the threads go: Little’s law

Little’s law relates three numbers for any system in a steady state:

requests in flight = arrival rate × time each request takes

In a thread-per-request server, every request in flight holds a thread from its first byte to its last, including all the time it spends waiting. The pool is a hard ceiling on concurrency, which makes it a ceiling on throughput: max throughput = threads ÷ latency.

Take a Swiggy-style checkout service on Tomcat, whose HTTP connector defaults to 200 request threads, running on an 8-vCPU pod. Each checkout uses 2 ms of CPU and waits 48 ms: 8 ms for its database and 40 ms for the payment provider.

normal day:   1,000 req/s × 0.050 s   =    50 threads busy (of 200)
              1,000 req/s × 2 ms CPU  =     2 CPU-s/s  → 25% of 8 vCPUs
pool ceiling:   200 threads ÷ 0.050 s = 4,000 req/s
CPU ceiling:    8 vCPUs ÷ 2 ms        = 4,000 req/s

On a normal day the pool and the CPU run out at the same point. Then, during a sale, the payment provider’s latency goes from 40 ms to about 1 s. Your code hasn’t changed and each request still needs 2 ms of CPU:

pool ceiling:   200 threads ÷ ~1 s    =   200 req/s
offered load:                           1,000 req/s
backlog:        1,000 − 200           =   800 more requests queued every second
CPU:            200 req/s × 2 ms      =   0.4 CPU-s/s → 5% of 8 vCPUs

Throughput has fallen 5× and CPU has fallen with it. After ten seconds 8,000 requests are queued, and the newest waits 8,000 ÷ 200 = 40 s for a thread, long after its client has given up. The cliff comes early: the pool fills once latency passes 200 threads ÷ 1,000 req/s = 200 ms. A provider that slows from 40 ms to 190 ms isn’t down by anyone’s definition, yet it saturates you.

Why not just add threads?

Serving 1,000 req/s at 1 s each needs 1,000 threads; a sale at 4,000 req/s needs 4,000. Threads aren’t free. The JVM reserves a stack for every platform thread, 1,024 KB by default on Linux/x64 and 2,048 KB on Linux/AArch64, so 10,000 threads reserve about 10 GB of address space on x64 and 20 GB on Arm. Only the pages a thread touches become resident, but deep framework stacks touch plenty, and every thread is a kernel task to schedule. JEP 444 puts it plainly: when each request holds an OS thread, the thread count becomes the limit long before CPU or network connections do.

More threads also push more concurrent calls at a dependency that is already slow. That comes back in the fixes.

The event loop: a pool of one

Node.js, Python’s asyncio and Netty-style servers flip the model: a few threads, often one, run every request’s CPU work, and waiting costs no thread at all. That is why they scale for I/O-heavy work, and why one blocking call hurts so much there: block the loop for 200 ms and every request on that process waits 200 ms. Node also runs file system calls and dns.lookup() on libuv’s worker pool, 4 threads by default, so four reads from a slow network file system stall every other file read.

Symptoms you’d see

The signature is latency up, throughput down and CPU down, all at once. A busy front end starves with the cores pinned; synchronous I/O starves with the cores idle.

• Low CPU, high latency, low throughput. The pod sits at 5–20% CPU while p99 climbs and requests per second fall. Autoscalers keyed on CPU see nothing to do.
• The pool is pinned at its maximum and the request queue grows. Azure’s guidance describes requests waiting for a free thread, long request queues, timeouts and HTTP 503s.
• A cliff, not a slope. All is well until dependency latency × request rate exceeds the pool, then latency jumps by seconds. A dependency that slows 5× can take you down.
• Unrelated endpoints fail together. The cart page needs no payment call but has no thread to run on. Health checks time out, the load balancer pulls the pod, and its traffic lands on pods that are already full (Figure 3).
• Thread dumps full of identical stacks: hundreds of threads in the same socket read, or queued for a database connection.
• .NET thread count creeping up. In Microsoft’s starvation tutorial, the pool grows fast to 2–3× the core count, then adds 1–2 threads a second while CPU stays well under 100%.
• Event-loop lag in Node.js or asyncio: timers fire late and every route’s p99 jumps at once after someone adds fs.readFileSync or a blocking driver to a handler.
• Memory grows with the thread count, and stack size becomes the reason you can’t raise the pool further.
• Callers time out and retry. Each retry holds another thread on the same slow call.

Why it happens

Blocking is the default

The first tutorial for most stacks uses blocking calls: JDBC, requests, File.ReadAllText, a synchronous SDK. The code reads top to bottom, stack traces make sense, and at low traffic it’s fine. Azure lists the usual reasons: it’s the most intuitive way to write the code, the caller needs the answer anyway, and the library only offers synchronous methods.

The block hides in a library

You can write careful async code and still block: a logging appender writes synchronously to a remote collector, an ORM lazy-loads a relation, an SDK fetches a credential on first use, a client resolves DNS on the calling thread. Azure points out that one synchronous call deep in a library blocks the whole call chain above it.

Sync-over-async and async-over-sync

In .NET, calling .Result or .Wait() on a task parks a thread-pool thread until the task finishes, and finishing the task may itself need a pool thread. Microsoft names sync-over-async as the most common cause of thread-pool starvation. In its tutorial, 125 concurrent clients against a .Result endpoint averaged 3.48 s and 33 requests a second; the await version averaged 533 ms and 228 a second, close to what Little’s law predicts: 125 ÷ 0.533 s ≈ 235.

The reverse, wrapping a blocking call in Task.Run to make it look async, just moves the block onto another pool thread; Azure warns it consumes more resources, not fewer.

Blocking calls inside async frameworks

This version is the sneakiest because the code looks asynchronous:

• A FastAPI async def handler that calls requests.get or a synchronous database driver runs on the event loop, so the whole process waits. Python’s asyncio guide says blocking code should not be called directly; a one-second call delays every task by one second.
• JDBC called on a Netty or Spring WebFlux event-loop thread, of which there are only a few per process.
• fs.readFileSync, child_process.execSync or zlib.inflateSync in an Express handler. The Node.js guide says not to use these synchronous APIs in a server.

No timeouts, or very long ones

A blocked thread is released only when the call returns. Python’s requests doesn’t time out unless you set a timeout, .NET’s HttpClient waits 100 seconds by default, and HikariCP lets a thread wait 30 seconds for a database connection. A hung dependency holds threads for as long as you let it.

Load tests with a fast fake

Load tests usually run against a mock that answers in 5 ms, so the cliff never shows. Production finds it on the first sale day the payment provider slows down. A new connection or client per request, improper instantiation, makes every wait longer still by adding handshakes.

How to detect it

Put CPU, latency and pool use on one panel

The tell is the scissors in Figure 4: busy threads rise to the limit while CPU falls. Graph them together:

• JVM with Micrometer: tomcat.threads.busy against tomcat.threads.config.max, plus hikaricp.connections.pending for threads queued on the database pool.
• .NET: dotnet.thread_pool.thread.count, dotnet.thread_pool.queue.length and dotnet.thread_pool.work_item.count from dotnet-counters.
• Node.js: event-loop delay from monitorEventLoopDelay in perf_hooks.
• Little’s law check: busy threads ÷ throughput = how long each request holds a thread. Compare it with CPU time per request.

# Share of the Tomcat pool in use (Micrometer names in Prometheus)
sum(tomcat_threads_busy_threads) / sum(tomcat_threads_config_max_threads)

# Seconds each request holds a thread; ≫ CPU per request means it's waiting
sum(tomcat_threads_busy_threads) / sum(rate(http_server_requests_seconds_count[1m]))

If 200 busy threads serve 200 req/s, each request holds a thread for 1 s while using 2 ms of CPU: 99.8% of that time is waiting.

Read the thread dumps

Take three dumps about ten seconds apart (jcmd <pid> Thread.print or jstack), group identical stacks and count them. Synchronous I/O looks like this, abridged:

"http-nio-8080-exec-173" ... java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.NioSocketImpl.read(...)
    at java.net.Socket$SocketInputStream.read(...)
    ...
    at com.example.checkout.PaymentClient.status(PaymentClient.java:57)
    at com.example.checkout.CheckoutController.pay(CheckoutController.java:31)

Two traps. A thread blocked in a native socket read reports RUNNABLE, which doesn’t mean it is using CPU. And threads waiting for a pooled database connection show TIMED_WAITING (parking) under the pool’s getConnection, which says the pool or the database is the wait. Virtual threads don’t appear in the old dump at all; JEP 444 adds jcmd <pid> Thread.dump_to_file -format=json <file>.

.NET: counters, then stacks

Microsoft’s recipe: run dotnet-counters monitor -n <app> and look for a thread count that climbs slowly and steadily while CPU stays well below 100%, often with a long queue and few completed work items. If the count settles above about three times the cores, code is blocking pool threads and the pool is compensating. Then run dotnet-stack report -n <app>: many stacks in Task.InternalWaitCore under your controller mean .Result or .Wait().

Node.js and asyncio

Run staging with node --trace-sync-io, which prints a stack trace whenever synchronous I/O happens after the first turn of the event loop. In Python, asyncio debug mode (PYTHONASYNCIODEBUG=1) logs every callback that runs longer than 100 ms.

Profile off-CPU time, not just CPU

A CPU flame graph is blind to waiting: a thread blocked on a socket uses no CPU, so the culprit never appears. Brendan Gregg’s off-CPU analysis records stacks and durations whenever threads leave the CPU. On the JVM, async-profiler’s wall-clock mode samples every thread whether it is running, sleeping or blocked:

# Linux, any process: 30 s of blocked stacks as an off-CPU flame graph
/usr/share/bcc/tools/offcputime -df -p "$PID" 30 > offcpu.stacks
./flamegraph.pl --color=io --countname=us < offcpu.stacks > offcpu.svg

# JVM: wall-clock samples per thread for 30 s
asprof -e wall -t -d 30 -f wall.html "$PID"

What a trace looks like

One downstream span covers almost the whole request span, and there is a gap before the handler starts: queue time waiting for a thread. Compare request time at the load balancer with time inside the handler; the difference is the wait. If your APM or profiler splits wall time into CPU and wait, a request that is 99% wait is this anti-pattern. Wire both into your observability stack.

Telling it apart from look-alikes

How to fix it

The principle: never hold a scarce thread while you wait, and bound how long and how many requests can wait on any one dependency.

Async I/O end to end

In .NET, make the call chain async from the controller down, so the thread returns to the pool at each await:

// BEFORE: sync-over-async. A pool thread is parked for the whole call.
[HttpGet("orders/{id}")]
public ActionResult<OrderView> Get(string id)
{
    var order = _db.Orders.FirstOrDefault(o => o.Id == id);          // blocks
    var pay = _http.GetFromJsonAsync<PaymentStatus>($"payments/{id}").Result; // blocks
    return new OrderView(order, pay);
}

// AFTER: async all the way. No thread is held while the DB or payment API works.
[HttpGet("orders/{id}")]
public async Task<ActionResult<OrderView>> Get(string id, CancellationToken ct)
{
    var order = await _db.Orders.FirstOrDefaultAsync(o => o.Id == id, ct);
    var pay = await _http.GetFromJsonAsync<PaymentStatus>($"payments/{id}", ct);
    return new OrderView(order, pay);
}

In Python, an async def handler must use non-blocking drivers, created once and shared:

# BEFORE: looks async, but both calls block the event loop for every request.
@app.get("/orders/{order_id}")
async def get_order(order_id: str):
    with pool.connection() as conn:                  # psycopg_pool, synchronous
        row = conn.execute("SELECT id, total FROM orders WHERE id = %s",
                           (order_id,)).fetchone()
    pay = requests.get(f"{PAYMENTS}/status/{order_id}")   # blocks, and no timeout
    return {"order": row, "payment": pay.json()}

# AFTER: asyncpg and httpx; the loop serves other requests while these wait.
http = httpx.AsyncClient(base_url=PAYMENTS, timeout=httpx.Timeout(1.0, connect=0.3))
# db = await asyncpg.create_pool(DSN, max_size=20), created once at startup

@app.get("/orders/{order_id}")
async def get_order(order_id: str):
    row = await db.fetchrow("SELECT id, total FROM orders WHERE id = $1",
                            order_id, timeout=0.5)
    if row is None:
        raise HTTPException(status_code=404)
    pay = await http.get(f"/status/{order_id}")
    pay.raise_for_status()
    return {"order": dict(row), "payment": pay.json()}

If you can’t go async yet, declaring the handler with plain def is safer: FastAPI runs it on a thread pool, which AnyIO caps at 40 threads by default. That is thread-per-request again, with a ceiling of 40 ÷ latency, but at least the loop stays free.

What you pay. Async “colours” the code: every caller up the chain must become async, and every library on the path needs an async version. Stack traces get fragmented and debuggers step awkwardly, request context (trace IDs, tenant, deadlines) must be propagated explicitly, and CPU-heavy work inside a coroutine still blocks the loop. Async doesn’t make the dependency any faster either.

Java: virtual threads keep the blocking style

Java 21 finalised virtual threads (JEP 444): cheap threads that unmount from their OS carrier thread when they block on I/O in the JDK, so you keep plain blocking code and lose the thread ceiling. In JEP 444’s own example, 10,000 one-second sleeping tasks finish at about 200 a second on a 200-thread pool and about 10,000 a second on virtual threads. In Spring Boot it is one property:

spring:
  threads:
    virtual:
      enabled: true   # each request runs on a new virtual thread; Tomcat's maxThreads no longer applies

Concurrency is now unbounded, so cap what the dependency sees with a semaphore, not a thread pool:

private final Semaphore paymentSlots = new Semaphore(100);   // in-flight calls, not threads

PaymentStatus status(String orderId) throws InterruptedException {
    if (!paymentSlots.tryAcquire(50, TimeUnit.MILLISECONDS)) {
        throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "payments busy");
    }
    try {
        return payments.status(orderId);          // blocking HTTP call with a 1 s timeout
    } finally {
        paymentSlots.release();
    }
}

What you pay. On JDK 21 to 23, a virtual thread that blocks inside synchronized is pinned: it holds its carrier, and since there is one carrier per core by default, as many pinned threads as cores stall every other virtual thread. JEP 491, delivered in JDK 24, removes that case; pinning remains for native frames, class loading and class initialisers. Spring Boot’s docs strongly recommend Java 24 or later for virtual threads. Don’t pool virtual threads, revisit ThreadLocal caches that assumed a few long-lived threads, and expect the database pool to become the new queue.

Bulkhead the calls you can’t make async

A legacy SOAP client or a driver with no async API will block whatever thread calls it. Give it its own small pool, so it can never take more than that, and fail fast when it’s full. This is a bulkhead:

legacy_pool = ThreadPoolExecutor(max_workers=16, thread_name_prefix="gst-soap")
slots = asyncio.Semaphore(16)        # the executor's own queue is unbounded

async def verify_gstin(gstin: str) -> dict:
    if slots.locked():               # all 16 busy: reject now, don't queue
        raise ServiceBusy("GST lookup")
    async with slots:
        loop = asyncio.get_running_loop()
        # soap_client needs its own socket timeout: nothing can interrupt a blocked thread
        return await loop.run_in_executor(legacy_pool, soap_client.verify, gstin)

Size bulkheads from normal traffic with headroom. Checkout makes 1,000 payment calls a second at 40 ms, so 1,000 × 0.04 = 40 are in flight; a 100-slot bulkhead gives 2.5× headroom. When payment slows to 1 s, at most 100 calls wait, 100 ÷ 1 s = 100 checkouts a second still succeed, the rest get a fast 503, and the cart and health check keep their threads.

Put a timeout on every wait

Timeouts bound how long a thread or slot is held; bulkheads bound how many. You need both. If payment hangs and HttpClient waits its default 100 s, a 200-thread pool serves 200 ÷ 100 = 2 requests a second. With a 1 s timeout the pool turns over 200 requests a second, still short of 1,000, which is why the bulkhead matters too. Retries multiply the hold: three attempts at a 1 s timeout keep a thread for 3 s, so 200 threads serve about 67 requests a second. Budget retries inside one deadline, as in timeouts, retries and backoff with jitter.

What you pay. Some slow-but-successful calls now fail, so you need a fallback or a clear error, and a timeout on a write needs idempotency before anyone retries it.

Don’t wait at all

The cheapest wait is the one you skip:

• Cache answers that can be a little stale, such as merchant config, FX rates or a partner’s menu; see no caching.
• Move work the user needn’t wait for onto a queue: receipts, SMS, analytics. A UPI-style “payment pending, we’ll confirm shortly” screen turns a blocking wait into a status the app polls; see synchronous vs asynchronous communication.
• Run independent calls in parallel: two 40 ms calls take 40 ms together, not 80.
• Make the dependency faster, with an index or a closer region. Every millisecond off its latency raises the pool’s ceiling.

Watch where the queue moves

Async I/O and virtual threads remove the thread limit, not the work. With them, 1,000 req/s at 1 s means 1,000 concurrent calls to the payment provider instead of 200, and Azure warns that unblocked threads can starve or throttle shared resources. The database pool becomes the next ceiling: HikariCP’s default of 10 connections at 8 ms per query allows 10 ÷ 0.008 = 1,250 queries a second, and threads queue behind it for up to 30 s. Size it deliberately and lower the wait; see connection pooling. For concurrency and parallelism in general, see concurrency vs parallelism.

Compare the fixes

When it’s fine

• Batch jobs, CLIs and migrations. One thread doing one thing, with nobody queued behind it.
• Low-concurrency internal tools. An admin page at 5 req/s × 200 ms keeps one thread busy.
• Fast local I/O. Azure notes that reading a small file from SSD can be cheaper synchronously than dispatching it asynchronously.
• Generous pools with strict timeouts. If rate × timeout fits the pool, blocking is bounded: 100 req/s × 0.5 s = 50 threads, inside 200, even if the dependency hangs.
• Workers where one thread per job is the design. A queue consumer with 16 workers, each blocking on an upload, starves no user request; the queue absorbs the wait, and you scale on backlog.

Cheat sheet

How it shows up in interviews

It rarely arrives by name. It comes as a follow-up in any design with a slow external call: a payment system or checkout that calls a payment provider, a notification service calling SMS and email providers, an API gateway or BFF fanning out, a web crawler with thousands of slow fetches, or an LLM gateway whose calls take 20 s (200 threads ÷ 20 s = 10 requests a second). The classic probe: “During the sale your payment provider’s p99 goes from 300 ms to 5 s. What happens to the product page?”

How to say it in the interview

“Each request holds a thread while it waits, so by Little’s law my ceiling is threads ÷ latency: 200 threads at 50 ms is 4,000 a second, but if payments slows to 1 s it’s 200, with CPU at 5%. Every thread ends up waiting on payments and even the health check fails. So I wouldn’t block threads on I/O: async drivers, or virtual threads on the JVM. Every remote call gets a timeout inside a request deadline, and each dependency gets a bulkhead sized from normal traffic, say 100 in-flight payment calls, so a slow provider degrades checkout with fast 503s instead of taking the whole service down. I’d alert on busy threads near max while CPU is low.”

Will async make my requests faster? No. A request still waits as long as its dependency takes. Async raises throughput and protects unrelated routes by not holding a thread during the wait; latency only improves if you also run independent calls in parallel or cut the dependency’s latency.

Async/await or virtual threads? On the JVM, virtual threads keep blocking-style code, stack traces and debuggers working, and JDK 24 removed the synchronized pinning problem. In .NET, Node and Python, async/await is the native model. Either way, cap concurrency to each dependency explicitly.

How big should the thread pool be? Little’s law: rate × latency, using a high percentile, plus headroom. 1,000 req/s at 200 ms needs 200 threads; the same traffic at a 2 s timeout needs 2,000 in the worst case, which is why you bound the time with timeouts and the share with bulkheads instead of sizing for disaster.

What happens to the database when you go async? It sees more concurrent queries, because the thread pool no longer throttles them. The connection pool becomes the limit, so size it from the database’s capacity, keep the acquire timeout short, and shed load rather than queue for 30 s.

How is this different from a busy front end? Both exhaust request slots. Here threads are idle and waiting, CPU is low, and async I/O helps. In a busy front end threads are computing, CPU is high, and the fix is moving work to a queue and workers.

Where to go next

• Busy front end: the CPU-bound sibling that exhausts the same slots.
• Chatty I/O: many small calls that multiply every blocking wait.
• Retry storm: how retries on blocked threads amplify a slowdown.
• Bulkheads: isolating pools so one dependency can’t sink the service.
• Timeouts, retries and backoff with jitter: deadlines that release blocked threads.
• Backend failure modes: how a slow dependency becomes a cascading failure.
• Connection pooling: the next queue after the thread pool.
• Improper instantiation: per-request clients that lengthen every wait.

References and further reading

1. Synchronous I/O antipattern (Azure Architecture Center)learn.microsoft.comWhy the pattern appears, why async wrappers over blocking calls cost more, the request-queue and 503 symptoms, and the warning that unblocked threads push more load onto shared resources.
2. JEP 444: Virtual Threads (OpenJDK)openjdk.orgLittle’s law for thread-per-request servers, why OS threads cap throughput before CPU does, the 200-thread pool versus virtual-thread example, pinning, and the JSON thread dump.
3. JEP 491: Synchronize Virtual Threads without Pinning (OpenJDK)openjdk.orgDelivered in JDK 24: virtual threads no longer pin inside synchronized, and the remaining cases (native frames, class loading and initialisers).
4. Debug ThreadPool starvation (.NET documentation)learn.microsoft.comdotnet-counters signals of starvation, thread injection at 1–2 threads a second, sync-over-async as the most common cause, and the .Result versus await load test.
5. Don’t Block the Event Loop (or the Worker Pool) (Node.js documentation)nodejs.orgOne event loop plus a small worker pool, the synchronous fs, zlib, crypto and child_process APIs to avoid in servers, and how one long task shrinks the pool.
6. Developing with asyncio (Python documentation)docs.python.orgBlocking code stalls every task on the loop, run_in_executor for blocking calls, and debug mode logging callbacks slower than 100 ms.
7. Off-CPU Analysis (Brendan Gregg)brendangregg.comWhy CPU profiles miss blocked time, off-CPU tracing and sampling, and offcputime with off-CPU flame graphs.
8. The java command: -Xss (Java SE 21 documentation)docs.oracle.comDefault platform-thread stack size: 1,024 KB on Linux/x64 and macOS/x64, 2,048 KB on Linux/AArch64 and macOS/AArch64.
9. Apache Tomcat 10.1: The HTTP Connectortomcat.apache.orgmaxThreads defaults to 200 request-processing threads, which caps simultaneous requests; acceptCount defaults to 100.
10. Spring Boot reference: Virtual threadsdocs.spring.iospring.threads.virtual.enabled, the advice to prefer Java 24 or later, and why thread-pool properties stop having an effect.