Chatty I/O

Chatty I/O means doing one job with many small I/O requests when a few larger ones would do: a query per cart line, an HTTP call per product tile, a cache read per key, a disk write per log line. Every request pays a fixed toll (a network round trip, protocol and parsing work, a log line, a trace span) that has nothing to do with how much data it carries. So the job’s latency becomes call count × round-trip time, and it grows with every item you add.

Picture it first: one rider per item

You need 12 things from a quick-commerce app that delivers in 10 minutes. You order milk and wait for the rider, then order bread and wait again. Twelve orders, twelve rides: two hours for one bag of groceries. Put all 12 in one basket and you wait 10 minutes.

The ride costs the same for a packet of salt as for a bag of rice, so splitting the order multiplies the cost and buys nothing. Twelve separate orders placed at once would also arrive in about 10 minutes, but 12 riders are on the road and you wait for the slowest.

Map it back:

• The ride is a round trip: network latency both ways, plus accepting, parsing, authorising and logging a request.
• An item is a row, key or record. Its size barely matters.
• One basket is a batch: one query with 80 ids, one MGET, one bulk call.
• Twelve riders at once is parallel calls: the same load on the store, and the slowest sets your wait.

Where the time goes

Before a single byte of your data moves, every remote call pays for:

• The network round trip. A fraction of a millisecond inside one zone, a millisecond or two across zones in a region, tens of milliseconds between regions.
• Connection set-up, if the call can’t reuse a connection: TCP and TLS handshakes add round trips of their own, which is why connection pooling exists.
• Per-request work: encoding and decoding, routing, an auth check, a log line, a trace span, and for a database, parsing and planning the statement.
• System calls. The Redis pipelining guide points out that executing a simple command is cheap, but the read() and write() system calls around it are not.
• Queueing at every hop: the client’s connection pool, a load balancer, the server’s worker pool.

So for a task that makes N calls, a useful model is:

time ≈ N × (round trip + per-call overhead) + total bytes ÷ bandwidth

When N is large and each call is small, the first term is everything.

A worked example: the cart page

A Flipkart-style cart service renders a cart during a sale. For each line it calls the pricing service for the price and the inventory service for stock, both in the same region. Assume each small call costs 1.5 ms: the round trip across zones plus the service’s own lookup. (Measure yours; the shape of the result doesn’t depend on the exact number.)

• Cart of 3 items: 3 × 2 × 1.5 ms = 9 ms. Nobody notices.
• Cart of 80 items: 80 × 2 × 1.5 ms = 240 ms.
• Batched: one call to each service carrying all 80 SKUs, say 3 ms each because they return more rows; sent side by side, about 3 ms.

The data is not the problem: 160 replies of about 100 bytes are 16 KB; at 10 Gbit/s that is 16,000 B ÷ 1,250,000,000 B/s ≈ 13 µs. The bytes cost microseconds, the waiting 240 ms: latency versus bandwidth in one page.

Distance makes it far worse, because every call pays the full round trip:

The last row uses Azure’s published inter-region latency: the monthly median round trip between its Central India and Southeast Asia (Singapore) regions is about 53 ms. The 240 ms cart takes over eight seconds if its dependencies move to Singapore.

Latency is the half users feel; the other half lands on your dependencies. At a sale peak of 2,000 cart views a second with 20 lines on average, the chatty version sends 2,000 × 20 × 2 = 80,000 calls/s to pricing and inventory; the batched one sends 2,000 × 2 = 4,000. The extra calls burn CPU on the toll, not the work, so fixing chatty I/O often raises throughput too. In the load test in Azure’s chatty I/O guidance, replacing 45 separate SELECTs with one query took throughput from 410 to 3,970 requests per minute.

Symptoms you’d see

Chatty I/O rarely shows up as one slow thing. It shows up as a lot of fast things:

• Latency grows with item count. A 3-item cart is quick, an 80-item cart is slow, with a near-straight line between. The slope is roughly calls per item × round trip (3 ms per item here).
• High latency, low CPU, small payloads. App servers mostly wait. Bytes per request are small; requests per second are high.
• A dependency takes many calls per page view. Pricing receiving 20 calls per cart view is a ratio, not a traffic spike.
• Traces full of tiny spans: dozens of 1–2 ms calls under one request, one after another.
• A busy connection pool with fast queries. A request running 20 queries of 1.5 ms holds its connection for 30 ms. By Little’s law, 2,000 requests/s × 0.030 s = 60 connections busy, against 6 for one 3 ms query.
• Latency jumps when something moves, with no code change. A database fails over to another zone or a module becomes a microservice, and the page gets several times slower because every round trip grew.
• Timeouts under load, as contention for I/O resources rises.

Why it happens

Nobody sets out to write chatty code. It grows out of code that reads well.

ORM lazy loading and N+1

The best-known source. You load 50 restaurants, touch restaurant.menu in a loop or a template, and the ORM quietly runs one query per restaurant: 1 + 50 round trips. The N+1 query problem covers it in depth: fetch strategies, JOIN pitfalls, DataLoader and framework guards. Chatty I/O is the general case: the same multiplication applies to every kind of I/O, not only ORM queries.

Loops over per-item calls

for line in cart: price = pricing.get(line.sku) reads like a function call, and a generated client makes it look like one. It is a network round trip. An await inside a loop is the most common chatty I/O in modern code.

Fine-grained interfaces

In-process, small objects with small methods are good design. Martin Fowler’s first law of distributed object design is “don’t distribute your objects”, because a remote call is orders of magnitude slower and can fail. His example: 100 in-process calls for 100 product prices are fine; remotely, ask for all 100 in one call. APIs with one endpoint per property (/users/42/name, /users/42/avatar) force several round trips per screen. Fowler’s fix is a Remote Facade: a coarse-grained interface over fine-grained objects.

Microservice splits

Splitting a monolith turns function calls into network calls. If services are cut by noun (prices, stock, ratings, delivery ETAs) and each exposes only get-by-id, a listing page that showed 24 products with one in-process loop now makes 24 × 4 = 96 remote calls. Fowler notes that microservice advocates favour coarse-grained interactions for exactly this reason: design batch-shaped APIs from day one.

Per-key cache reads

A cache makes each read fast, not free. Fifty GETs at 0.5 ms each are 25 ms. The Redis docs give the extreme case: at a 250 ms round trip, a client sending one command at a time manages at most 4 commands a second, even if the server could handle 100,000.

Tiny file and blob I/O

Writes: opening a file, appending one record and closing it per event, or committing every row on its own, which PostgreSQL’s docs warn makes the server do a lot of work per row. Reads: a file byte by byte, or a blob in small ranged GETs. A 10 MiB blob in 4 KiB ranges is 10,240 ÷ 4 = 2,560 requests; at an assumed 10 ms each, 25.6 seconds for what one streaming GET does in a fraction of a second.

Clients do it too: a mobile app making one request per tile, over a network whose round trips are far longer than a data centre’s.

Why it survives code review

In development, N is 3 and the database is on your laptop, where a round trip is tiny. No single span looks wrong, and averages hide the long carts. The bill arrives with real data, real distances and peak traffic.

How to detect it

Latency graphs tell you something is slow. To prove chatty I/O, count.

Count calls per request

Add a request-scoped counter in middleware: queries, cache commands and outbound calls per request, logged with the route and an item count (cart lines, tiles). Many APM tools already show queries per transaction. Two checks make it sharp:

• The slope test. Group requests by item count. If calls per request rise with it, the code loops over I/O.
• The ratio test. Pricing calls per second ÷ cart views per second should be a small constant such as 1. A value near 20 means a loop.

Azure’s guidance adds a habit: read the traces of the slowest requests, not averages, because the long carts are where the pattern hides.

pg_stat_statements: calls versus rows

pg_stat_statements groups statements that differ only in their constants and records, for each one, how many times it ran (calls), the total rows it returned or affected (rows) and its timing. Chatty I/O has a clear signature: a huge calls count, about one row per call and a tiny mean time.

-- The most frequently run statements, and how many rows each call returns
SELECT left(query, 45)                             AS statement,
       calls,
       round(rows::numeric / NULLIF(calls, 0), 1)  AS rows_per_call,
       round(mean_exec_time::numeric, 3)           AS mean_ms,
       round(total_exec_time::numeric)             AS total_ms
FROM pg_stat_statements
ORDER BY calls DESC
LIMIT 10;

statement                                | calls      | rows_per_call | mean_ms | total_ms
SELECT price FROM prices WHERE sku = $1  | 24,102,360 |           1.0 |   0.021 |  506,150
SELECT qty FROM stock WHERE sku = $1     | 24,102,360 |           1.0 |   0.019 |  457,945
SELECT * FROM carts WHERE id = $1        |  1,205,118 |           1.0 |   0.034 |   40,974
(illustrative output)

The price and stock lookups each run 20 times per cart (24,102,360 ÷ 1,205,118 = 20), return one row each time and take about 20 microseconds. The database isn’t slow; it is being asked too often. Since PostgreSQL 18, IN lists that differ only in length fold into one entry, so a batched query shows up as a single row with a sensible calls count.

Redis INFO commandstats

INFO commandstats reports calls, total usec and usec_per_call for each command. Sample it twice, a minute apart, and divide the change in cmdstat_get calls by your page views in that minute:

# Commandstats (illustrative)
cmdstat_get:calls=1841220077,usec=1196793050,usec_per_call=0.65,rejected_calls=0,failed_calls=0
cmdstat_mget:calls=90321,usec=1625778,usec_per_call=18.00,rejected_calls=0,failed_calls=0

Each GET costs Redis 0.65 µs of CPU, while the client waits a whole network round trip for it. Tens of GETs per page view and almost no MGET point at a loop.

The staircase in a trace

In a distributed trace, chatty I/O is a staircase: dozens of short child spans under one parent, each starting where the last ended. Two services alternating is the signature of a loop body making two calls; see observability: logs, metrics and traces.

Telling it apart from look-alikes

They combine, too: a chatty page can multiply queries until you have a busy database.

How to fix it

Every fix moves the loop to the other side of the wire: send the list once, and let the server iterate where iteration is cheap.

Batch the database reads

If prices and stock live in one database, replace per-line queries with one set-based query:

# BEFORE: two queries per cart line, each waiting for the last
for line in cart.lines:
    line.price = db.fetch_val("SELECT price FROM prices WHERE sku = %s", line.sku)
    line.stock = db.fetch_val("SELECT qty FROM stock WHERE sku = %s", line.sku)

# AFTER: one query for the whole cart
skus = [line.sku for line in cart.lines]
rows = db.fetch_all(
    """SELECT p.sku, p.price, s.qty
         FROM prices p JOIN stock s ON s.sku = p.sku
        WHERE p.sku = ANY(%s)""",
    skus,                      # one array parameter, however long the cart
)
by_sku = {r.sku: r for r in rows}
for line in cart.lines:
    row = by_sku.get(line.sku)  # a missing SKU is a business decision, not a crash
    line.price, line.stock = (row.price, row.qty) if row else (None, 0)

Prefer one array parameter (= ANY($1)) to a hand-built IN (…) list, so the statement text stays the same whatever the cart size. JOINs and ORM eager loading (includes, select_related, prefetch_related, join fetch) do the same job; the N+1 article compares them.

You pay with bigger result sets, and with row explosion if you JOIN two to-many relations. Cap batches at a few hundred to a few thousand ids and chunk anything larger.

Batch endpoints and bulk APIs

Between services, the equivalent is an endpoint that takes many ids:

// BEFORE: two HTTP calls per line, awaited one by one
for (const line of cart.lines) {
  line.price = await pricing.get(`/prices/${line.sku}`);
  line.stock = await inventory.get(`/stock/${line.sku}`);
}

// AFTER: one batch call per service, both in flight together
const skus = cart.lines.map((l) => l.sku);      // chunk if above the API's limit
const [prices, stock] = await Promise.all([
  pricing.post('/prices:batchGet', { skus }),   // → { "sku-1": 499, ... }
  inventory.post('/stock:batchGet', { skus }),
]);
for (const line of cart.lines) {
  line.price = prices[line.sku] ?? null;        // missing → show "unavailable"
  line.stock = stock[line.sku] ?? 0;
}

A GET /products?ids=… works for short lists; URL length limits push bigger batches into a POST body. Managed stores ship the same idea: DynamoDB’s BatchGetItem reads up to 100 items or 16 MB in one call, and may fetch them in parallel internally.

You pay with new failure modes. BatchGetItem returns a partial result when it hits a size or throughput limit and lists the rest in UnprocessedKeys; AWS recommends retrying those with exponential backoff, not immediately. It returns items in no particular order, so key results by id. One slow item now delays the whole reply, and a batch URL caches worse in an HTTP cache than per-item URLs. Return per-item results keyed by id, so the caller can render whatever arrived.

Redis: MGET and pipelining

# BEFORE: 50 GETs, 50 round trips
names = [r.get(f"product:{sku}:name") for sku in skus]

# AFTER: one MGET, one round trip (values in key order, None if missing)
names = r.mget([f"product:{sku}:name" for sku in skus])

# A different command per key? Pipeline them: one round trip, many commands.
pipe = r.pipeline(transaction=False)
for sku in skus:
    pipe.hgetall(f"product:{sku}")
products = pipe.execute()

MGET is one command with many keys. A pipeline sends many commands without waiting for each reply, which suits mixed commands such as an HGETALL per key. It saves more than round trips: many commands share one read() system call and their replies one write(), and the Redis docs report operations per second rising towards ten times the unpipelined baseline.

You pay with memory and cluster constraints. Redis queues pipelined replies in memory, so send long pipelines in chunks (the docs suggest something like 10,000 commands). With clustering enabled, MGET works only when every key is in one hash slot, or you get a CROSSSLOT error. Hash tags such as {cart:42} co-locate keys but concentrate load on one shard; the alternative is pipelining per node. And a pipeline is not a transaction.

Coarse-grained APIs and a BFF

When one screen needs data from several services, compose it close to the services, not on the phone. A backend-for-frontend (BFF) takes one request per screen, calls each service’s batch endpoint once and in parallel, and returns the screen in one response; see API composition, BFF and GraphQL federation. GraphQL fixes the client half, but naive resolvers make the server chatty instead, so pair it with a per-request batching loader (REST vs GraphQL vs gRPC).

You pay with larger payloads, one more service to own, APIs shaped around screens, and partial-failure decisions: if ratings time out, show the tile without stars.

Buffer and batch writes

Writes get chatty in the other direction: one event, one statement, one commit.

-- BEFORE: one statement and one commit per event (autocommit), thousands of times
INSERT INTO order_events (order_id, kind, at) VALUES ($1, $2, $3);

-- AFTER: one statement inserts a whole buffered batch (equal-length arrays)
INSERT INTO order_events (order_id, kind, at)
SELECT * FROM unnest($1::bigint[], $2::text[], $3::timestamptz[]);

-- Bulk loads: COPY streams every row in one command
COPY order_events (order_id, kind, at) FROM STDIN WITH (FORMAT csv);

PostgreSQL’s advice for loading many rows is to commit once, not per row, and to prefer COPY, which it says is almost always faster than INSERT even when inserts are prepared and batched in one transaction. For files, keep the file open and write through a buffer: one million 100-byte events, one write() each, is 1,000,000 system calls; through a 64 KiB buffer it is 100,000,000 ÷ 65,536 ≈ 1,526.

You pay with durability and freshness. A memory buffer dies with the process, so data you can’t lose goes to a durable queue first. Records appear only when the batch flushes, so flush on size or time (say 500 rows or 200 ms). One bad row can fail a whole multi-row statement.

Parallelise what you can’t batch

Sometimes the dependency has no batch API and you can’t add one. Then overlap the calls, with a limit. The 160 calls of an 80-item cart, 10 in flight at a time, take 160 ÷ 10 × 1.5 ms = 24 ms instead of 240. This needs non-blocking I/O or a pool of threads; see synchronous I/O.

Cache what repeats

If the same small reads repeat across requests (product names, a restaurant’s rating), the cheapest round trip is none. A local cache with a short TTL, plus one batched call for the misses, removes most of them; see no caching. You pay with staleness and memory.

Compare the fixes

The other ditch: extraneous fetching

Batching is easy to overshoot. A cart that needs a name, price and thumbnail calls /products?ids=… and gets full descriptions, specifications and 20 image URLs each: 80 × 30 KB = 2.4 MB instead of 80 × 200 B = 16 KB. That is extraneous fetching, the opposite anti-pattern. Azure frames it as fewer calls against more data per call, and suggests splitting an object’s small, hot part from the rarely used rest.

Aim for as few round trips as the screen needs, each carrying only what it shows. Projections keep batches lean: a SQL column list, DynamoDB’s ProjectionExpression, a GraphQL selection set, a fields= parameter.

When it’s fine

Chatty I/O is a cost, not a sin. Leave it alone when:

• N is small and bounded. An order with two delivery addresses makes three lookups; batching saves a millisecond and adds code.
• There is no round trip. An in-process cache hit is a memory read. SQLite runs inside your process, so its docs argue that 200 statements per page, excessive for a client/server database, is fine there.
• Nobody is waiting. A nightly export can afford per-item calls, if it doesn’t flood a dependency that live traffic shares.
• Per-item calls buy something. Streaming renders the first tiles early, separate calls isolate failures, and write batching adds a flush interval of delay that a payment status may not tolerate.

Cheat sheet

How it shows up in interviews

Chatty I/O is rarely the question; it is the follow-up. It comes up in any design that renders lists from several sources: a cart or listing during a sale, a Swiggy-style restaurant list with ratings and ETAs, a feed needing author profiles, an IRCTC-style search showing availability per train and class. It also hides in write paths: events, logs and notifications written a row at a time.

The probes: “Each tile needs price, stock and rating from three services. What does one page cost?” and “What happens if the inventory database moves to another region?” Answer with the multiplication, then the batch, then its cost, using numbers to know for round trips.

How to say it in the interview

“Chatty I/O is doing one task with many small round trips. Each call pays a fixed cost, mostly round-trip time, so latency becomes calls times RTT, not data size. An 80-item cart making two 1.5 ms calls per item waits 240 ms; with those services in Singapore, over eight seconds. I’d spot it by counting calls per request, checking that latency rises with item count, and finding statements with huge call counts at one row each. The fix is to move the loop to the server: one ANY query, batch endpoints, MGET or pipelining, a BFF, buffered writes. I’d parallelise only what I can’t batch, since that hides latency but not load, and project fields so batches don’t become extraneous fetching.”

Isn’t calling them all in parallel good enough? It fixes latency, not load. The dependency still gets 80,000 calls a second, and with 80 calls per page more than half of pages wait on at least one p99-slow call. Batch first; parallelise the rest with a limit.

How big should a batch be? Big enough to amortise the round trip, small enough to bound memory, payload and the damage of one failure: tens to a few hundred items for service APIs. Respect hard limits such as BatchGetItem’s 100 items or 16 MB, and send chunks of a long list in parallel.

What if part of a batch fails? Return per-item results keyed by id, render what arrived, and retry only the failed keys with backoff, as DynamoDB does with UnprocessedKeys. Never let one bad id fail 79 good ones.

Does GraphQL fix chatty I/O? Only on the client side: one query per screen. Naive resolvers still fetch one record per field on the server; a per-request batching loader turns those into one call per type.

How is this different from N+1? N+1 is the ORM-and-database case. Chatty I/O covers every kind of I/O: HTTP and RPC calls, cache commands, files and blobs. The fix has the same shape everywhere: fewer, bigger requests.

Where to go next

• The N+1 query problem: the ORM case in depth.
• Extraneous fetching: the ditch on the other side of batching.
• API composition, BFF and GraphQL federation: composing a screen from many services.
• REST vs GraphQL vs gRPC: how protocol choice shapes round trips.
• No caching and synchronous I/O: the neighbouring anti-patterns.
• Connection pooling: why long call chains exhaust pools.
• Observability: logs, metrics and traces: the spans and counters that expose the staircase.
• Latency vs throughput vs bandwidth: why bytes are cheap and waits are not.

References and further reading

1. Chatty I/O antipattern (Azure Architecture Center)learn.microsoft.comCommon causes (per-record queries, per-property HTTP endpoints, small file writes), detection steps, and the load test where one query replaced 45 and throughput rose from 410 to 3,970 requests per minute.
2. Microservices and the First Law of Distributed Objects (Martin Fowler)martinfowler.comWhy remote interfaces must be coarse-grained: 100 in-process calls for prices are fine, 100 remote ones should be one batched call.
3. Redis pipeliningredis.ioRound-trip arithmetic (250 ms RTT caps a client at 4 requests/s), how pipelining also saves read() and write() system calls, and sending long pipelines in batches to bound server memory.
4. Redis INFO commandredis.ioThe commandstats section: calls, usec and usec_per_call for each command type.
5. Redis multi-key operationsredis.ioWhy MGET needs every key in one hash slot when clustering is enabled, CROSSSLOT errors, and hash tags.
6. PostgreSQL documentation: pg_stat_statementspostgresql.orgPer-statement calls, rows and mean_exec_time, normalisation of constants, and IN lists of different lengths squashed into one entry.
7. PostgreSQL documentation: Populating a Databasepostgresql.orgCommit many inserts once instead of per row, and why COPY beats even prepared, batched INSERTs for bulk loads.
8. Amazon DynamoDB API Reference: BatchGetItemdocs.aws.amazon.comUp to 100 items or 16 MB per call, partial results in UnprocessedKeys retried with exponential backoff, and no ordering guarantee.
9. Azure network round-trip latency statisticslearn.microsoft.comMonthly median round-trip times between Azure regions, including about 53 ms between Central India and Southeast Asia (Singapore).
10. Many Small Queries Are Efficient In SQLitesqlite.orgWhy 200 statements per page is excessive for a client/server database but fine for an in-process engine with no round trips.