Scaling reads

Scaling reads is the playbook for systems where far more people look at data than change it: feeds, product pages, short-link redirects, event pages. You make each read cheaper, then stop most reads before they reach the primary database, climbing a ladder of techniques and stopping at the first rung that meets your numbers. Every rung buys throughput with a little staleness, and the skill is saying exactly how much.

Picture it first: the enquiry counter before Diwali

A big station has one enquiry counter and one clerk with the master register of trains. The week before Diwali, thousands of people ask the same few questions. The station improves things one step at a time:

1. Tab the register, so the clerk finds any train in seconds.
2. Photocopy the register for three more windows. Each copy lags a little whenever the master changes.
3. Write the 20 most-asked trains on a whiteboard. Most people read it and leave, but someone must wipe a line when a platform changes.
4. Put a display board at the entrance. Most travellers never reach the counter.
5. Print tomorrow’s timetable booklet overnight, so nobody works out a route at the window.
6. Put the 10:00 train everyone asks about on every board, so no single board draws the whole crowd.

Map it back: the register is your primary database, tabs are indexes, photocopies are read replicas, the whiteboard is a cache, the entrance board is a CDN, the booklet is a precomputed view, and the 10:00 train is a hot key. Each step costs more than the one before, and the station takes it only when the queue demands it.

Spotting the pattern

Listen for these cues in the prompt:

• A lopsided read:write ratio. Feeds, product pages, redirects and leaderboards are read 100 to 1,000 times per write; Facebook’s memcache paper notes users consume an order of magnitude more than they create.
• Many people read the same thing. One product, one event, one celebrity post. Shared reads make caching pay; the AWS caching guidance warns that when every request needs a unique answer, the hit rate is negligible.
• A strict read latency target, such as p99 under 200 ms for a page.
• Predictable hot items: a flash-sale phone at 12:00, the tatkal train list at 10:00, a cricket final.
• A global audience reading data that lives in one region.

It does not apply when:

• Writes dominate. Telemetry, click streams and delivery-partner location pings are write problems; replicas and caches don’t help.
• The read must be perfectly fresh. A wallet balance before a debit reads the primary. Apply the pattern to the browse path around it.
• The scale is small. At 50 reads per second, one indexed database is the design.

The rest of this lesson uses one running example: sale-day product pages on a large Indian shopping site, peaking at 100,000 page views per second against about 200 writes per second (stock, prices, reviews), a 500:1 ratio. Two assumptions to replace with your own benchmarks: one database node serves about 10,000 simple indexed queries per second, and a tuned page view costs 2 queries, so one node serves about 5,000 views per second.

Levels 0 and 1: make one database fast

Level 0: the version that breaks

The first draft runs this on every view, on a primary with no index on product_id:

SELECT * FROM reviews WHERE product_id = 42 ORDER BY created_at DESC LIMIT 10;
-- then, in a loop, once per review:
SELECT * FROM users WHERE id = ?;

To find ten reviews, the database reads all 20 million rows, then fetches each author separately: the N+1 query problem, 11 queries per page. Assume the scan costs 50 ms of CPU. A 16-core primary manages 16 ÷ 0.05 s = 320 scans per second, so about 300 page views per second, and stock updates queue behind the scans just as the sale starts.

Level 1: fix the query

Before adding any box to the diagram, make each read cheap:

• Index the access path. An index on (product_id, created_at DESC) turns the scan into a short B-tree walk; make it covering with INCLUDE and PostgreSQL answers from the index alone, best when table pages rarely change (database indexes, query plans).
• Kill N+1. Fetch all ten authors in one WHERE id = ANY(...), or copy the author name onto the review. Eleven queries become two.
• Select only what you render instead of dragging every column over the wire (extraneous fetching).
• Pool connections. 500 app servers × 20 connections is 10,000 connections; a pooler multiplexes them onto a few hundred (connection pooling).
• Scale up. A bigger machine is the quickest fix, but it has a ceiling and is still one machine (vertical vs horizontal scaling).

CREATE INDEX reviews_by_product
  ON reviews (product_id, created_at DESC)
  INCLUDE (rating, title, author_name);

SELECT rating, title, author_name, created_at
FROM reviews
WHERE product_id = 42
ORDER BY created_at DESC
LIMIT 10;

You gain a jump from about 300 to about 5,000 views per second on the same hardware. You pay a little on every write, since each insert updates each index. It breaks at about 5,000 views per second per node; the example needs 20 times that, and one primary is a single point of failure.

Say: “First I make each page two indexed queries. Caching a slow query only hides it until the first miss.”

Level 2: read replicas

The primary streams its write log to read-only copies; writes go to the primary and reads spread across the replicas. Amazon Aurora supports up to 15 replicas per cluster, with lag usually well under 100 ms that rises during write bursts. PostgreSQL’s docs call a hot standby eventually consistent with the primary. Mechanics are in data replication.

You gain near-linear read capacity, n × 5,000 views per second, plus failover targets. You pay for full copies at full price; every replica replays every write, so replicas never add write capacity; and reads are stale by the lag, small on a good day and unbounded on a bad one.

It breaks on count and cost. 100,000 ÷ 5,000 = 20 replicas flat out, or about 29 at a safer 70% CPU. Aurora’s cap of 15 tops out at 15 × 5,000 = 75,000 views per second, and you would be paying for dozens of database servers to re-answer the same few thousand popular pages. A small, repeatedly read hot set is exactly what a cache is for (database scaling compares the options).

Say: “Replicas take the long tail of misses and give me failover. The hot set belongs in a cache.”

The read-your-writes trap

Asha changes her delivery address, taps Save, and the page reloads 200 ms later from a replica that hasn’t received the change. She sees her old address. Four fixes, bluntest first:

1. Pin after a write. For N seconds after a user writes, with N above your p99 lag (say 5 s), send their reads to the primary. At 200 writes per second with about three follow-up reads each, that is roughly 600 extra reads per second on the primary: cheap.
2. Carry a position token. After the commit, read the primary’s log position with pg_current_wal_lsn() and keep it in the session. Use a replica only if its pg_last_wal_replay_lsn() has reached the token; otherwise wait briefly or read the primary. MySQL can do the same with GTIDs.
3. Read critical screens from the primary: order confirmation, wallet balance, the address chosen at checkout.
4. Keep reads monotonic. Route each user to the same replica so a refresh never jumps back in time.

Level 3: cache the hot set

Put an in-memory cache such as Redis in front of the database and use it cache-aside: check the cache; on a miss, read a replica, store the result with a TTL, return it. On a write, update the database, then delete the key. Facebook’s memcache design deletes rather than overwrites because deletes are idempotent. Alternatives such as read-through and write-through are compared in caching strategies; the basics are in caching 101.

What to cache. Cache the assembled object (product, seller, rating summary, top reviews) under one key such as product:42:v3, so a hit is one GET instead of two queries plus assembly. Rendered responses are cheaper per hit but multiply by every language, currency and device; leave those to the CDN. Also cache “not found” with a shorter TTL (negative caching), so requests for a deleted product don’t all reach the database.

How big. One million products × 4 KB ≈ 4 GB: the whole catalogue fits in memory. Redis’s benchmark page shows one instance doing about 180,000 SETs per second without pipelining; it runs commands mostly on one thread, so you scale with shards (distributed caching). Planning at 100,000 operations per second per shard, four primaries with replicas give headroom.

The arithmetic that matters is database reads = reads × (1 − hit rate). Small changes in hit rate are large changes in database load.

So effort goes into the hit rate: cache whole objects, use the longest TTL each screen tolerates, and warm the cache before a known spike.

An in-process L1 cache skips the network hop. The AWS guidance lists its costs: servers disagree, backing-store load grows with fleet size, and every deploy starts cold. Keep it for small, very hot data with TTLs of seconds; Redis client-side caching (“tracking”) can push invalidations to it.

Invalidation is TTL plus delete-on-write, with one race to know. A reader misses and loads an old row from a lagging replica; a writer commits and deletes the key; then the reader stores its old value until the TTL. Facebook’s fix is leases: a miss gets a token that a delete voids, so the stale set is rejected. At very large scale, deletes come from the commit log, not app code; Facebook’s mcsqueal daemon broadcast them from MySQL’s log (change data capture; more failure modes in hot keys, cache stampede and invalidation).

You gain sub-millisecond hits, and the database tier shrinks from 20-plus replicas to a primary and two replicas. You pay with a staleness window, another cluster to run, and invalidation code on every write path. It breaks when misses spike (cold start, mass expiry), when one key outgrows a shard, or when every user reads something different, like their own feed.

Level 4: cache at the edge

Many responses are identical for everyone: the anonymous product page, an event page, catalogue JSON, images. Let a CDN keep copies near users, controlled by headers from the origin:

Cache-Control: public, max-age=30, stale-while-revalidate=30

The response is fresh for 30 seconds. stale-while-revalidate, from RFC 5861, then lets the edge serve the stale copy for up to 30 more seconds while it refreshes in the background, so no user waits on the origin. RFC 5861 states the cost: tolerate responses as old as max-age plus the window, here 60 seconds. Its sibling stale-if-error keeps the last good copy flowing when the origin fails.

Origin arithmetic. A popular page costs the origin about one refresh per edge location per 30 seconds. With 100 locations (an assumption), the hottest product costs about 100 ÷ 30 ≈ 3 requests per second instead of tens of thousands. Proxies that collapse concurrent misses into one fetch, such as Nginx or Varnish, keep it low right after expiry.

Personal data stays out. Member prices, cart counts and “delivery by Tuesday to 560034” come from a small uncacheable API call after the page loads. Per-user responses carry Cache-Control: private, which RFC 9111 says a shared cache must not store; s-maxage gives shared caches their own lifetime.

You gain traffic that never reaches your servers and short hops for a global audience. You pay with up to 60 seconds of lost control: a wrong price stays live until a purge reaches every location. Cache keys must vary on the right things (language, currency) and nothing else, or the hit rate collapses. It breaks for personal or fast-changing data (live stock, seat maps) and long-tail content that is rarely popular in any one location.

Level 5: precompute the answer

Sometimes even a miss costs too much, or nothing is shared: every home feed is different, and a rating average over 50,000 reviews is expensive. Then store the answer, not the ingredients:

• Denormalised counters. Keep rating_avg and rating_count on the product row, updated when a review is written (data modelling and denormalisation).
• Materialised views. Store a query’s result and refresh it on a schedule, such as “top 100 deals per category” every minute. Staleness equals the refresh interval.
• Read models (CQRS). The write side publishes changes through a transactional outbox or change data capture, and projectors build documents shaped for each screen (event sourcing and CQRS).
• Fan-out-on-write timelines. When someone posts, push the post ID into each follower’s cached timeline, so loading a feed is one range read.

The fan-out arithmetic shows both sides (illustrative numbers). At 5,000 posts per second and 200 followers each, writes become 5,000 × 200 = 1,000,000 timeline inserts per second, while a feed load drops from about 200 lookups to 1. Then a celebrity with 50 million followers posts: 50 million inserts, 50 seconds of the entire fan-out capacity for one post. The fix is a hybrid: skip fan-out above a threshold (say 100,000 followers) and merge those few accounts’ posts at read time (push vs pull).

You gain one-lookup reads with predictable latency. You pay with write amplification, storage, seconds of pipeline lag, and backfills whenever a view’s shape changes. It breaks when the amplified writes become the bottleneck, which makes it a write-scaling problem.

Say: “I’m trading write amplification for read speed: one post becomes 200 inserts, so a feed load is one read. Celebrities are pulled at read time.”

Level 6: shard the read path and tame hot keys

The top rung handles two different problems.

Volume. The data or miss traffic outgrows one cluster. Shard the database by product_id (database sharding) and place cache keys with consistent hashing. For a global audience, keep read copies near users; Aurora’s global database keeps PostgreSQL read copies in up to 10 other Regions (multi-region).

Skew. Sharding spreads keys, not the traffic on one key; Facebook’s memcache paper notes one key can draw 20% of a server’s requests. Scale the example to 1,000,000 views per second: logged-in users bypass the CDN, and each of their page loads reads the hero deal’s price-and-stock object, 100,000 reads per second on one key. That fills a shard at our planning ceiling, and at 4 KB per value it is about 3.2 Gbit/s (the Redis docs’ own example), beyond a 1 Gbit/s link.

Combine several of these tools:

1. A short-TTL L1. If 200 app servers each refresh the key at most once a second, Redis sees at most 200 reads per second for it.
2. Key copies. Store p:42#0 to p:42#7 on different shards; readers pick one at random, writers delete all eight. Facebook replicates high-traffic keys within a pool for the same reason.
3. Single-flight. On a miss, one request per key per process loads the value and the rest wait for it (Go’s singleflight package). Facebook’s leases do it fleet-wide, issuing a lease per key at most once every 10 seconds; for herd-prone keys, peak database queries fell from 17,000 to 1,300 per second.
4. Serve stale while one caller refreshes. With a soft and a hard TTL (AWS guidance), one caller refreshes after the soft TTL while others keep the old value, up to the hard TTL.
5. Jitter TTLs. 50,000 keys loaded at 10:00:00 with an exact 300-second TTL all expire at 10:05:00. With 300 ± 30 seconds, the misses spread over a minute: about 830 per second.
6. Pre-warm known spikes. The tatkal window opens at 10:00 and the sale at 12:00; load hot keys at 9:55.

The read path for a hot entity, in sketch form:

def get_product(pid):
    key = f"product:{pid}:v3"
    val = l1.get(key)                       # in-process, 1 s TTL
    if val is None:
        val = redis.get(key)
        if val is None:                     # one loader per key; others wait for it
            val = single_flight.do(key, lambda: load_and_fill(pid, key))
        l1.set(key, val, ttl=1)
    return val

def load_and_fill(pid, key):
    val = replica.fetch_product_page(pid)   # 2 indexed queries
    redis.set(key, val, ex=300 + random.randint(-30, 30))   # jittered TTL
    return val

def update_price(pid, price):
    primary.execute("UPDATE products SET price = %s WHERE id = %s", (price, pid))
    redis.delete(f"product:{pid}:v3")       # after commit: delete, don't overwrite

Putting the layers together

Stack the rungs and traffic shrinks at every layer. With illustrative splits for the running example, the CDN serves the 60% of views from anonymous visitors, L1 serves a quarter of the rest, and Redis hits 95% of what remains. Only 1,500 views per second miss every cache: 1,500 × 2 = 3,000 queries per second, 30% of one replica. The primary takes the 200 writes plus about 600 pinned reads per second from users who just saved something.

Every layer that absorbs traffic serves something older than the primary. State the cost per layer, not one vague “eventually consistent” (strong vs eventual consistency).

Which classic problems use it

Most design problems have a read-heavy path. Know which rung is enough, and spend interview time on the hard part.

Common mistakes

• Caching before indexing. The cache hides a 50 ms scan until a cold start sends thousands at once.
• No invalidation plan. TTL-only means edits stay invisible for the whole TTL; overwriting instead of deleting invites races.
• Read-after-write surprises. Users see their own stale data after a save. Answer with pinning, a log-position token, or delete-on-write.
• Replicas for a write problem. Every replica replays every write, so 15 replicas add no write throughput (scaling writes).
• A cold-start stampede. A deploy empties every L1, or 50,000 keys share one TTL. Pre-warm, jitter, coalesce, cap the misses.
• Caching personal responses publicly. One missing private can show one customer’s address to the next.
• No plan for the cache going down. If it takes the database with it, the cache is a single point of failure.

Cheat sheet

• Database reads = reads × (1 − hit rate); 95% → 99% is five times less load.
• Worst-case CDN staleness = max-age + stale-while-revalidate.
• Redis reads for a hot key behind L1 ≤ app servers ÷ L1 TTL.
• Jitter spreads expiries: 50,000 keys over 60 s ≈ 830 misses/s.

How to say it in the interview

“Reads outnumber writes 500 to 1 with a 200 ms p99 target, so this is a read-scaling problem. First, each page becomes two indexed queries; one node then serves about 5,000 views a second. We need 100,000, so I add cache-aside Redis holding the assembled product, with a jittered five-minute TTL and a delete after each committed write. At a 95% hit rate the database sees 5,000 misses a second; at 99%, 1,000, which two replicas handle. Anonymous pages go through the CDN with a 30-second max-age and stale-while-revalidate. For the 12:00 hero deal I pre-warm and add a one-second L1 with single-flight. Staleness: up to a minute on the product page, none at checkout, and your own edits show at once because your reads go to the primary for five seconds after a write.”

Why not just add more read replicas? Each is a full database at full price that replays every write, and Aurora caps a cluster at 15. The example would need 20 to 29 to re-answer the same few thousand pages; a cache serves that hot set from memory for far less. Replicas are for misses and failover.

How do you keep the cache consistent with the database? Commit to the database, then delete the key, with a TTL as backstop. Use leases or versioned keys against the stale-set race, and at scale drive deletes from the commit log, as Facebook’s mcsqueal did.

What happens if the cache cluster goes down? The database would see 100,000 requests a second against a budget of 5,000. Cap the misses allowed through and shed the rest, serve stale from L1, fail over to a small spare pool, and test the cache-off path.

A user changes their address but still sees the old one. Why? A lagging replica or a stale cache entry. Delete the key on write, and send that user’s reads to the primary for a few seconds, or only to replicas that have replayed past their write’s log position.

How do you handle a celebrity post or a hot product? Sharding doesn’t help, since one key lives on one shard. Add a short L1, copy the key across shards, coalesce misses, serve stale while one caller refreshes, and pre-warm. For feeds, merge celebrity posts at read time instead of fanning them out.

Where to go next

• Caching strategies: cache-aside versus read-through and write-through.
• Hot keys, cache stampede and invalidation: how caches fail under skew.
• Content delivery network: cache keys, purging and origin shielding.
• Distributed caching: sharding and resizing a cache cluster.
• Database indexes: the Level 1 work every later rung depends on.
• Data replication: where replica lag comes from.
• Database sharding: partitioning when one cluster is not enough.
• Event sourcing and CQRS: read models built from a change stream.
• Scaling writes: the companion playbook for write-heavy systems.

References and further reading

1. Scaling Memcache at Facebook (Nishtala et al., NSDI 2013)usenix.orgLook-aside caching with delete-on-write, leases against stale sets and thundering herds (17K/s down to 1.3K/s), hot keys, Gutter pools and commit-log invalidation (mcsqueal).
2. Caching challenges and strategies (Amazon Builders’ Library)aws.amazon.comWhen a cache helps, local versus external caches, cache addiction, soft and hard TTLs, negative caching and request coalescing.
3. Amazon Aurora User Guide: Replication with Amazon Auroradocs.aws.amazon.comUp to 15 Aurora Replicas per cluster, replica lag usually well under 100 ms and rising with write load, and cross-Region read copies.
4. PostgreSQL documentation: Hot Standbypostgresql.orgRead-only queries on a standby, and why its data is eventually consistent with the primary.
5. PostgreSQL documentation: System administration functions (recovery information)postgresql.orgpg_current_wal_lsn and pg_last_wal_replay_lsn, the building blocks of an LSN read-your-writes token.
6. PostgreSQL documentation: Index-only scans and covering indexespostgresql.orgHow an INCLUDE index answers a query without touching the table, and when it does not help.
7. Redis documentation: Redis benchmarkredis.ioSample single-instance throughput (about 180,000 SETs per second without pipelining) and the 4 KB × 100k/s ≈ 3.2 Gbit/s bandwidth example.
8. Redis documentation: Client-side caching referenceredis.ioServer-assisted invalidation for in-process caches (tracking and broadcasting modes).
9. RFC 9111: HTTP Cachingrfc-editor.orgShared versus private caches, the private and s-maxage directives, and when a cache may serve stale responses.
10. RFC 5861: HTTP Cache-Control Extensions for Stale Contentrfc-editor.orgstale-while-revalidate and stale-if-error, with the max-age plus window arithmetic for worst-case staleness.