No caching

No caching means fetching or recomputing the same data from its origin (a database, a partner API, a disk, an expensive function) on every request, even though it changes far less often than it is read. Every request pays the origin’s full latency, and the origin carries load in proportion to page views rather than to how much data actually exists. The fix is a cache with a deliberate freshness rule, plus a plan for the day the cache isn’t there.

Picture it first: the enquiry counter

A busy station has one enquiry window. Every passenger asks the same few questions: which platform is the Rajdhani on, is the Shatabdi late? For each passenger the clerk phones the control room, waits on hold and repeats the answer. Two hundred passengers means two hundred calls to a control room that is also trying to run the trains.

A smarter clerk writes the answers on a whiteboard: “Rajdhani, platform 3 (as of 10:05)”. She phones only when an entry is ten minutes old, or when the control room rings her to say a platform changed. The control room now gets one call per train every ten minutes, however many passengers arrive.

Two new problems appear. A passenger sometimes reads a platform that changed four minutes ago. And if someone wipes the board at the busiest hour, the control room gets two hundred calls in a minute.

Map it back:

• The control room is the origin: your database, a partner API, a slow computation.
• A phone call is an origin round trip, paid for in latency and in load.
• The whiteboard is the cache; “as of 10:05” is its TTL; the control room ringing back is invalidation.
• The wiped board is a cache flush or outage: every read lands on the origin at once.

The arithmetic of a hit ratio

Two formulas carry this whole topic. With R reads per second and a hit ratio h (the share of reads the cache answers):

• origin load = R × (1 − h)
• mean latency ≈ h × cache latency + (1 − h) × origin latency

Take a product page during a Flipkart-style sale: 50,000 product reads a second, a PostgreSQL lookup that takes 20 ms end to end, and a Redis hit that takes 1 ms.

• No cache: the database serves all 50,000 reads a second, and every request waits 20 ms.
• 95% hits: 50,000 × 0.05 = 2,500 reads/s reach the database. Mean latency is 0.95 × 1 + 0.05 × 20 = 0.95 + 1.0 = 1.95 ms, a tenth of before.
• 99% hits: 50,000 × 0.01 = 500 reads/s.

Moving from 0% to 90% removes 45,000 reads a second. Moving from 95% to 99% removes only 2,000, yet it divides the database’s load by five. You provision the origin for the miss rate, 1 − h, so each step towards 100% is worth more than it looks.

That bottom row is the formula’s dark side. If the cache disappears, origin load jumps from R × (1 − h) back to R, a multiplier of 1 ÷ (1 − h). At 95% hits, 2,500 reads a second become 50,000: 20×. Amazon’s Builders’ Library describes the trap: a team adds a cache, the dependency shrinks its fleet because load fell, and the service is now “addicted” to a cache it can’t survive losing.

Two refinements keep the maths honest. A miss also pays for the failed cache lookup, so strictly it costs 21 ms and the mean at 95% is 1 + 0.05 × 20 = 2.0 ms; a cache with a low hit ratio can make requests slower. And a mean hides the tail: while more than 1% of reads miss, p99 is still a miss.

Symptoms you’d see

• A latency floor at origin speed. p50 never drops below the database or API round trip. A cached service has a two-humped histogram, a tall peak at cache speed and a small one at origin speed; one hump is the giveaway.
• A busy read-heavy database. Primary CPU and read IOPS follow page views while writes are a trickle.
• A few cheap statements top calls. In pg_stat_statements, a primary-key lookup leads by calls: millions a day, one row each, well under a millisecond. Nothing is slow; there is just a lot of it.
• Throughput that stops at the origin. Load tests plateau at whatever the database can serve, and more app pods don’t move the ceiling. Azure’s uncached sample showed exactly that.
• Bills and 429s from partners. Geocoding the same restaurant addresses, fetching the same exchange rates, resolving the same IFSC codes: each request calls a metered or rate-limited API. You pay per call and hit the partner’s quota at peak.
• The origin’s bad minute becomes yours. A 30-second database failover takes down product pages whose content hasn’t changed in a week, because nothing absorbs the gap.
• Spikes land on the database directly. At the 10 am tatkal opening or a sale’s first minute, 10× traffic on a few pages becomes 10× database load in seconds, faster than replicas or autoscaling can react.

Why it happens

It was fine at launch. With a few hundred users the database answers in 2 ms, and a cache looks like premature optimisation. Nobody revisits that when traffic grows a thousandfold.

Fear of stale data. A team worries that someone will see an old price, so it caches nothing, even data that is fine a minute old. That trades a rare, bounded staleness for a permanent cost on every request.

Nobody owns invalidation. Five services write to the products table. Whoever adds a cache must make every writer clear it, so nobody does.

The abstraction hides the network. product.category.name in an ORM, or rates.get("USD") in a client library, reads like a field access. Code review sees a getter, not a round trip.

“Our data is too dynamic.” Some of it is. Stock changes every second, but a product’s name, images and description change monthly, and GST rates change on budget day. Calling the whole entity dynamic because one field is means caching none of it. Split it: cache the slow fields, fetch the fast one.

The database’s own cache feels like enough. PostgreSQL may serve 99.9% of blocks from its buffer cache, so “it’s already cached”. Yet every query still crosses the network, takes a connection, and gets parsed, planned and executed. The buffer cache saves disk reads, not work.

How to detect it

Compare reads with writes

A table read 500 times per write is a strong candidate; one at 2:1 isn’t. PostgreSQL keeps per-table counters:

-- Rows read versus rows written per table (cumulative since the last stats reset).
SELECT relname,
       seq_tup_read + coalesce(idx_tup_fetch, 0) AS rows_read,
       n_tup_ins + n_tup_upd + n_tup_del         AS rows_written,
       round((seq_tup_read + coalesce(idx_tup_fetch, 0))::numeric
             / nullif(n_tup_ins + n_tup_upd + n_tup_del, 0), 1) AS read_write_ratio
FROM pg_stat_user_tables
ORDER BY rows_read DESC
LIMIT 10;

The counters are cumulative, so sample twice and subtract to get rates.

Find the calls that repeat

Rank statements by how often they run:

SELECT calls,
       round(mean_exec_time::numeric, 2) AS mean_ms,
       rows / nullif(calls, 0)           AS rows_per_call,
       left(query, 70)                   AS query
FROM pg_stat_statements
ORDER BY calls DESC
LIMIT 10;

pg_stat_statements replaces constants with placeholders, so it shows which statement repeats, not whether the same parameters do. Count those in the application: log the key you would cache under (product id, URL, API arguments) and compute the repeat factor, total lookups ÷ distinct keys.

Say you see 3,000,000 product lookups in a minute across 12,000 distinct products: a repeat factor of 250. With a 5-minute TTL, if the same products stay popular, each loads at most once per 5 minutes: 12,000 ÷ 300 s = 40 loads a second against 50,000 reads, a ceiling near 99.9% hits. Evictions, writes and the long tail pull the real figure down, but a factor in the hundreds says a cache will pay. Access logs give the same answer for HTTP:

# Most-requested paths in the last log file: a short, steep list means repetition.
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -20

Check hit ratios and the edge-to-origin ratio

If a cache exists, measure it. Redis reports keyspace_hits and keyspace_misses in INFO stats; the hit ratio is hits ÷ (hits + misses). A CDN reports a hit ratio per path. Then divide origin requests per second by edge requests per second: a ratio near 1.0 means nothing between them absorbs reads. No hit-ratio metric anywhere usually means there is no cache to measure.

Look at latency and headers

If p50 sits at origin latency, nothing is answering from memory. Then check what your API tells browsers and CDNs:

curl -sI https://shop.example.in/api/products/1042 | grep -iE 'cache-control|etag|age'

Empty output means you never told any cache it may reuse the response. That isn’t the same as “not cached”: RFC 9111 lets caches guess a lifetime for responses without explicit freshness, typically 10% of the time since Last-Modified. Caching you didn’t choose is worse than either choice.

Telling it apart from look-alikes

They combine. A loop that reads the same tax table once per cart line is chatty I/O and no caching; batch the loop and cache the table. A busy database recomputing the same report for every viewer is no caching for a computation.

How to fix it

Cache data that is read far more often than it changes, shared across many requests, and safe to serve a little old. For each candidate, write down the staleness the product can accept: that number becomes the TTL. Then pick a layer: the browser and CDN for public HTTP responses, a shared cache such as Redis for entities, process memory for tiny reference data.

Cache-aside with a TTL and jitter

Before: every page view queries the database, though prices change once a day.

from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool

pool = ConnectionPool("dbname=shop", kwargs={"row_factory": dict_row})

def get_product(product_id: int) -> dict | None:
    with pool.connection() as conn:
        return conn.execute(
            "SELECT id, name, brand, price_paise, image_url FROM products WHERE id = %s",
            (product_id,),
        ).fetchone()

After: look in Redis first, load from the database on a miss, and store the result with a TTL.

import json, random
import redis

cache = redis.Redis(host="cache", port=6379, socket_timeout=0.05)
TTL_S, NEG_TTL_S, MISSING = 300, 30, "__missing__"

def get_product_cached(product_id: int) -> dict | None:
    key = f"product:v3:{product_id}"          # version the format in the key
    try:
        raw = cache.get(key)
    except redis.RedisError:
        raw = None                            # cache trouble: treat as a miss
    if raw is not None:
        value = json.loads(raw)
        return None if value == MISSING else value

    product = get_product(product_id)         # the original query
    try:
        if product is None:                   # negative cache: remember "no such id"
            cache.set(key, json.dumps(MISSING), ex=NEG_TTL_S)
        else:                                 # jitter spreads expiries apart
            cache.set(key, json.dumps(product), ex=TTL_S + random.randint(0, 60))
    except redis.RedisError:
        pass                                  # the cache is best effort
    return product

The details matter. JSON keeps values readable by any service. The v3 in the key lets a new format start clean instead of misreading old entries; Amazon advises treating the cache format with the care of a persistent store. A 50 ms socket timeout keeps a sick cache from adding seconds to every request. Jitter matters because keys loaded together, after a deploy or a flush, would otherwise expire together and come back as one wave of misses.

You pay with staleness of up to 6 minutes, memory (12,000 products × 2 KB ≈ 24 MB, which is small), a new system to run, monitor and size, and a cold start whenever the cache is flushed. The pattern itself is covered in caching strategies, and what to drop when memory fills in cache eviction policies.

Invalidate on write, and keep the TTL as a safety net

A TTL alone means a price edit takes up to 6 minutes to show. For data people edit and expect to see, delete the key when you write:

def update_price(product_id: int, price_paise: int) -> None:
    with pool.connection() as conn:       # commits when the block exits cleanly
        conn.execute("UPDATE products SET price_paise = %s WHERE id = %s",
                     (price_paise, product_id))
    cache.delete(f"product:v3:{product_id}")

Deleting beats overwriting: two writers racing to SET can leave the older value behind, while a delete makes the next reader load the truth. A narrow race remains: a reader that missed before your write can store the old row after your delete; Facebook’s memcache leases exist partly to reject such stale sets. Keep the TTL, so a missed invalidation heals within minutes. When many services write the table, drive deletes from change data capture instead of trusting every writer.

You pay with a delete on every write path, and staleness up to the TTL whenever one path forgets.

Read-through and in-process caches

Read-through moves the loading logic into the cache layer: callers ask the cache, and the library or proxy calls your loader on a miss. Every caller behaves the same and the app code shrinks. Amazon notes the cost: an inline cache sits on the availability path, and callers can’t route around it when it fails.

In-process caches suit tiny, hot, slow-changing data: GST slabs, the city list, feature flags. A hit is a memory read with no network at all. You pay three ways. Pods disagree for up to the TTL. Origin load scales with fleet size: 200 pods refreshing a table once a minute is 200 ÷ 60 ≈ 3.3 reads a second, trivial for one table but not for a million keys. And every deploy starts cold. A small in-process layer in front of Redis also gives you something to serve if Redis goes down.

Cache “not found” too

Lookups by user-supplied ids can miss forever when the answer is “no such thing”: a pincode you don’t deliver to, an expired coupon code, a bot walking product ids from 1 to 10 million. Without an entry for the absence, each one goes to the database every time. Amazon has seen failures to cache negative results raise failure rates in production. The code above stores a sentinel with a 30-second TTL. You pay with a newly created item staying invisible for up to that TTL.

Coalesce misses so a hot key loads once

When a hot key expires, every request that sees the miss loads it at the same moment: a stampede. Coalescing lets one caller load while the rest wait for its answer. In Go, singleflight does it per process:

var group singleflight.Group // golang.org/x/sync/singleflight

func Product(ctx context.Context, id string) (*Product, error) {
	if p, ok := cache.Get(ctx, id); ok {
		return p, nil
	}
	// Concurrent misses for the same id share one database call.
	v, err, _ := group.Do("product:"+id, func() (any, error) {
		p, err := db.LoadProduct(ctx, id)
		if err == nil {
			cache.Set(ctx, id, p, 5*time.Minute)
		}
		return p, err
	})
	if err != nil {
		return nil, err
	}
	return v.(*Product), nil
}

With 200 pods, a hot key now costs at most 200 concurrent loads instead of one per request. Across pods you need a lock or a lease. Facebook’s memcache hands out one lease token per key every 10 seconds and tells other clients to wait briefly; on keys prone to herds, peak database load fell from 17K to 1.3K queries a second. You pay with waiters sharing the leader’s slowness or error, including its cancelled context if you pass the caller’s. Serving the stale value during the refresh avoids the wait. More in hot keys, cache stampede and invalidation.

Let HTTP do the caching

For public GET responses, the cheapest cache is one you don’t run: the browser’s and the CDN’s. Tell them what they may do:

GET /api/restaurants/4821/menu HTTP/1.1
Host: api.example.in

HTTP/1.1 200 OK
Cache-Control: max-age=60, stale-while-revalidate=30, stale-if-error=600
ETag: "menu-4821-v117"
Content-Type: application/json

GET /api/restaurants/4821/menu HTTP/1.1
Host: api.example.in
If-None-Match: "menu-4821-v117"

HTTP/1.1 304 Not Modified
Cache-Control: max-age=60, stale-while-revalidate=30, stale-if-error=600
ETag: "menu-4821-v117"

Each directive earns its place:

• max-age=60 (RFC 9111): for 60 seconds any cache may answer without asking you. s-maxage sets a separate lifetime for shared caches such as CDNs.
• stale-while-revalidate=30 (RFC 5861): for 30 seconds more, a cache may serve the stale copy at once while it revalidates in the background.
• stale-if-error=600: if your origin answers 500, 502, 503 or 504, caches may serve the old copy for up to 10 minutes.
• ETag and If-None-Match (RFC 9110): when the copy is stale, the cache asks “still v117?”, and you answer 304 Not Modified with no body, skipping building and sending an 18 KB menu.

The stale extensions come from an informational RFC, so check what your CDN honours. You pay with losing the ability to recall a response already in a million browsers. Keep max-age short for things that change, and give versioned assets (app.3f9c1e.js) a year.

Put a CDN in front of static and semi-static content

Images, scripts and fonts behind versioned URLs never change, so cache them for a year. Category pages and product listings change a few times a day: cache them at the edge for a minute with s-maxage, and purge on publish. During a sale, a 60-second edge TTL means each edge location fetches each page at most once a minute, however many shoppers arrive. With 50 locations and 2,000 hot pages that is at most 50 × 2,000 ÷ 60 ≈ 1,670 origin requests a second, whether page views run at 20,000 or 200,000.

You pay with purge delays, a CDN bill, and cache-key design: query strings, cookies and Vary headers can split one page into thousands of entries. Personalised pages need splitting into a cacheable shell and a small per-user API call. See content delivery network.

Plan for the day the cache isn’t there

Size the origin for losing part of the cache, and shed load beyond that. Say the database handles 8,000 reads a second and the cache is spread across 10 shards:

• Normal: 2,500 misses a second.
• One shard lost: its 5,000 reads a second had 4,750 hits, which now miss: 2,500 + 4,750 = 7,250/s. Under capacity.
• Two shards lost: 2,500 + 9,500 = 12,000/s, 50% over capacity.
• All lost: 50,000/s, over six times capacity.

The defences, roughly in the order you’d build them:

• Fail open, but capped. On a cache error, go to the origin through a concurrency limit sized to what it can take, here about 8,000 reads a second, and shed the rest with a fast 503 or a degraded page. Azure suggests a circuit breaker so the origin isn’t swamped; see backpressure and load shedding.
• Serve stale. Amazon uses a soft TTL that triggers a refresh and a hard TTL that bounds how long old data may be served when the refresh fails. stale-if-error does the same at the edge.
• Layer the caches. A small in-process cache in front of Redis keeps the hottest keys alive through a Redis outage.
• Keep spare cache capacity. Facebook’s Gutter pool, about 1% of a cluster’s memcached servers, takes over a failed server’s keys with short TTLs instead of sending misses to the database; it cut client-visible failures by 99%.
• Warm up first. Facebook fills a cold cluster’s cache from a warm cluster rather than the databases, reaching full capacity in hours instead of days.
• Test with the cache off, as Amazon does, to prove the safeguards work.

Without these, a cache outage plus client retries becomes a retry storm. You pay with shed requests, stale pages and idle spare capacity.

Compare the fixes

To verify, rerun the load test: origin ÷ edge requests should fall from about 1.0 towards 0.05, and p50 to cache speed. Then rerun it with the cache disabled and confirm the cap holds.

When it’s fine

• Strong consistency at decision time. A UPI-style debit must read the real balance, and a tatkal booking must check the seat inside the transaction. Cache the availability display, not the commit check.
• Unique-per-request or write-heavy data. One-time OTPs, long-tail search queries and live location pings are rarely read twice; a cache would mostly miss and add a hop.
• Low traffic. Twenty reads a second against a database that handles 5,000 doesn’t need another system to run.
• Computing is cheaper than caching. A 50 µs calculation beats a 1 ms network round trip to a cache.
• Sensitive per-user data, unless the key includes the user and the response is marked private. A shared cache keyed only by URL is a data leak waiting to happen.

Cheat sheet

How it shows up in interviews

Caching is expected in almost every read-heavy prompt: a URL shortener’s redirects, a product page in a flash sale, a restaurant menu, a news feed, a typeahead. Leaving it out is the anti-pattern; adding it without numbers or a failure plan is the next mistake.

The probes: “Your database is at 90% CPU and 98% of queries are reads. What do you do?” “What hit ratio do you need?” “What happens when Redis restarts during the sale?” “How stale can this be?” Answer with the arithmetic first, then the TTL tied to a business tolerance, then the cliff.

How to say it in the interview

“Product data is read thousands of times per change, so querying the database on every view makes it scale with page views. I’d add cache-aside: Redis keyed by product id, JSON values, a 5-minute TTL with jitter, a delete after each committed write, and a short negative TTL for missing ids. At 50,000 reads a second and 95% hits, the database sees 2,500, and the mean drops from 20 ms to about 2. Public pages get Cache-Control and an ETag so the CDN absorbs most traffic, and hot keys refresh through single-flight. Then I size for the cliff: without Redis the database would see 20× its load, so misses go through a cap sized to the database, and we serve stale or shed the rest.”

How do you choose a TTL? From the business, not the code: how stale can a price, menu or profile be before it harms someone? That number is the ceiling. Add jitter, invalidate on write for things users edit, and check the hit ratio to see whether the TTL is too short.

How do you keep the cache consistent with the database? Write the database first, then delete the key; never update the cache in place from several writers. Keep a TTL to heal missed deletes, and for many writers drive invalidation from change data capture. If a read must reflect the latest write, read the database for that path.

What happens if Redis goes down? The origin sees 1 ÷ (1 − h) times its normal load: 20× at 95%. So the app fails open through a concurrency cap, serves stale where it can, keeps a small in-process cache for the hottest keys, and sheds the rest. We test this with the cache disabled before the sale, not during it.

Your hit ratio is 60%. What do you check? Whether keys contain something unique, like a session id or timestamp; whether the TTL is shorter than the gap between reads; evicted_keys, which says the cache is too small; and whether writes invalidate too broadly. The repeat factor tells you what ratio is possible at all.

Where to go next

• Caching strategies: cache-aside, read-through, write-through: the patterns in depth.
• Hot keys, cache stampede and invalidation: leases, locks and early refresh.
• Cache eviction policies and distributed caching: what stays, and how a cache spreads across nodes.
• Content delivery network and HTTP and HTTPS: caching at the edge and the headers behind it.
• Design a distributed cache: building the cache itself.
• Chatty I/O, busy database, extraneous fetching and retry storm: the neighbouring anti-patterns.
• Database scaling and scaling reads: replicas and other ways to carry read load.

References and further reading

1. No-Caching antipattern (Azure Architecture Center)learn.microsoft.comCauses of the anti-pattern, detection steps, failing open to the data store with a circuit breaker, and the load test whose throughput plateaued without a cache.
2. Cache-Aside pattern (Azure Architecture Center)learn.microsoft.comLoad on demand, invalidate on write, and why you update the store before deleting the cached item.
3. Caching challenges and strategies (Amazon Builders’ Library)aws.amazon.comServices “addicted” to their cache, local versus external caches, soft and hard TTLs, negative caching, request coalescing and testing with the cache disabled.
4. Scaling Memcache at Facebook (Nishtala et al., NSDI 2013)usenix.orgDemand-filled look-aside caching, leases that cut a herd’s peak database load from 17K/s to 1.3K/s, the Gutter pool and cold-cluster warm-up.
5. RFC 9111: HTTP Cachingrfc-editor.orgFreshness, max-age, s-maxage, private, no-store, heuristic freshness, validation, and why Set-Cookie does not prevent caching.
6. RFC 9110: HTTP Semanticsrfc-editor.orgETag, If-None-Match and the 304 Not Modified response that carries no content.
7. RFC 5861: HTTP Cache-Control Extensions for Stale Contentrfc-editor.orgstale-while-revalidate and stale-if-error, with worked examples of each window.
8. Redis documentation: INFOredis.iokeyspace_hits, keyspace_misses and evicted_keys, the counters behind a cache hit ratio.
9. PostgreSQL documentation: pg_stat_statementspostgresql.orgPer-statement calls, rows and mean_exec_time, and how constants are normalised away.
10. PostgreSQL documentation: The Cumulative Statistics Systempostgresql.orgpg_stat_user_tables counters (idx_tup_fetch, seq_tup_read, n_tup_ins, n_tup_upd, n_tup_del) used for read-to-write ratios.