Design a web crawler

A web crawler starts from a handful of URLs, downloads each page, pulls out the links, and queues the ones it has never seen — then repeats, forever. The loop fits on a whiteboard; at a billion pages a day it becomes a real design problem. You must fetch 11,600 pages a second without hammering a small restaurant’s single-server website, remember 100 billion URLs, notice that a quarter of what you download is a copy, escape a calendar that offers “next month” until the year 9999, and revisit a cricket-final live blog every few minutes while leaving a railway’s refund-rules page alone for weeks. This is the component that feeds a web search engine’s index.

Understand the problem

A crawler is the first stage of search and nothing more: it finds and downloads pages, and hands them on. It does not rank, index or answer queries — that is the search engine built on top (see also search end to end). Say that out loud early, because the interesting trade-offs here are different: throughput against politeness, memory against accuracy, and freshness against bandwidth.

Functional requirements

1. The system should crawl continuously from seeds. Starting from seed URLs, sitemaps and feeds, it downloads pages, extracts links and queues new URLs — a never-ending loop that mixes new pages with revisits.
2. The system should be polite. It obeys robots.txt, keeps the load on every host (and every server IP) low, identifies itself, and backs off as soon as a site signals stress.
3. The system should store each new page once and hand it to indexing, with its metadata (status, headers, fetch time, content hash) and outgoing links. Known URLs and content it already has are not fetched or stored again.
4. The system should keep important pages fresh. Each page is revisited at a rate that matches how important it is and how often it changes.

Non-functional requirements

• Throughput. 1 billion fetches a day, new pages and revisits together: about 11,600 pages a second, with headroom for 2× peaks.
• Politeness. Zero robots.txt violations. By default one request in flight per host and several seconds between requests; a 429 or 503 slows the host on its very next fetch.
• Freshness. The top ~100 million pages (home pages, news sections) revisited at least daily, breaking-news fronts every few minutes, the long tail within weeks. A new link on an important page is fetched within about an hour.
• Scale. 100 billion known URLs and growing; about 50 million distinct hosts touched a day.
• Durability and availability. Fetched content is stored durably. A crashed node loses at most minutes of discovered URLs and pauses only its own hosts, never the whole crawl.
• Robustness and cost. Malformed HTML, 2 GB “pages”, servers that answer one byte a second and infinite URL spaces must all be survivable. Bandwidth and storage dominate the bill, so unchanged pages should not be downloaded in full.

Out of scope

• Indexing, ranking and serving queries — the search engine design covers them.
• Full JavaScript rendering, beyond a short deep dive on when and how to render.
• Images, video and PDFs, other than recognising and skipping them.
• The deep web: forms, logins and paywalls.
• Legal workflows such as takedowns, beyond a per-host block list.
• Small, feed-driven crawls such as polling news sites for an aggregator — see design a news aggregator.

The set-up

Plan the approach

Build the loop one requirement at a time until it works on one machine, then spend most of the interview in the deep dives. Name the shape early: “a frontier that decides what to fetch, fetchers that fetch politely, a processor that stores pages and extracts links, and two ‘have we seen this?’ checks — one for URLs, one for content.”

Core entities

• URL record. The normalised URL and its 64-bit fingerprint, host, priority, state, last fetch time and status, ETag and Last-Modified validators, content checksum and simhash, a short change history (visits and changes seen) and the time it is next due.
• Host record. Hostname and registered domain, resolved IPs with their DNS TTL, the parsed robots.txt rules and when they were fetched, the current delay between requests, the earliest next fetch, error and back-off state, and a page budget.
• Page. The compressed response — headers and body — stored in a large append-only file, plus the final URL after redirects.
• Fetch event. What the crawler tells the indexer: “this URL was fetched, here is where the bytes are, here is what changed”.
• Seed and sitemap. Entry points with a priority and a source.

API

The crawler has no public users; its API is internal. Three surfaces matter: how work gets in, how operators steer it, and what comes out.

Work in. Seeds, sitemaps and urgent recrawls carry a priority and a reason, so the frontier can rank them.

POST /v1/crawl-requests
Content-Type: application/json

{
  "urls": [
    "https://news.example.in/sitemap-news.xml",
    "https://shop.example.com/"
  ],
  "priority": "high",
  "reason": "sitemap_lastmod"
}

{ "accepted": 2, "duplicates": 0, "rejected": [] }

Steering. Operators can slow, pause or block a host — for example after a site owner writes in.

PUT /v1/hosts/shop.example.com/policy
Content-Type: application/json

{ "max_parallel": 1, "min_delay_ms": 10000, "paused_until": null, "blocked": false }

Work out. Every fetch publishes one event to a pages.fetched topic. The indexer, the link-graph builder and the analytics jobs all consume it; none of them call the crawler.

{
  "url": "https://news.example.in/cricket/final-report",
  "final_url": "https://news.example.in/cricket/final-report",
  "status": 200,
  "fetched_at": "2026-10-27T10:42:07Z",
  "content_type": "text/html; charset=utf-8",
  "changed": true,
  "checksum": "b7e1f0c4a92d3e58",
  "simhash": "9f3a52c107e4b6d8",
  "duplicate_of": null,
  "blob": "crawl-2026-10/part-00412.warc.gz@88213440",
  "out_links": 57
}

On the wire, the crawler identifies itself. RFC 9309 says the robots.txt product token should appear in the User-Agent header, and the header should describe the crawler’s purpose:

GET /cricket/final-report HTTP/1.1
Host: news.example.in
User-Agent: Mozilla/5.0 (compatible; HandbookBot/1.0; +https://bot.example.org/about)
Accept-Encoding: gzip, br
If-None-Match: "a41f"
If-Modified-Since: Mon, 26 Oct 2026 18:05:00 GMT

A site can then write user-agent: HandbookBot in its robots.txt, and the owner knows where to complain.

Back-of-envelope estimates

These numbers decide how many machines you need and which data structures fit in memory. (For the method, see back-of-envelope estimation.)

Fetch rate. 1,000,000,000 ÷ 86,400 s ≈ 11,600 pages/s, about 23,000/s at peak. For scale: Common Crawl’s October 2026 archive — 2.16 billion pages in two weeks — works out to about 1,800 pages/s, and IRLbot averaged 1,789 pages/s on one server in 2007. We are roughly six times either.

Bandwidth. Assume 100 KB of uncompressed HTML per page: IRLbot measured 25 KB in 2007, and Common Crawl’s archive is about 190 KB per captured page today, headers and all. 11,600 × 100 KB ≈ 1.16 GB/s ≈ 9.3 Gbps on average, about 19 Gbps at peak. gzip and Brotli shrink the wire traffic, but parsers still see the uncompressed bytes.

Storage. 1 billion × 100 KB = 100 TB a day; at IRLbot’s measured 5:1 compression, ≈ 20 TB a day, ≈ 600 TB per 30-day month, less the duplicates we skip.

Links. IRLbot found about 59 crawlable links per page; call it 50. 50 billion links a day ÷ 86,400 ≈ 580,000 URL-seen checks/s — the hottest loop in the system.

URL-seen set. Plan for 100 billion URLs and growing: IRLbot discovered 41.5 billion unique URLs while downloading 6.4 billion pages, about 6.5 new URLs per page. As strings (70 bytes on average, plus overhead) that is about 10 TB; as 8-byte fingerprints 800 GB; as a Bloom filter at 10 bits per URL 125 GB.

URL metadata. About 200 bytes per record × 100 billion = 20 TB, in a sharded key-value store such as Cassandra or a Bigtable-style table.

DNS. IRLbot made one lookup per 29 connections (260 million for 7.6 billion), so a cache turns 11,600 potential lookups a second into 11,600 ÷ 29 ≈ 400/s.

robots.txt. One fetch per host per day — RFC 9309 says a cached copy should not be used for more than 24 hours: 50 million ÷ 86,400 ≈ 580/s, about 5% extra requests.

Concurrency. At about 1.5 s per fetch (DNS, connection set-up, server think time, download), Little’s law gives 11,600 × 1.5 ≈ 17,000 open connections — an async I/O problem, not a CPU one. And if each host is fetched at most once every 10 s, 11,600 × 10 ≈ 116,000 hosts must be ready in rotation just to keep the fetchers busy.

Fleet. Budget 250 pages/s per node, since nodes also extract links and check duplicates: 11,600 ÷ 250 ≈ 47, doubled for peaks and failures ≈ 100 nodes. Each handles about 116 pages/s on average, 5,800 URL checks/s, 175 open connections, 1,200 hosts in rotation and 1 billion URL fingerprints (8 GB).

High-level design

Build the loop one requirement at a time. By the end of this section it works on one cluster at modest scale.

The system should crawl continuously from seeds

The algorithm has not changed since the first crawlers. Mercator’s authors list the parts every crawler needs: a URL frontier of URLs to download, a DNS resolver, an HTTP fetcher, a link extractor, and a test for “have we seen this URL before?”. Add a content store for the bytes and you have the loop.

1. Seeds — popular home pages, curated lists, sitemaps — go into the frontier.
2. The fetcher takes the next URL, resolves the host name and sends a GET.
3. The parser reads the HTML. The raw bytes go to the content store; the links come out.
4. Each link is made absolute (Mercator saw about 80% of links written as relative paths), filtered — http and https only, no .jpg or .zip, not too long — and checked against the URL-seen set.
5. Only never-seen URLs join the frontier.

With a FIFO frontier this is a breadth-first search of the web: fine for a few thousand pages, but rude, unprioritised and forgetful.

The system should be polite

Two rules make a crawler welcome. First, ask permission: before fetching anything from a host, fetch /robots.txt and obey the group that matches your product token. Second, go gently: never send a host more than it can comfortably serve.

A FIFO frontier breaks the second rule by accident. Links are mostly relative, so the queue fills with runs of URLs on the same host, and 500 workers pulling from it can hit a small site with dozens of simultaneous requests.

So split the frontier: one queue per host, a rule that a host has at most one request in flight, and a minimum gap between its requests. Put a robots.txt check, backed by a cache of parsed rules per host, in front of the queues.

Once cached, the robots.txt check is an in-memory longest-match lookup per URL. Mercator kept an LRU cache of 2¹⁸ (about 262,000) hosts’ rules; at 50 million hosts a day you shard the cache along with the hosts. What a 404 or a 503 on robots.txt means, and how the gap between requests adapts, are deep dives.

The system should store each new page once and hand it to indexing

A billion separate objects a day would be slow and expensive, since object stores charge per request. Instead, append compressed responses to large files — the WARC format used by web archives and Common Crawl — in blob storage, and keep a file-and-offset pointer per page.

Before storing, ask a second “have we seen this?” question — about the content. The same page appears under many URLs: http and https, with and without www, on mirror sites. Mercator fingerprinted every downloaded document and skipped any it had already seen; a study cited by the IRLbot authors found 25–30% of the pages a large search engine downloaded were exact copies.

The URL database holds one URL record per known URL, sharded by host so a host’s rows live together; it is the crawler’s memory between visits. The pages.fetched topic decouples everything downstream — the indexer can fall an hour behind and the crawler never notices. A log such as Kafka beats a plain message queue here, because several consumers read the same events at their own pace.

The system should keep important pages fresh

An index starts going stale the moment a page is fetched, so the crawl must loop over known pages too. Add a recrawl scheduler that scans the URL database for rows whose next_due has passed and feeds them into the frontier, and make each revisit cheap with a conditional GET: send the ETag as If-None-Match (and the Last-Modified date as If-Modified-Since), and a server whose page has not changed replies 304 Not Modified with no body.

You now have a working crawler: it discovers, fetches politely, stores pages once and revisits them — fine at a few hundred pages a second. The deep dives are about what breaks at 11,600, starting with the frontier, which now mixes new URLs and revisits.

Deep dives

How do we design the URL frontier for priority and politeness?

The frontier serves two masters. Priority says “fetch the most valuable URL next”. Politeness says “but not from a host you fetched two seconds ago”. A frontier that serves only one of them fails.

Bad: one big FIFO queue. Breadth-first order is simple, but a link from a spam farm waits in the same line as one from a national newspaper, host locality makes it rude, and it gets hijacked: IRLbot’s authors found that a breadth-first crawl eventually meets a quickly branching site that dominates the queue after three or four levels.

Good: one queue per host, served round-robin. Each host has at most one request in flight, so the bursts are gone — Mercator’s weak politeness guarantee — yet it still drew complaints, because a fast host got requests back to back with no pause. And a host with one important page and a million junk ones gets a million turns.

Great: two layers — priority in front, politeness behind. This is Mercator’s design, and it is still the standard shape.

• Front queues decide what matters. A prioritiser scores each URL — importance from the last index build, domain reputation, whether it came from a sitemap or feed, how overdue a revisit is, depth in its site — and puts it in one of k FIFO queues, F1 (highest) to Fk.
• Back queues decide who may go now. Each back queue holds URLs for exactly one host, and a table maps host → queue. Mercator ran about three times as many back queues as fetch threads.
• A min-heap of ready times, one entry per back queue, keyed by when that host may next be contacted. A worker pops the root, waits if needed, fetches the queue’s head URL, and pushes the queue back with a new time.
• Refills come from the front. When a back queue empties, the worker takes a URL from a randomly chosen front queue, biased towards high priority; if that URL’s host already has a back queue it joins it, otherwise the empty queue now belongs to the new host.

The front end decides the mix: high-priority URLs win most refills, yet low ones are never starved. The back end decides the timing. Per node it is small — about 1,200 hosts in rotation at our average rate, each a heap entry plus a short buffer. Mercator kept the bulk of each queue on disk and only its head and tail in memory, so billions of queued URLs cost a few gigabytes of RAM.

How long should a host wait? Mercator waited ten times as long as the last download from that host took: a server that answers in 200 ms is revisited after 2 s, one that takes 3 s waits 30 s. The rule adapts by itself — a struggling server slows down, so the crawler slows down. Add three more inputs:

• A floor and a ceiling. Never less than about a second, never more than a minute or two, so tiny sites are safe and slow sites still make progress.
• Back-off on distress. A 429 or 503 at least doubles the delay and honours Retry-After; a run of 5xx pauses the host. Google describes the same loop as a crawl capacity limit that rises while a site responds quickly and falls when it slows down or returns 5xx or 429.
• crawl-delay. It is not part of RFC 9309, and Google ignores it, but many site owners still write it. Honouring it, capped at a sane maximum, earns goodwill cheaply.

Be polite per server, not just per host. A shared-hosting machine can serve thousands of small sites from one IP address; each host is polite alone, yet together they can flatten it. IRLbot added a per-IP limit: by default 1 second between requests to a server and 40 seconds between requests to a host for low-ranked domains, both shortened for well-ranked ones, enforced with two heaps — one keyed by host, one by IP.

How do we know we’ve seen a URL before, at 100 billion URLs?

Two problems hide in this question. First, make “the same URL” mean the same string. Then make a set lookup affordable at 580,000 checks a second.

Normalise before you hash. Two spellings of one page must produce one fingerprint, or every later check is wasted. Apply the safe rules from the URI standard, then a few learned ones:

input   HTTPS://News.Example.IN:443/cricket/./live/../final?utm_source=wa&id=7#scorecard

1. lowercase the scheme and host       https://news.example.in:443/...
2. drop the default port               https://news.example.in/...
3. resolve "." and ".." segments       /cricket/final
4. drop the #fragment                  (browsers never send it)
5. strip known tracking parameters     utm_*, fbclid, session IDs
6. decode needlessly escaped chars     %7E → ~

output  https://news.example.in/cricket/final?id=7

Be conservative: http and https, www and the bare domain, trailing slashes and parameter order can all name different resources on some sites, so learn those equivalences per host from redirects and rel="canonical" links. Then hash the result to a 64-bit fingerprint, as Mercator did.

Bad: a hash set of URL strings. About 10 TB of RAM, growing by billions of URLs a week. On disk with a seek per lookup it is worse: Mercator measured about 8 ms per seek, which capped its crawl at 75 pages/s even with a cache absorbing five checks in six. IRLbot’s authors note that a seek costs 3–5 ms even with RAID, when their crawler needed a check every 5–10 µs.

Good: 64-bit fingerprints in RAM, sharded. 100 billion × 8 bytes = 800 GB, about 1.2 TB with hash-table overhead, so 12 GB on each of 100 nodes. (Mercator stored only 5 bytes per fingerprint and fitted 1 billion URLs in just over 5 GB.) Collisions are negligible: the expected number of colliding pairs is about n² ÷ 2⁶⁵ = 10²² ÷ (3.7 × 10¹⁹) ≈ 270, so about 270 URLs out of 100 billion are never crawled. What you pay: RAM in proportion to every URL you have ever discovered, a long reload after a restart, and no room for per-URL data.

Good, with a catch: a Bloom filter. A Bloom filter sets k bits per URL in an m-bit array. With 10 bits per URL and 7 hash functions, the false-positive rate is (1 − e^−0.7)⁷ ≈ 0.8%, and 100 billion URLs fit in 125 GB — 1.25 GB per node, ten times less than fingerprints. The catch is the direction of the error.

A false positive says “seen” for a URL you never fetched, so about 0.8% of genuinely new URLs are skipped — silently and permanently, since the same URL always hits the same bits. You cannot delete, list or attach data either, and the error climbs as the filter fills. A Bloom filter is a fine pre-filter in front of an exact store — its “definitely new” answers skip the expensive check — but a risky only check.

Great: check in batches on disk, with a small cache in front. Stop looking URLs up one at a time; sort them and merge. Mercator’s best version did this, and IRLbot generalised it as DRUM (Disk Repository with Update Management):

1. Fingerprint each URL and append it to one of k RAM buffers chosen by the fingerprint’s top bits; the URL text goes to a side file.
2. When a buffer fills, write it sequentially to its disk bucket.
3. Every few minutes, load one bucket at a time, sort it, and walk it alongside the sorted file of every fingerprint ever seen — a single sequential pass for all buckets. Fingerprints not found are new: add them to the file and send their URL text to the frontier.

Run the numbers for one node. At 5,800 checks a second, a 10-minute batch holds 5,800 × 600 ≈ 3.5 million fingerprints. Its 8 GB share of the seen file is read and rewritten in about 16 seconds at 1 GB/s. The same checks as 4 ms seeks take 14,000 seconds — almost four hours, far longer than the batch window. IRLbot’s model put DRUM thousands of times ahead of earlier disk methods at large crawl sizes, and in its 41-day run the URL check never became the bottleneck.

What you pay: new URLs wait minutes before reaching the frontier (fine for a crawler, not for an online database), and the code is more intricate. A small LRU cache of recent fingerprints in front catches repeats — IRLbot measured 60–73% hit rates once the cache held more entries than the links parsed between two visits to the same site. And because DRUM stores a value per key, IRLbot reused it for its robots.txt cache and domain reputation counts.

The recommendation: fingerprint normalised URLs and check them in batches against sorted files on local SSD, with an LRU cache — and optionally a Bloom pre-filter — in front. RAM stays flat while the URL set grows.

How do we avoid fetching and storing duplicate or near-duplicate content?

Normalisation cannot catch the same content under genuinely different URLs: mirrors, one wire-service story on fifty news sites, printer-friendly versions, pages that differ only in an ad slot or a “generated at 10:42” footer. Manku, Jain and Das Sarma give the rule for a search crawler: if a new page is a near-duplicate of one you already have, ignore it and its outgoing links, which probably lead to near-duplicates too.

Bad: no content check. If a quarter of downloads are exact copies, as the study cited by IRLbot found, a quarter of your storage and indexing budget buys nothing, and search results fill up with the same article five times.

Good: an exact checksum. Hash the body to 64 bits and look it up in a content-seen set — Mercator’s content-seen test: 8 bytes and one lookup per page. IRLbot’s authors note two blind spots: it misses pages that differ only in counters, timestamps or ads, and it can merge identical pages on two sites whose relative links point to different places — so record the URL as “a copy of X” rather than forgetting it.

Better: shingles. Split the text into overlapping runs of words (“shingles”), hash them, keep a small sample, and compare samples to estimate how much two pages overlap. Accurate, but bulky: Broder’s shingle-based fingerprints take 24 bytes per page, and finding every stored page that shares enough shingles with a new one is its own indexing problem.

Great: simhash, with an index for Hamming distance. Charikar’s simhash turns a page into a 64-bit fingerprint with an unusual property: similar pages get similar fingerprints. Extract weighted features (words and phrases, with stop words removed); hash each feature to 64 bits; for each bit position, add the feature’s weight if that bit is 1 and subtract it if it is 0; the sign of each sum is one bit of the fingerprint. A small edit nudges a few sums, so only a few bits flip. A cryptographic hash such as SHA-256 does the opposite: one changed byte flips about half the bits.

Manku and colleagues tested simhash on 8 billion pages and found that 64-bit fingerprints work, and that calling two pages near-duplicates when their fingerprints differ in at most 3 bits gives precision and recall both near 0.75. Fewer bits misses copies; more bits merges pages that are really different.

The hard part is the lookup: “is any stored fingerprint within 3 bits of this one?” Probing every variant takes 41,664 lookups (the ways to choose 3 of 64 bits). The trick is the pigeonhole principle: split the 64 bits into six blocks, and two fingerprints within 3 bits must agree exactly on at least three blocks. Keep several sorted copies of the table, each with a different choice of blocks moved to the front; probe each for an exact match on its leading bits, then check the few candidates. With 8 billion (2³⁴) fingerprints, the paper’s 20-table design returns about 8 candidates per probe; a 4-table design saves space but returns about 256,000. One table is 64 GB, under 32 GB compressed, and in batch mode 200 mappers check a million new fingerprints in under 100 seconds — about 10,000 pages a second, roughly our crawl rate.

In the pipeline, an exact match stops processing; a near-match records duplicate_of, skips storage and links, and lowers the URL’s recrawl priority. The paper notes that near-duplicate status often becomes one input to crawl priority rather than a hard yes or no — the safer default at 75% precision.

How do we distribute the crawl across 100 machines?

How do you split the work so that politeness, the URL-seen check and the robots.txt cache stay correct without a central bottleneck?

Bad: a central frontier service. One database holds the frontier and the URL-seen set and absorbs 580,000 checks and 11,600 pops a second, each pop needing a fleet-wide per-host lock — a hot spot and a single point of failure.

Also bad: partition by URL hash. Load spreads evenly, but each host’s URLs land on every node. Each node is polite on its own, so a site can be visited by 100 polite crawlers at once — 100 times the load you meant. Every node also fetches the same robots.txt and resolves the same host name.

Good: partition by host. Hash the hostname to choose its owner node. Everything about a host then lives on one machine: its back queue, its robots.txt, its DNS entry, its slice of the URL-seen set. Politeness becomes a local decision again. Mercator’s distributed mode split host names among its processes this way. After link extraction, a host splitter sends each URL to its owner. Most links stay home — Mercator found about 80% of links are relative, and IRLbot also saw over 80% of parsed URLs stay on the same server — so only about 20% travel: 580,000 × 0.2 ≈ 116,000 URLs/s, × ~100 bytes ≈ 11.6 MB/s ≈ 93 Mbps fleet-wide, about 1% of the page traffic. Mercator buffered outgoing URLs and sent them in batches; do the same, deduplicating and compressing each batch.

Great: host partitioning on a consistent-hash ring, with a durable exchange and checkpoints.

• A ring, not a modulus. With hash(host) mod 100, adding a 101st node moves almost every host; with consistent hashing and virtual nodes, about 1%. Keep membership in etcd or ZooKeeper with leases, so a dead node’s range is reassigned automatically.
• Partition by registered domain when budgets are per domain. IRLbot’s authors note that with per-domain budgets the split must be by pay-level domain (example.co.uk, not blog.example.co.uk). Politeness inside the node is still per host and per IP.
• Make the exchange a durable log such as Kafka, partitioned by owner. A restarted node replays from its last offset, and bursts meet backpressure instead of dropped URLs.
• Checkpoint often. Mercator checkpointed one to four times a day. Checkpoint each node’s frontier and URL-seen files to blob storage every few minutes; a dead node’s successor loads the checkpoint and replays the log, losing at most minutes.
• Per-IP limits across nodes. Two hosts on one shared IP may land on different nodes; a small shared table of the busiest IPs, each with a token budget, closes the gap.

The URL database, content store and simhash tables stay shared services, the first sharded by the same host hash. This is sharding with the shard key chosen for the one invariant you cannot break: politeness.

How do we keep pages fresh without recrawling everything?

The budget is fixed: 1 billion fetches a day. Suppose the index holds 10 billion pages. Which ones deserve a fetch today?

Bad: a uniform cycle. Visit every page every N days. With 10 billion pages and 1 billion fetches a day, N = 10. A live-score page that changes every few minutes is almost always stale, while a page untouched since 2019 is fetched 36 times a year for nothing. Uniform is fair to pages, not to users.

Good: revisit by change rate. Model each page’s changes as random events at an average rate λ — a Poisson process. The chance a page has changed t after your last visit is 1 − exp(−λt). Estimate λ per page from its history: how many visits found a change. If you revisit every I, the share of time your copy matches the live page — its freshness — is (1 − exp(−λI)) ÷ (λI). Mercator already did a simple version: pages that did not change between downloads drifted to lower priority.

Two lessons fall out. A single interval suits nobody: daily visits keep a weekly-changing page 93% fresh but an hourly one only 4%. And returns diminish fast: for a daily-changing page, four fetches a day instead of one buy 63% → 88% for 4× the cost. A page that changes every few minutes cannot be kept fresh by polling at all; its fetches would keep a dozen slower pages fresh instead.

Great: spend the budget where it buys the most freshness that users will notice.

• Score revisits by value. Revisit priority ≈ importance × probability the page has changed, where importance comes from search clicks and links pointing at the page, and the probability is 1 − exp(−λt) since the last visit. Revisits and new URLs then compete in the same front queues on one scale.
• Use hints cheaper than a fetch. Sitemaps give each URL a lastmod date — the page’s modification date, not the sitemap’s. changefreq and priority are hints only; the protocol itself says crawlers may visit “hourly” pages less often. A sitemap holds at most 50,000 URLs or 50 MB, so large sites publish sitemap index files, and news sites also publish feeds.
• Make unchanged revisits nearly free. Conditional GETs turn “no change” into a 304 with no body. RFC 9110 says a server must ignore If-Modified-Since when If-None-Match is present, since the ETag comparison is the more accurate one. Suppose 70% of the daily billion fetches are revisits and half of those find an unchanged page on a server that supports validators: 350 million × 100 KB ≈ 35 TB a day not downloaded — a third of the uncompressed traffic.
• Respect each host’s crawl budget. Google frames it as what it can crawl (the capacity limit, or hostload) and what it wants to crawl (demand from popularity and staleness). A slow host gets fewer fetches however high the demand; the frontier’s delays already enforce that.
• Let status codes steer the schedule. A 404 or 410 means visit less and eventually drop — Google lowers the crawl frequency of such URLs gradually. A 301 moves the record; a 5xx or 429 keeps the old copy and slows the host. Google follows up to 10 redirect hops; cap yours similarly.

How do we survive spider traps, slow hosts, DNS and robots.txt?

The web is not a tidy graph. Some of it is accidentally infinite, some is slow, and a little is hostile. Each needs its own defence.

Spider traps are URL spaces with no end: a calendar with a “next month” link, faceted search where ten on/off filters make 1,024 URLs per category, session IDs minted on every visit, a server that returns the same page for any path so relative links grow /a/b/a/b/a/b/…, and wildcard DNS that answers for any subdomain. IRLbot’s authors found some built for spam, some innocent, and some made to trap robots.

• Bad: a global depth limit. It stops some traps but also cuts off legitimate deep sites, and many traps are shallow and wide (faceted search) rather than deep.
• Good: URL-shape rules and per-host caps. Reject URLs over 2,048 characters (the sitemaps protocol’s own URL limit), deeper than about 16 levels or with a repeating segment; collapse a path whose only change is a counter; cap new pages per host per day. Cheap — but a hard cap cannot tell a trap from a genuinely huge site, and IRLbot notes legitimate sites with hundreds of millions of pages.
• Great: budgets tied to reputation, and deferral instead of deletion. IRLbot’s STAR gives each pay-level domain a page budget in proportion to the number of other domains linking to it — an asset spammers must pay for, unlike pages and subdomains. Most domains got 10 pages; the top 10,000 got between 10 and 10,000. Its BEAST scheme never deletes an over-budget URL: it spreads them over disk queues re-read at exponentially longer intervals, so spam cannot snowball, yet a domain that gains reputation gets its URLs back. In its run, 90% of domains contributed fewer than 586 pages each, while the largest, a blog host, contributed 347,613. The near-duplicate check helps too: a trap usually serves one template over and over.

Slow and hostile hosts. Give every fetch hard limits: a few seconds to connect, about 30 seconds in total, a minimum transfer rate and a size cap — Googlebot reads only the first 2 MB of an HTML file for search. IRLbot’s authors also describe a delay attack: sites that deliberately slow every HTTP and DNS reply to a crawler’s address. With async I/O and one request in flight per host, a slow host ties up one connection and slows only itself. Retry failures later through the frontier, never in a tight loop (see timeouts, retries and backoff).

DNS. Mercator found that DNS lookups took 70% of each worker’s time, because the standard resolver allowed one outstanding lookup per process; a multi-threaded resolver talking to a local name server cut that to 14%. Today: an async resolver, a caching resolver on every node, lookups started when a host enters the back queues, caching within each record’s TTL, negative caching, and a per-domain cap on new host names against wildcard DNS. More in the Domain Name System.

robots.txt, precisely. RFC 9309 (2022) standardised Martijn Koster’s 1994 convention, and says plainly that its rules are not access control. Pick the group whose user-agent matches your product token, case-insensitively, else *; apply the most specific (longest) matching allow or disallow rule, with allow winning ties; /robots.txt itself is always allowed. Then the fetch result:

Note the asymmetry: a missing robots.txt means “crawl freely”, a broken one means “crawl nothing” — getting it backwards is a classic bug. Google’s implementation shows the details: on a 5xx it stops crawling the site for 12 hours while retrying, then uses the last good copy for up to 30 days.

What about pages that need JavaScript to show their content?

Some pages are app shells: an empty container plus a script bundle, with content only after the JavaScript runs. Google describes three stages — crawl, render, index — with a separate render queue where pages wait seconds or longer for a headless Chromium, and the rendered HTML is parsed for links again.

Copy that shape: a render queue and a pool of headless browsers, separate from the fetchers. Render selectively — pages with little visible text and a large script bundle, where the page matters. Cache shared scripts per site, and send every sub-resource fetch through the same robots.txt and politeness checks. Budget it honestly: a render costs CPU-seconds and hundreds of megabytes of memory, where parsing HTML costs milliseconds. If 5% of fetches need rendering, that is 50 million renders a day, about 580 a second; at an assumed 2 CPU-seconds each, around 1,200 cores busy — and twenty times that if you rendered everything.

The final design

Put the pieces together. Each of about 100 nodes runs the same stack for its slice of hosts; the URL database, content store, simhash tables and event log are shared services.

1. Schedule. Seeds, sitemaps and feeds, plus the scheduler scanning the URL database for due rows, push URLs into the owning node’s frontier with a priority from importance, domain budget and staleness.
2. Pick. The biased picker fills per-host back queues from the priority queues; a worker pops the host whose wait ends first from the ready-time heap.
3. Fetch. The fetcher checks the cached robots.txt, takes the IP from the DNS cache and sends a conditional GET with tight timeouts and a size cap; response time and any 429 or 5xx set the host’s next allowed fetch.
4. Process. The processor computes a checksum and a 64-bit simhash. Copies are recorded and go no further; new content is appended, compressed, to the content store.
5. Follow links. Links are normalised and filtered. Those for other nodes’ hosts leave through the URL exchange; local ones, and those arriving from peers, pass the batched URL-seen check, and never-seen URLs join the frontier.
6. Publish. Each stored page produces a pages.fetched event with its blob offset and hashes for the indexer.
7. Record. Status, validators and whether the page changed update the URL database’s change-rate estimate and next due time, closing the recrawl loop.
8. Render, when needed. Important JavaScript app shells go to the render queue; the rendered HTML returns to step 4.

Check it against the requirements. Throughput: about 100 nodes of 250 pages/s give 11,600 pages/s with 2× headroom, on about 9.3 Gbps. Politeness: every host has one owner and one clock, robots.txt is never older than 24 hours, and distress slows a host on its next fetch. Scale: the URL-seen set grows on disk, not in RAM, and domain budgets stop traps snowballing. Freshness: revisits compete with new URLs on one value scale, and conditional GETs make unchanged revisits cheap. Durability: content sits in replicated blob storage, frontiers are checkpointed and the exchange is a replayable log, so a dead node costs minutes on 1% of hosts.

What interviewers look for by level

Follow-up questions

What if a site owner says we are overloading their server?

Act first, then fix the cause. Set a host policy through the steering API — a longer min_delay_ms, or a pause — which the owning node applies on its next heap pop. Then find out why the adaptive delay missed it: perhaps the host shares an IP with hundreds of others and needs a per-IP cap, or it fails slowly instead of returning a 503. Tell the owner they can also answer 429 or 503 with Retry-After, or narrow access in robots.txt.

What if breaking news must be in the index within minutes?

Do not try to poll the whole web faster. Keep a small, high-priority set of news hosts whose sitemaps and feeds are polled every minute or two; new URLs from them enter the top front queue after a small in-memory URL-seen check instead of waiting for the next batch. Politeness still applies, so large publishers effectively get many short visits rather than one long one. Smaller-scale feed polling is a design of its own — see the news aggregator.

What if we only want one topic, language or country?

That is a focused crawler. Add a classifier after parsing that scores each page’s relevance, and let a link inherit part of its parent page’s score as a priority hint. Domain filters (a country’s top-level domain, a language detector) prune early. The risk is tunnel vision — relevant pages reachable only through irrelevant ones — so keep a small budget for exploring low-scoring links.

What if a node dies in the middle of the crawl?

Its lease expires and its range of the ring passes to its neighbours, which load its latest checkpoint from blob storage and replay the URL-exchange log from that offset. In-flight fetches are simply repeated; the content check makes the few double downloads harmless. Hosts in that range pause for a minute or two — 1% of the crawl, not all of it.

What if the same crawler must also collect data for training models?

Give it its own product token, such as HandbookBot-Data, so site owners can allow search crawling and refuse data collection in robots.txt — RFC 9309 matches groups per token, so no new protocol is needed. Priorities shift from freshness to breadth and quality filtering. Keep both behind one politeness layer, because the site sees one load either way.

How to say it in the interview

“I’m designing a crawler that feeds a search index: a billion fetches a day, about 11,600 pages a second and 9 gigabits of page traffic, with zero robots.txt violations and important pages refreshed daily.

The core is a loop: a frontier picks the next URL, async fetchers download it, a processor stores it and extracts links, and URL-seen and content-seen checks stop repeated work. The frontier has priority queues in front and one queue per host behind, with a min-heap of each host’s next allowed fetch time. A host waits about ten times its last response time, longer after a 429 or 503. robots.txt is cached for at most 24 hours, and a 5xx on it means ‘crawl nothing’.

The URL-seen check is the hot loop: 580 thousand checks a second against 100 billion URLs. Strings would need 10 terabytes of RAM and a Bloom filter silently skips about 1% of new URLs, so I fingerprint normalised URLs and check them in batches with a sorted merge on local disk. For content, a checksum catches exact copies and a 64-bit simhash catches near-copies within 3 bits.

I hash hosts onto about 100 nodes on a consistent-hash ring, so each host’s politeness state lives on one machine; only about 20% of links cross nodes, in batches over a durable log.

Finally, a scheduler estimates each page’s change rate and spends the fetch budget where it buys the most freshness, with sitemaps and conditional GETs keeping it cheap, while per-domain budgets, URL rules and timeouts keep spider traps and slow hosts from eating the crawl.”

Concepts used in this design

• How search engines work — where the crawler sits in the larger system.
• Search end to end — the crawl-to-results pipeline this design feeds.
• Bloom filters — a compact URL-seen pre-filter, and its one-sided error.
• Checksums — fingerprints for URLs and exact-duplicate pages.
• Domain Name System — why lookups bottleneck crawlers, and caching them.
• HTTP and HTTPS — conditional GETs, status codes and redirects.
• Rate limiting — politeness is rate limiting from the client’s side.
• Consistent hashing — assigning hosts to nodes so few move when the fleet changes.
• Database sharding — the host as the shard key.
• Message queues and Kafka — the URL exchange and the pages.fetched log.
• Backpressure and load shedding — keeping the exchange from drowning in links.
• Blob storage — append-only files for 20 TB of pages a day.
• Timeouts, retries and backoff with jitter — surviving slow and failing hosts.
• Back-of-envelope estimation — the arithmetic showing URL checks, not fetches, are the hot loop.

References and further reading

1. High-Performance Web Crawling (Najork and Heydon, Compaq SRC Research Report 173, 2001)cs.cornell.eduThe Mercator crawler: front and back queues with a heap for politeness and priority, fingerprint-based URL-seen tests, host-partitioned distribution, and the DNS bottleneck.
2. IRLbot: Scaling to 6 Billion Pages and Beyond (Lee, Leonard, Wang and Loguinov, ACM TWEB 2009)irl.cs.tamu.edu6.38 billion pages in 41 days on one server; DRUM batched disk checks, per-host and per-IP delays, and STAR/BEAST domain budgets against spam and traps.
3. RFC 9309: Robots Exclusion Protocolrfc-editor.orgrobots.txt groups and longest-match rules, 4xx versus 5xx semantics, the 24-hour cache limit and the 500 KiB parsing minimum.
4. Detecting Near-Duplicates for Web Crawling (Manku, Jain and Das Sarma, WWW 2007)static.googleusercontent.com64-bit simhash fingerprints with a Hamming distance of 3 on 8 billion pages, and permuted sorted tables for fast lookups.
5. Optimize your crawl budget (Google Search Central)developers.google.comCrawl capacity limit (hostload) plus crawl demand; slowing down on 5xx and 429; 304 responses and sitemaps with lastmod.
6. How HTTP status codes affect Google’s crawlersdevelopers.google.comRedirect hop limits, 304 handling, 4xx as “gone”, and 429 and 5xx as signals to slow down.
7. Google’s robots.txt specificationdevelopers.google.comHow one large crawler caches robots.txt, handles 5xx for 12 hours and 30 days, and ignores crawl-delay.
8. Sitemaps XML format (sitemaps.org)sitemaps.orglastmod, changefreq and priority as hints; 50,000 URLs and 50 MB per sitemap; sitemap index files.
9. Common Crawl: October 2026 crawl archive now availablecommoncrawl.orgA real monthly crawl: 2.16 billion pages and 371 TiB of uncompressed content from 41.4 million hosts in two weeks.
10. Understand the JavaScript SEO basics (Google Search Central)developers.google.comCrawl, render and index as separate stages, with a render queue and headless Chromium for JavaScript pages.