580k/s
“seen this URL?” checks, 50× the fetches
A billion pages a day is 11,600 fetches a second. Each brings about 50 links, and every link needs a check.
Which URL goes next?
Bad: One big FIFO queue
Links bunch by host, so 500 workers swamp one small site.
Good: Priority front, per-host back
One request in flight per host; a heap picks whose wait ends first.
100 billion URLs. As strings, 10 TB of RAM; a Bloom filter silently skips new ones.
The trick
Batch, sort, merge. A node’s 10-minute batch sweeps one sorted file in 16 s; seeks take ≈ 4 hours.
One timestamp changed. Is it the same page?
A checksum says new. Simhash gives similar pages similar 64-bit fingerprints; within 3 bits is a near-copy, so skip it and its links.
Split the crawl by host, not by URL
Hash URLs and every node visits one small site, politely, at once. Hash hosts and one node owns its queue, robots.txt and DNS.
4%
fresh: an hourly page, crawled daily
The same daily visit keeps a weekly page 93% fresh. So estimate each page’s change rate and spend fetches where users will notice.
A calendar offers “next month” until 9999. Cut it off:
- URL rules: ≤ 2,048 characters, ≤ 16 levels, no repeats
- Budget domains by how many other domains link in
- Defer over-budget URLs instead of deleting them
Remember
A per-host, prioritised frontier keeps you polite. Batched URL checks and simhash stop repeat work. Partition by host; revisit by change rate.