42 TB
the whole index, compressed from 144 TB
20 billion pages, 8 trillion postings. Store gaps between docIDs in as few bytes as each needs; it fits in 1,000 servers’ RAM.
Split the index how?
Bad: By term
Intersecting “ipl score” ships a 15 MB list. The “ipl” shard runs hot.
Good: By document
Each shard ranks its own 20M pages and returns a 1 KB top ten.
A leaf slow once in 1,000 calls makes 63% of queries slow
Every query waits for the last of 1,000 answers. Google measured p99 at 10 ms for one leaf, 140 ms for all.
Slowness comes from a garbage-collection pause or a noisy neighbour, not from the query.
The trick
No reply by 15 ms? Ask another replica of that shard. Slow queries fall from 63% to ≈ 0.1%.
20 billion pages in, ten links out
Cheap BM25F and PageRank scores on millions, pruning the rest. The learned model sees only 200 per shard, which keeps hundreds of signals affordable.
40%
of queries end at the result cache
“ipl score” repeats millions of times a day. Caching the head turns a 100,000/s peak into 60,000/s at the index: 12,000 leaves, not 21,000.
“ipl score” can’t wait for next week’s rebuild. Index incrementally:
- Route each page update to its shard
- Leaves add it to an in-memory segment in seconds
- A bitmap hides the old copy
Remember
A compressed inverted index, split by document into 1,000 shards in RAM. Fan out through a tree, rank in two phases, hedge slow leaves, cache the head, index fresh pages incrementally.