96 s
to grep GitHub’s 115 TB, per query
Code needs substrings, punctuation and regexes, which word analysers throw away. So GitHub built Blackbird, an n-gram index, and checks every hit against the text.
Shard by the file’s hash: a thousand forks, one blob.
Dedupe, even load and cheap fork indexing fall out of one key. The price is that every query visits every shard.
Twitter, 2012: thousands of queries read the segment one thread is writing tweets into.
No locks
After each tweet, the writer bumps a volatile maxDoc. Readers skip postings above it: visibility is one number.
“burgundy dress” should find a red one. Words alone can’t.
Amazon kept lexical retrieval for exact tokens and added a two-tower retriever, product vectors precomputed. Both candidate sets go to the existing rankers.
5%
of Uber’s indexed log fields were ever queried
And over 80% of queries aggregated. Uber moved logs to ClickHouse columns, materialised only hot fields and cut hardware cost by more than half.
GitHub left Elasticsearch for its own engine. Yelp went the other way. Who was wrong?
Neither
Build only the stage your constraint bends. Word analysers broke GitHub’s substrings; Yelp’s own engine slowed every ranking push.
Copy Blackbird for Bazaar’s 50 million products?
Bad: Copy the architecture
A custom Rust engine, for data a handful of Elasticsearch nodes hold.
Good: Copy the reasoning
Name the query shape first. Shoppers type words, not code.
Remember
Real search is the series’ recipe bent by one constraint. Name the constraint first, say the year with the number, and build only the stage it bends; buy the rest.