Layer 3 · Go deeper · 9 min
Go deeper
Size it, build it, then make the calls an interviewer will push you on.
Back of the envelope
Size it up
Tap through each step of the arithmetic.
How long does Blackbird take to index all of GitHub?
How many tweets and searches a second did Earlybird face in 2011?
Build-up · 7 steps
Bend the recipe, one constraint at a time
Columns are the series’ five stages; rows are six published systems. Dashed cells kept the default. Accent cells are where one dominant constraint forced something different.
Step 1 of 7
The series’ default recipe
Parts 1–7 built this for Bazaar: an inverted index of immutable segments refreshed every second, document-partitioned shards a coordinator fans out to, cheap retrieval, then a learned reranker.
4 interview moments
You’re the candidate
The interviewer pushes on the hard parts. Pick an approach and see how it holds up.
Deep dive 1 of 4
Interviewer
“GitHub, 2023. Developers search 115 TB of code for punctuation, substrings and regexes. How do you serve that?”
Your move. Pick an approach:
Deep dive 2 of 4
Interviewer
“Boosted trees brought Airbnb big wins, then long spells of neutral experiments. How do you start moving ranking to neural networks?”
Each listing is unique, and even the most popular can be booked at most 365 times a year.
Your move. Pick an approach:
Deep dive 3 of 4
Interviewer
“Your two-tower retriever learns from Amazon’s purchase logs. How do you treat products that were shown but not bought?”
Your move. Pick an approach:
Deep dive 4 of 4
Interviewer
“Uber moves its logs from Elasticsearch to ClickHouse. Logs have no fixed schema, and a field’s type can change. How do you lay them out?”
Over 80% of queries aggregate, and only 5% of indexed fields were ever queried.
Your move. Pick an approach:
Build-up · 5 steps
Make a tweet searchable without locks
Earlybird, from its 2012 paper: one writer thread and thousands of query threads share the newest segment. Visibility is a single number, maxDoc.
Step 1 of 5
Segments fill in time order
A server held 12 segments then, each capped at 2²⁴ doc IDs, about 8.4 million tweets in practice. Only the newest takes writes; when full, a compressed read-only copy is swapped in.