Search end to end: from crawl to ranked results

Search end to end is one system with two paths and a loop. The indexing path turns every committed change, or every crawled page, into an enriched, searchable document within seconds. The query path understands the query, retrieves about a thousand candidates, narrows them through ever smarter rankers to one page, and serves it in under 200 ms. The learning loop logs what users saw and did and retrains the rankers.

This is part 7 of the search engines series. Part 1 mapped the kinds of search, part 2 built the inverted index, part 3 put it on disk, part 4 chose the engine, part 5 ran the cluster and part 6 scaled to the web. Here we assemble one system for Bazaar’s 50 million products and keep the earlier promises: the freshness budget, the rebuild procedure, intent classification and NDCG.

Picture it first: a sari shop in Chandni Chowk

New stock arrives all day. A boy writes each bolt into the register and pins a tag on its shelf; the tags lag the register by minutes, so before quoting a price the salesman checks the register.

A customer says, “laal kurti, 500 tak, ladies.” The counter boy translates: red, kurta, under ₹500, women. One helper pulls everything tagged “red kurta”; another pulls pieces that look like what she described. Their piles are merged. A junior keeps 30 of 200 in her size and budget; the senior salesman lays the best ten out in order. The owner slips in one piece a brand pays to promote and makes sure the ten don’t all come from one weaver. At closing he notes what was unfolded and what was bought, and on Sunday he coaches the staff.

Map it back: the register is the database, the tags are the index and their lag is the freshness budget. The counter boy is query understanding, the helpers are lexical and vector retrieval, junior and senior are the L1 and L2 rankers, and the owner is the business layer. The notes are click logs, coaching is retraining, and re-tagging the shop overnight is a reindex.

The whole system on one page

Fig. 1 answers “walk me through search”: three loops at three speeds.

Indexing path, seconds. Sellers write only to the database. A change data capture connector streams committed changes into Kafka; an indexer enriches each product and bulk-writes it behind the alias products. A daily job publishes sales and ratings into the same topic. All of it is asynchronous and rebuildable: part 4’s rule that the index is a copy.

Query path, milliseconds. The search API checks its cache, rewrites the query, retrieves, fuses, ranks in stages, applies business rules and hydrates the winners.

Learning loop, days. Logged pages and clicks become training data for the next models.

Web search keeps the skeleton and swaps the front end:

Ingestion and the freshness budget

Part 3 said most staleness comes before the index. Follow one price cut, ₹649 to ₹449, through a healthy pipeline (Fig. 2):

1. Commit. The seller’s transaction commits; the clock starts.
2. CDC. Debezium reads the WAL through a replication slot, polling every 500 ms by default: up to 0.5 s. It stamps each event with source.ts_ms, the commit time.
3. Kafka. Up to 0.1 s. Events are keyed by product ID, so one product’s changes stay in order.
4. Enrich. Cached joins take about 0.1 s. A price change skips the embedding model; a title change waits for it, up to 2 s.
5. Batch. The indexer flushes a _bulk every second or 5 MB: up to 1 s. It keeps only each product’s newest version per batch, so 40 stock changes to one kurta during a sale become one write.
6. Bulk write. Primary and replica apply it, versioned externally: about 0.2 s.
7. Refresh. Searchable at the next refresh: up to 1 s.

Worst case: 0.5 + 0.1 + 0.1 + 1 + 0.2 + 1 = 2.9 s for a price, 4.9 s for a title; typical is about half. Target p99 within 5 s normally, 60 s during a sale.

Hydration hides most of the gap: the API reads live price and stock from the database, so the shopper sees ₹449 at once; only filters and sorting wait. This is eventual consistency by design; the Elasticsearch page covers the sync pattern itself.

Measure lag in seconds, not messages

Bazaar averages 30 million changes ÷ 86,400 s ≈ 347 a second and peaks at 5,000, so 50,000 messages behind is 10 s at the peak but 144 s at night. Measure event-time lag: after each successful bulk, record now − source.ts_ms per document; page when p99 exceeds 10 s for 5 minutes. A canary product whose price changes every 30 s, plus a probe that searches for it, catches a stuck refresh or a wrong alias.

The web variant

For web search the ingest is part 6’s crawl loop: frontier, robots.txt, fetch, render, SimHash dedupe, index. Staleness is set by the recrawl interval: 1 billion fetches a day over 50 billion URLs revisits each page every 50 days. Web freshness is a scheduling problem, not a pipeline one.

Enriching the document

The indexer denormalises each product, joining what retrieval and ranking need so no query joins at read time:

• Category IDs and names along the path (Women › Ethnic wear › Kurtas), for filters, facets and boosts.
• Seller rating, so a 2-star seller can be demoted.
• Popularity: 30-day sales in log₂ buckets, so a product is rewritten only when it crosses a bucket.
• Stock as an in_stock flag that flips only at zero; exact counts would rewrite the document on every order (part 3’s delete-plus-add).
• Language: English, Hindi and transliterated title sub-fields so “kurta”, “कुर्ता” and “kurtaa” meet (part 2).
• Embedding: a 768-dimension vector of title and attributes, recomputed only when they change.

The rule: index what retrieval, filtering or L1 needs. Anything only L2 reads lives in the feature store, which updates without rewriting Lucene documents.

A new embedding model is the expensive change. At an illustrative 2,000 products a second per GPU, 50,000,000 ÷ 2,000 = 25,000 s, about 7 hours on one GPU, and old and new vectors can’t be mixed. That means a rebuild.

Rebuilding the index without downtime

You rebuild to change an analyser, a field type, the shard count or the embedding model, or after corruption. Never in place: build products_v8 beside the live products_v7, then switch (Fig. 3).

1. T0, mark and load. Record the Kafka offsets first, then stream a database snapshot into products_v8, created with the new mapping, no replicas and refresh_interval: -1, as Elastic’s indexing speed guide advises. At an illustrative 20,000 documents a second, 50 million take about 42 minutes.
2. T1, replay. A second consumer group for v8 starts at the recorded offsets and replays the changes made during the load, 2,520 s × 347 ≈ 875,000 events, in about 3 minutes at 5,000 a second. Overlap is harmless: external versions reject anything older than v8 holds, an idempotent consumer. Kafka’s retention must outlast the rebuild.
3. T2, validate. Both groups now keep their index current. Restore refresh and the replica, then check per-category document counts against the database, NDCG@10 on a golden set of 2,000 rated queries, the zero-result rate on yesterday’s top 10,000 queries, and p99 under shadow traffic.
4. T3, swap. One _aliases call removes v7 and adds v8, atomically: the alias never points at both, or neither.
5. T4, rollback window. Keep v7 fed for 48 hours; rolling back is the reverse call.

Backfills. Adding a field, such as fabric extracted by a new model, needs no rebuild. Send backfill events through a separate, throttled topic so 50 million updates don’t queue ahead of live price changes.

Reconciliation. Pipelines drift silently: a dropped event, a delete that never arrived. A nightly job compares per-category counts and a 1% sample of (id, version) pairs between database and index and republishes mismatches; Debezium can re-emit a key range with an incremental snapshot.

Query understanding

Part 1 promised intent classification in the query path. It lives in a query-understanding stage that turns text into a structured request in about 20 ms. Take “Red Kurtha under 500 for women”:

1. Normalise: lowercase, Unicode NFKC, and map “₹500”, “rs 500” and “500/-” to one form.
2. Spell-correct: “kurtha” → “kurta”, from a dictionary mined from query logs, keeping the original as a fallback.
3. Transliterate and expand: “laal” → red, “tunic” → kurti. Keep synonyms directional where meanings differ: “kurti” may show kurtas, but “kurta” shouldn’t drag in every kurti.
4. Extract entities with a tagger and dictionaries: colour red, gender women, price at most 500; brands, sizes and fabrics likewise.
5. Classify the category (Women › Ethnic wear › Kurtas, probability 0.92) and the intent. Navigational queries (“fabindia”, an order ID) jump to a store or page; informational ones (“how to wash silk”) add help articles; transactional ones get product search.
6. Embed the cleaned text for kNN, in parallel with steps 4 and 5.

{
  "text": "red kurta",
  "filters": { "gender": "women", "price_max": 500, "in_stock": true },
  "boosts": { "category": "women/ethnic/kurtas", "colour": "red" },
  "intent": "transactional",
  "rewrites": ["kurtha → kurta"]
}

Colour is a boost, not a filter: harden an entity into a filter only when the model is confident and the shopper explicit, since a wrong filter empties a good query.

Relax when results run out. Under about 10 results, relax one step at a time and say so: weakest filter to a boost, most words instead of all, fuzzy matching, then vector results. Zero-result logs are your best source of new synonyms and spellings.

Retrieval: lexical, vector and fusion

L0 retrieval turns 50 million products into about 800 candidates with two retrievers running in parallel:

• Lexical: BM25 over title, brand and attributes (part 2) with the filters, top 500. It nails exact tokens: brands, model numbers, “XL”.
• Vector: kNN over the embeddings with the same filters, top 500. It finds meaning without shared words, such as a burgundy kurta for “red”. Amazon’s semantic product search (KDD 2019) added such a matcher beside its lexical one, answered within 20 ms at thousands of queries a second, and raised conversion in every category it tested online.

BM25 scores are unbounded and cosine similarities aren’t, so adding them needs normalisation. Reciprocal rank fusion ignores scores and uses ranks. Cormack, Clarke and Büttcher’s RRF paper (SIGIR 2009) scores each document as:

RRF(d) = Σ over result lists   1 / (k + rank of d in that list),   k = 60

They fixed k = 60 in a pilot and never changed it; Elasticsearch’s rank_constant and Vespa’s reciprocal_rank both default to 60. Five candidates, worked by hand:

Being found by both retrievers roughly doubles a score, and BM25’s favourite, P1, falls to third because the vector side ranked it fourth. The constant flattens rank gaps: with k = 60, rank 1 outscores rank 10 by 70 ÷ 61 ≈ 1.15×; with k = 1 it would be 11 ÷ 2 = 5.5×. You gain fusion without tuning; you pay by discarding score magnitudes, so a perfect title match counts no more than a lucky first place.

The ranking funnel

Heavy models can’t score 50 million products, so ranking is a funnel: each stage costs more per item and sees fewer items (Fig. 4).

L1 uses features already in the hit (BM25, vector similarity, sales bucket, rating, stock) to drop obvious misfits cheaply. L2 pays for rich features and a trained model. The business layer caps any seller at three of the top 20, puts sponsored products in fixed, labelled slots such as 3 and 8, and applies merchandising pins and light personal boosts.

Learning to rank

There are three ways to frame training:

• Pointwise: predict each product’s grade or purchase probability alone. Simple, but it ignores that only order matters.
• Pairwise: learn that the purchased product should outrank a skipped one; RankNet minimises a pairwise cross-entropy.
• Listwise: optimise a list metric such as NDCG directly. But NDCG is flat or jumps as scores change, so it has no useful gradient.

Burges’s report From RankNet to LambdaRank to LambdaMART builds the bridge. LambdaRank scales each pairwise gradient by the size of ΔNDCG, the change in NDCG if the two documents swapped, so an error near the top counts more than one at rank 10. LambdaMART fits those gradients with gradient-boosted trees; an ensemble of them won Track 1 of the 2010 Yahoo! Learning to Rank Challenge. It remains the default first model: fast on CPUs and happy with unscaled features.

Airbnb’s Applying Deep Learning to Airbnb Search (KDD 2019) shows the usual arc: replacing a hand-made scoring function with a GBDT gave one of the largest step improvements in bookings in its history; neural nets came later, trained on booked-versus-not-booked pairs from the same search, weighted by ΔNDCG.

Where it runs. Elasticsearch runs LTR inference since 8.12, on some paid subscription levels, for GBDT models trained elsewhere, such as XGBoost’s LambdaMART: as a rescorer, or a retriever since 9.1, with a mandatory window_size of at least from + size. Vespa runs a cheap first phase on content nodes, a second phase on each node’s best 100 by default, and a global phase, often a cross-encoder, on the merged best 100. Bazaar runs its rankers in a separate service so models ship without touching the cluster.

Features. L2’s 150 features mix query features (length, category, intent), product features (price, rating, return rate), query-product features (BM25 per field, vector similarity, past click-through) and user features (price band, brands). An online feature store, often Redis, serves them in one batched read; training reads the same definitions from the offline store as of each impression’s time, or you get training–serving skew.

Serving the page

The search API also owns five jobs:

• Result cache. Key on normalised query, filters, sort, page, region and ranker version, with a 60-second TTL; keep personalisation out of the key by caching the L1 top 200 and re-ranking per user. If the 10,000 commonest queries carry 40% of 5,000 searches a second (measure yours), the cache answers 2,000 a second while recomputing only 10,000 ÷ 60 ≈ 167 a second.
• Hydration. One batched read for 20 IDs from a product cache fed by the same CDC stream.
• Pagination. Keep the rerank window fixed, as Elastic’s LTR docs warn, or results reshuffle between pages.
• Autocomplete: popular-query prefixes in a trie, 20–50 ms per keystroke, never through the funnel (design typeahead).
• Deadlines and degraded modes. L2 late: serve L1 order. kNN late: BM25 alone. Query understanding down: the raw query. Cluster overloaded: serve stale cache, then shed load. Hydration slow: a cached snapshot; checkout re-checks.

Measuring relevance offline

Build a judgement list: queries, products and grades from 0 to 3 (3 exactly what was asked, 2 a good substitute, 1 related, 0 wrong), from human raters (consistent, expensive) or clicks (cheap, biased; see below). 2,000 queries × 20 results = 40,000 judgements, at 30 seconds each about 333 rater-hours. Elasticsearch’s _rank_eval API computes these metrics over such a list:

• Precision@k: the share of the top k that is relevant (grade 1 or more).
• Recall@k: the share of all relevant products that made the top k.
• MRR: the mean over queries of 1 ÷ the rank of the first relevant result; suits navigational queries.
• NDCG@k: graded relevance discounted by position and normalised by the best possible order, from Järvelin and Kekäläinen’s 2002 paper:

DCG@k  = Σ for i = 1..k   (2^grade_i − 1) / log2(i + 1)
NDCG@k = DCG@k / IDCG@k        IDCG = DCG of the ideal order

The 2002 paper used raw grades as gains; the exponential gain in Burges’s report is the common form today.

For “red cotton kurta”, judges graded six products: red cotton kurta 3, maroon kurta 2, red rayon kurta 2, red printed kurta 2, red kurta set 1, red saree 0. Ranker A (BM25 with boosts) and ranker B (with LambdaMART) both miss the printed kurta and return the other five in different orders (Fig. 5).

Precision@5 is 4 ÷ 5 = 0.8 for both, and so is recall@5 (four of five relevant products), so neither can tell them apart. MRR, counting grade 2 or more as relevant, gives A 1 ÷ 2 = 0.5 and B 1.0. NDCG@5 gives A 0.63 and B 0.88: B put the grade-3 kurta first, worth all 7 points, where A put it third, worth 3.5. Neither reaches 1.0 because both missed the printed kurta. With raw grades as gains, A scores 0.70 and B 0.85: the same order, but a smaller gap.

Online tests and learning from clicks

Offline metrics predict; online tests decide. Track conversion per search and click-through, and the pain signals: zero-result rate, reformulation rate (another search within a minute) and abandonment (no click). Guard p99 latency and ad revenue.

A/B tests split users, and small gains need big samples: detecting a 1% relative lift on 4% conversion (to 4.04%) needs about 16 × p(1 − p) ÷ δ² = 16 × 0.04 × 0.96 ÷ 0.0004² ≈ 3.84 million searches per arm at 80% power and 5% significance.

Interleaving is the faster screen. Each user sees one merged list; in team-draft interleaving the two rankers take turns picking their best remaining result, like captains choosing players, and the ranker whose picks earn more clicks wins. Chapelle, Joachims, Radlinski and Yue’s large-scale study (TOIS 2012) found it needs 10 to 100 times less data than absolute click metrics. Screen by interleaving, then confirm business metrics by A/B test.

Why you can’t train naively on clicks

Joachims and colleagues tracked searchers’ eyes on Google in a 2005 study. Users looked at results 1 and 2 almost equally but clicked result 1 far more. With the top two secretly swapped, they still favoured whatever came first: trust bias. With the list reversed, they clicked less relevant results: quality bias. A click means “better than what I skipped”, not “relevant”. Train on raw clicks and the model promotes whatever sat first, and new products never get seen. The fixes:

• Relative labels: “Click > Skip Above”, a clicked result beats unclicked ones ranked above it.
• Propensity weighting: estimate how often each position is examined, for example by swapping adjacent results on 1% of traffic, and weight clicks by the inverse.
• Exploration: reserve a slot on some pages for new or rarely shown products.

The training pipeline. Log impressions with request ID, ranker version, positions and features; join clicks and purchases by request ID; label them (purchase 3, add-to-cart 2, click 1, skipped 0); attach features as of impression time; train; check NDCG on held-out days; then shadow, interleave, A/B test and ramp (canary and feature flags). Refresh features daily, the L2 model weekly, the embedding model a few times a year, since each one means a rebuild.

Going deeper: failure modes

• Stale index after a CDC outage. Debezium stops for 2 hours. The replication slot keeps every WAL file the connector still needs, so the primary’s disk grows; on restart, 2 × 3,600 × 347 ≈ 2.5 million events drain in about 500 s at 5,000 a second. Alert on event-time lag and slot size; if PostgreSQL invalidates the slot (max_slot_wal_keep_size), events are lost, so rebuild from a fresh snapshot.
• A bad model shipped. Offline NDCG looked fine, but a feature arrives null in production and conversion falls 3%. Defend with shadow scoring, a 1% canary that rolls back on guardrails, and the ranker version in every log line and cache key.
• Popularity feedback loops. Bestsellers stay on top because they are on top. Debias clicks, explore, and give new products a decaying boost.
• Zero-result spike after an analyser change. A new Hindi normaliser splits a common token and zero results jump from 3% to 9%. Query-log replays catch it before the swap; after it, the fix is one alias call.
• Cache stampede on a trending query. After a cricket final, “jersey” gets 3,000 searches a second; when its entry expires, every request during the 150 ms recompute misses: 3,000 × 0.15 = 450 identical searches. Coalesce requests, serve stale while revalidating and jitter TTLs (hot keys and stampedes).
• Hot shard during a festive sale. Sale traffic overloads one shard’s copies, and every fanned-out query waits for the slowest. Add replicas before the sale (part 5), let adaptive replica selection avoid the slow node, and set timeouts that return partial results.

Cheat sheet

How to say it in the interview

“Writes go only to Postgres. Debezium streams changes into Kafka; an indexer adds category, seller rating, sales and, when the title changes, an embedding, and bulk-writes with external versions, so a price is searchable in about 3 seconds. Rebuilds load a new index from a snapshot, replay Kafka and swap an alias. At query time I spell-correct, transliterate and extract filters and intent, take 500 by BM25 and 500 by kNN, fuse with RRF, cut to 200 with a cheap model and 50 with LambdaMART, apply diversity and ad slots, and hydrate live prices, in 200 ms at p99. Rankers are judged by NDCG offline, screened by interleaving and shipped by A/B test.”

How fresh is the index, and how do you know? About 3 seconds: CDC up to 0.5 s, batching 1 s, refresh 1 s. I record now minus the commit time at bulk acknowledgement, alert on its p99, and probe with a canary product.

How do you change the analyser or embedding model? Load a new index from a snapshot, replay Kafka from offsets recorded first, validate counts and golden queries while both consumers run, swap the alias atomically, and keep the old index fed for 48 hours.

Why RRF instead of adding the scores? BM25 and cosine scores live on different scales, so a weighted sum needs normalising and tuning. RRF uses only ranks and needs no training; you lose score magnitudes.

How do you know a new ranker is better? NDCG@10 on a judgement list, then interleaving, then an A/B test on conversion with latency and zero-result guardrails.

Why not train the ranker on raw clicks? Users click result 1 even when result 2 is better. Use clicked-over-skipped pairs, propensity weights and exploration.

Where to go next

• Next in the series: Search case studies: how real companies built these pieces, compared side by side.
• Earlier parts: Types of search, Search data structures, Disk I/O and durability for search, Choosing a search engine, Elasticsearch in depth and How Google Search works.
• Design product search for an online store, Design typeahead, Design a web crawler, Design a search engine like Google and Design RAG search over company documents: the interview problems.
• Change data capture, Transactional outbox, Embeddings and semantic search and Vector databases: the ingestion pipeline and the vector half.

References and further reading

1. Cormack, Clarke and Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009)cormack.uwaterloo.caThe RRF formula, the constant k = 60 fixed in a pilot and never tuned again, and why k damps outlier rankings.
2. Burges, From RankNet to LambdaRank to LambdaMART: An Overview (Microsoft Research MSR-TR-2010-82)microsoft.comDCG with (2^l − 1) gains and log(1 + i) discounts, why NDCG has no useful gradient, ΔNDCG-weighted pairs, and the 2010 Yahoo! LTR Challenge win.
3. Järvelin and Kekäläinen, Cumulated Gain-Based Evaluation of IR Techniques (ACM TOIS, 2002)cc.gatech.eduThe original CG, DCG and normalised DCG on a four-point relevance scale, with log-of-rank discounting and the ideal vector.
4. Joachims, Granka, Pan, Hembrooke and Gay, Accurately Interpreting Clickthrough Data as Implicit Feedback (SIGIR 2005)cs.cornell.eduEye-tracking evidence of trust bias (swapped top two) and quality bias (reversed lists), and the Click > Skip Above strategy.
5. Chapelle, Joachims, Radlinski and Yue, Large-Scale Validation and Analysis of Interleaved Search Evaluation (ACM TOIS, 2012)cs.cornell.eduTeam-draft interleaving, 1–2 orders of magnitude less data than absolute metrics, and ten interleaved queries worth about one judged query.
6. Elasticsearch documentation: Search using Learning To Rankelastic.coLTR since 8.12 on some subscription levels, as a rescorer or (since 9.1) a retriever; window_size mandatory, at least from + size and constant across pages.
7. Elasticsearch documentation: Aliaseselastic.coMultiple alias actions applied atomically, so an alias never points at both indices during a swap; must_exist to fail the whole list.
8. Vespa documentation: Phased rankingdocs.vespa.aiFirst phase on content nodes, second phase on each node’s best 100 by default, global phase on the merged best 100, and reciprocal_rank with k = 60.
9. Debezium 3.7 documentation: PostgreSQL connectordebezium.iosource.ts_ms versus ts_ms for lag, MilliSecondsBehindSource, poll.interval.ms of 500 ms, slots retaining WAL during outages, incremental snapshots.
10. Haldar et al., Applying Deep Learning To Airbnb Search (KDD 2019)arxiv.orgFrom a hand-made scoring function to GBDT to neural nets; booked-versus-not-booked pairs weighted by ΔNDCG; A/B tests on conversions.