Flip the table. Each term lists the documents containing it.
cotton AND kurta walks two sorted lists together. Cost follows the lists’ lengths, not the table’s size.
Titles were stemmed. Why doesn’t “Kurtas” match?
Bad: Analyse only documents
The index holds kurta. The raw query asks for “Kurtas”.
Good: One analyzer, both sides
Stem and lowercase queries exactly like documents. Synonyms are the deliberate exception.
2×
the gain from repeating kurta 20 times
BM25 favours rare terms: kurta’s IDF is 5.30, cotton’s 2.30. Repeats saturate and long fields are discounted, so stuffing can’t buy rank.
Never edit in place. Each refresh writes a new segment.
New products are searchable after a refresh, ≈ 1 s. The translog, fsynced before the ack, keeps them durable. Merges drop deleted documents.
Which shard holds the best kurta? Ask them all.
Each shard returns just its top 10 IDs and scores. The coordinator merges them, then fetches documents only from the winners’ shards.
Every search asks every shard. Three costs follow.
- Deep pages: 5 shards × 10,000 hits, sorted to keep 10
- Local IDF: small or skewed shards rank differently
- Size shards 10–50 GB, not a thousand tiny ones
You write to the database, then Elasticsearch, and crash in between. The index silently drifts.
The fix
Stream committed changes with CDC; upsert by primary key with the row’s version, so stale updates get rejected.
Remember
Analyse text into terms; keep an inverted index in immutable segments; rank with BM25; fan out to shards, then fetch winners. Feed it by CDC as a rebuildable view, never the truth.