Embeddings and semantic search

An embedding is a learned map from an item (a word, a passage, a product, a photo, a user) to a fixed-length vector of numbers, trained so that items people treat as similar land close together. Semantic search embeds every document once, embeds each query with the same model, and returns the nearest vectors, so “light office wear for summer” finds linen kurtas that share none of those words. In an interview you must know what the vectors capture and miss, how to chunk, rerank and blend them with keyword search, what they cost, and how to prove retrieval works.

Picture it first: a bookseller who shelves by feel

A bookseller at a Sunday book bazaar has 20,000 second-hand books on the pavement, arranged by feel rather than title: short stories in one corner, thrillers in another, cricket memoirs by the wall. Ask for “something like Malgudi Days, but set in a city” and he points at the nearest short-story piles, though no title shares your words.

His feel has limits. Ask for an ISBN and he is lost without the printed number. Ask for “anything not about cricket” and he drifts to the cricket wall, because “cricket” is what he heard. For the best five, he grabs thirty by feel and his daughter reads each back cover against your request. Fat anthologies are split so each story gets its own pile. And when a new partner arrives with a different sense of where things go, every book is re-shelved.

Map it back: a book’s spot is its embedding, his feel is the model, walking to your request is embedding the query with the same model, the nearest piles are the top k, the ISBN miss is why you keep lexical search, “not about cricket” is the negation blind spot, the daughter is a cross-encoder reranker, splitting anthologies is chunking, and the new partner is a model change that forces re-embedding.

What an embedding is

An embedding model maps an item to d floats, typically 384 to 3,072. No single number means “colour” or “formality”; meaning lives in directions and distances learned from data, and because query and documents share the model, “related” becomes “nearby”.

The idea arrived in steps, each behind a design choice you will defend:

• Words (2013). Mikolov et al.’s word2vec learned word vectors from 1.6 billion words in under a day with a skip-gram model that predicts neighbouring words; vector(King) − vector(Man) + vector(Woman) lands closest to Queen. But “bank” got one vector, beside “river” or “loan”.
• Contextual tokens (2018). BERT gives each token a vector that depends on its sentence, but raw outputs make poor sentence embeddings: Sentence-BERT (Reimers and Gurevych, EMNLP 2019) measured average similarity correlations of 29.19 for the [CLS] vector and 54.81 for averaged outputs, both below averaged GloVe vectors.
• Sentences (2019). Sentence-BERT fine-tuned BERT so one vector per sentence works with cosine. Finding the most similar pair among 10,000 sentences by feeding pairs to BERT takes 10,000 × 9,999 ÷ 2 = 49,995,000 model calls, about 65 hours on a V100; embedding each sentence once takes about 5 seconds.
• Passages (2020). Dense Passage Retrieval (Karpukhin et al.) trained question and passage encoders over 21 million Wikipedia passages and beat BM25 by 9–19 points of top-20 accuracy: 78.4% against 59.1% on Natural Questions.
• Images and text (2021). CLIP (Radford et al.) trained image and text encoders on 400 million image–caption pairs, so text can find photos.

What you store per item is small, and every field earns its place:

{
  "id": "hr-policy-04#chunk-3",
  "doc_id": "hr-policy-04",
  "text": "Leave policy › Earned leave: 18 days a year. Unused days carry forward, up to 45 days.",
  "embedding": [0.0213, -0.0087, 0.0441, "… 1,536 floats"],
  "model": "text-embedding-3-small@1536",
  "acl_groups": ["employees-in"],
  "updated_at": "2026-10-20T09:14:00Z"
}

How embedding models are trained

Almost every modern embedding model is trained contrastively: take positive pairs (a question and its answering passage, a query and the product bought, a caption and its photo) and train two encoders, the two towers, so a positive pair outscores the query paired with anything else. The negatives decide quality:

• In-batch negatives. In a batch of 128 pairs, each question’s 127 other passages are free negatives; DPR did exactly this.
• Hard negatives. Random ones are too easy. DPR added one BM25 result per question that shares its words but lacks the answer, which helped substantially; a second did not.

Two consequences follow. First, similar means whatever the training pairs meant: a model trained on paraphrases puts “iPhone 15 case” next to “iPhone 15”; one trained on purchases learns they differ. Second, queries and documents go through the same model, often with different prompts. The Sentence-Transformers docs separate symmetric search (question against question) from asymmetric search (short query, long passage); pick a model trained for yours. multilingual-e5 requires query: or passage: before every input, even in Hindi.

Similarity, precisely

• Cosine similarity, a · b ÷ (‖a‖ ‖b‖), compares directions only, from −1 to 1.
• Dot product, a · b, rewards length as well as direction.
• Euclidean (L2) distance, ‖a − b‖, is small for vectors close in absolute position.

For vectors of length 1 they agree: ‖a − b‖² = ‖a‖² + ‖b‖² − 2 a · b = 2 − 2 cos(a, b), so all three rank identically. OpenAI’s embedding guide normalises its vectors to length 1 for this reason: a dot product is cheaper than a cosine, with the same ranking.

So normalise once at write time and use the dot product, with the index metric the model was trained for. The deliberate exception: in a two-tower recommender, a longer item vector can encode popularity, and the dot product is the point.

Scores are relative, not probabilities

A cosine of 0.8 means nothing on its own. The multilingual-e5 model card notes that its scores cluster between 0.7 and 1.0, a side effect of a low training temperature (0.01), and that only relative order matters. Many models squeeze scores into a narrow band (anisotropy), so 0.82 is a strong match for one model and noise for another. If you need a threshold, say for “no good answer” in RAG or for duplicate detection, calibrate it on labelled pairs for that model and recalibrate when the model changes.

What embeddings are bad at

• Negation and constraints. “Not spicy”, “without onion” and “under ₹500” embed almost like their opposites; make constraints filters.
• Exact tokens. SKUs, error codes, PIN codes and names: “XJ-2041” and “XJ-2014” look alike.
• Numbers and dates. “Policy for 2024” and “for 2026” are near neighbours.
• Rare and out-of-domain terms. A drug name or internal project code the model never saw lands somewhere arbitrary.
• Long inputs. One vector for a 30-page contract averages away detail, and input past the limit is silently cut off.

Encoders: bi-encoders, cross-encoders and late interaction

Three ways to put a model between a query and a document; production systems usually combine two.

• A bi-encoder embeds query and document separately; relevance is a dot product. Documents are embedded once, offline, and an ANN index finds the nearest among millions in milliseconds. You gain speed and scale. You pay with accuracy: each document became one vector before anyone asked.
• A cross-encoder reads the query and one document together, with attention across both, and outputs a relevance score. You gain the best accuracy. You pay one model call per candidate: scoring all 40 million Quora questions this way would take over 50 hours per query, per the Sentence-BERT paper. So the Sentence-Transformers docs use it only to rerank about 100 candidates.
• Late interaction, from ColBERT (Khattab and Zaharia, SIGIR 2020), keeps a small vector per token; a document’s score sums, over query tokens, each one’s best match among the document’s tokens (MaxSim). Reranking 1,000 BM25 results, ColBERT was over 170× faster and used 13,900× fewer FLOPs than a BERT cross-encoder, at competitive quality. You pay in storage: a 300-token chunk at, say, 128 dimensions × 2 bytes is 300 × 256 B = 76.8 KB, against 3 KB for one 768-dimension float32 vector, about 25× more.

Retrieve, then rerank

The standard answer combines them: a cheap, high-recall first stage narrows millions of chunks to about 100, and an expensive, precise second stage picks the few the LLM will read, where a wrong passage becomes a confident wrong answer.

Do the arithmetic before you promise a reranker. Embedding a 12-token question is one tiny model call; reranking 50 candidates of about 500 tokens is 50 × 500 = 25,000 tokens of model work per query, about 2,000 times more. At 100 queries a second that is 2.5 million tokens a second: at an illustrative 100,000 tokens a second per GPU, about 25 GPUs. Twenty candidates and a smaller model cut that several-fold, which is why rerankers are common in RAG (tens of queries a second) and rarer in e-commerce search (thousands).

Chunking documents for RAG

A 30-page PDF embedded as one vector will not surface its paragraph about earned leave. Chunk because:

• Models have a limit: 512 tokens for multilingual-e5-large, 8,192 for OpenAI’s text-embedding-3 models; the rest is truncated.
• One vector should hold one idea. A chunk covering five topics sits between all five and close to none.
• The LLM’s context is a budget. Five 500-token chunks are 2,500 tokens; five whole documents could be 50,000.

The strategies, in rising order of effort:

• Fixed-size windows with overlap: every 500 tokens, overlapping by 50. Simple, and blind to structure.
• Structure-aware chunks: split on headings, paragraphs, table rows or code functions, then merge small pieces and split big ones to a target size. The default for structured documents.
• Semantic chunking: cut where consecutive sentences’ embeddings stop being similar. Costs an embedding pass; helps most on transcripts and other unstructured text.

Starting points, not facts: 300–800 tokens per chunk and 10–20% overlap, then tune on your evaluation set. Smaller chunks match questions precisely but lose context; bigger ones keep context but blur the vector and spend the LLM budget. Overlap rescues a straddling answer only if the answer is shorter than the overlap.

Four refinements earn their keep:

• Metadata on every chunk: document ID, section path, timestamps, language, and the source document’s permission groups, so retrieval can filter.
• Heading prefixes: prepend “Leave policy › Earned leave” before embedding, as in Fig. 4, so a chunk saying “this limit” still names its subject.
• Contextual prefixes: an LLM writes 50–100 tokens situating each chunk in its document. Anthropic reported 49% fewer failed top-20 retrievals with contextual embeddings and BM25, and 67% with reranking. You pay one LLM call per chunk at indexing time.
• Parent-document retrieval: search small chunks for precision, then give the LLM the enclosing section.

Worked sizing. A company has 10,000 documents × 20 pages × 500 tokens a page = 100 million tokens. With 500-token chunks and a 50-token overlap, each chunk advances 450 tokens: 100,000,000 ÷ 450 ≈ 222,000 chunks, or 222,000 × 500 = 111 million tokens to embed. At 1,536 float32 dimensions that is 222,000 × 6,144 B ≈ 1.4 GB of vectors, plus about 0.4 GB of text. Through an API at $0.02 per million tokens, embedding costs 111 × 0.02 ≈ $2.22. That is a pgvector-sized problem; vector databases covers when it stops being one.

Hybrid retrieval

Lexical and dense retrieval fail in opposite places, so serious systems run both.

• Lexical (BM25) wins on exact tokens: “error TS-999”, “iPhone 15 Pro Max 256 GB”, names, PIN codes, clause numbers. Even DPR lost to BM25 on SQuAD, whose questions were written while reading the passage and so share its words.
• Dense wins on paraphrase (“can I take unused leave next year” against “carry forward”), spelling variants and, if the model saw such data, code-mixed queries like “sasta wala phone 5G” and cross-lingual search, such as a Hindi question over English documents.

BM25 scores are unbounded and cosines are not, so fuse ranks with reciprocal rank fusion, summing 1 ÷ (60 + rank) over lists; search end to end works an example by hand, and types of search shows why scores don’t mix. Elasticsearch, OpenSearch, and Postgres with pgvector all do this in one query.

Choosing a model

Pick on six axes, then test on your data:

• Dimensions. 384 for small, fast models; 768 or 1,024 for base and large open models; 1,536 and 3,072 for OpenAI’s text-embedding-3-small and -large. Storage and ANN cost grow linearly with d.
• Input length. 512 tokens for many open encoders, 8,192 for the OpenAI models; your chunks must fit.
• Languages. multilingual-e5-large covers about 100 languages, and its card warns that low-resource ones may degrade. For Indian traffic, test Hindi, Tamil, Bengali and romanised Hinglish separately.
• Domain. Legal, medical, code and catalogues have their own vocabulary; fine-tuning on a few thousand of your own pairs often beats a bigger general model.
• Hosting. An API needs no GPUs and scales instantly, but your text leaves your network and the provider decides the model’s lifetime. A self-hosted open model stays in your VPC on a fixed version, and you run the GPUs.
• Price. In October 2026 OpenAI lists $0.02 per million tokens for text-embedding-3-small and $0.13 for -large.

Matryoshka shortening. Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022) trains one vector whose first 64, 128, 256 … dimensions each work alone, reporting up to 14× smaller embeddings at the same accuracy. OpenAI’s text-embedding-3 models (25 January 2024) use it: the dimensions parameter shortens the output, and -large cut to 256 dimensions still beats the older ada-002 at 1,536 on MTEB. Re-normalise any vector you truncate yourself. With int8 or binary quantisation, shortening is the cheapest memory lever.

Benchmarks. MTEB (Muennighoff et al., EACL 2023) covered 8 task types, 58 datasets and 112 languages, and found that no model wins every task; MMTEB (2025) spans over 500 tasks in 250+ languages. Use leaderboards to shortlist, never to pick.

Measuring retrieval quality

“The answers look good” is not a metric. Build a labelled set and measure three layers separately.

The set. Sample 300–500 real queries, stratified by topic, language and tenant, and mark the chunks that answer each. An LLM can propose labels cheaply; have humans spot-check about 10%. Version the set, and add every production failure to it.

The metrics (worked in search end to end):

• Recall@k: did a correct chunk make the top k? Measure it where you hand off: recall@50 for the reranker, recall@5 for the LLM. If recall@50 is 0.7, no reranker or prompt rescues the other 30%.
• MRR: how high the first correct chunk sits; suits one-answer questions.
• NDCG@10: graded relevance discounted by position, for lists people read.

The three layers fail independently:

1. ANN recall: does the index return what exact search would? A property of the index settings (vector databases).
2. Retrieval quality: do the exact top k contain the right chunks? A property of model, chunking and hybrid set-up.
3. Answer quality: is the answer correct and faithful to the chunks it cites? A property of prompt and LLM.

Swapping models while the index drops 10% of neighbours, or prompts while retrieval misses, tunes the wrong layer.

Cost and scale

Embedding throughput. Self-hosted, measure tokens per second per GPU for your model and chunk length. At an illustrative 100,000 tokens a second, the 111-million-token corpus above takes 1,110 s, about 19 minutes on one GPU; 50 million product titles of about 50 tokens (2.5 billion tokens) take about 7 hours, matching search end to end’s 2,000 products a second. Through an API, your tokens-per-minute quota is the limit, and those 2.5 billion tokens cost 2,500 × $0.02 = $50 on text-embedding-3-small, $325 on -large.

Batch and online. Backfills run as batch jobs. Edits flow through change data capture to an embedding worker that re-embeds only chunks whose text hash changed.

Queries. 5 million a day × 12 tokens = 60 million tokens, $1.20 a day through the API, so query embedding is about latency, not money: an API round trip adds tens of milliseconds and a dependency; a small self-hosted encoder takes a few. Traffic is skewed, so cache embeddings of normalised head queries, keyed by model version.

Storage. N × d × bytes: 1.4 GB for the corpus above; 100 million × 768 × 4 B = 307 GB in the vector databases example. Shorten, quantise, or both.

Re-embedding on a model change. Two models’ vectors live in different spaces, often of different dimensions, so a new model means re-embedding everything. Build the new index beside the old, backfill, dual-write edits, compare recall@k on the labelled set, then switch an alias and keep the old index for rollback: the blue-green rebuild applied to vectors, with the patterns of online data migrations.

Limits and numbers

Figures from the papers are for their year and hardware; product limits and prices are from current documentation (October 2026).

When to pick it, and when not

Pick embeddings when users describe what they want instead of naming it: questions over documents, product discovery, “more like this”, cross-lingual or image search, and deduplication where wording varies.

Pick something else, or add it, when:

• Queries are identifiers. Order IDs, SKUs and error codes want a key lookup or BM25.
• The constraint is structured. “Under ₹500, size M, in stock” is a filter, not a similarity.
• Silent errors are unaffordable. Legal or medical retrieval needs a reranker, citations and an evaluation set.
• The corpus is tiny. Under about 200,000 tokens, Anthropic suggests putting it all in the prompt instead.

Common interview usage

Going deeper: failure modes

• Query–document asymmetry. A six-word query and a 500-word passage look different to a symmetric model. Use an asymmetric model with its prefixes, and check recall separately for short and long queries.
• Stale embeddings. A document edited this morning keeps last week’s vector if the embedding worker lags or drops updates. Track embedding lag like any CDC lag, and alert on it.
• Chunk boundaries split the answer, as in Fig. 4. Chunk by structure, prefix headings, and test questions whose answers sit near boundaries.
• Permission leaks in RAG. Filtering after generation is too late: the LLM has already read the HR file. Filter by the user’s ACL groups inside retrieval, sync ACLs from the source, and treat a chunk without ACL metadata as private (multi-tenancy and isolation).
• Drift and domain shift. New products, slang and policy names arrive after training; the text-embedding-3 models know nothing after September 2021. Watch recall@k on fresh queries.
• Plausible but wrong. “Not gluten-free” or a cosine-above-0.8 rule copied from an old model yields confident nonsense. Turn constraints into filters and recalibrate thresholds per model.
• PII in vectors. Embeddings are not anonymised: Morris et al. (Vec2Text, 2023) reconstructed 92% of 32-token inputs exactly from their embeddings, including full names from clinical notes. Protect vector stores like the source text (encryption and key management), and delete vectors with their documents.

Cheat sheet

How to say it in the interview

“I’d chunk documents by headings to about 500 tokens, prefix each chunk with its heading path, and store tenant, ACL groups and model version with every vector. Ten thousand documents make about 222,000 chunks, 1.4 GB of vectors and a couple of dollars of embedding, so pgvector is enough. At query time I embed the question with the same model, run kNN and BM25 in parallel with the ACL filter inside both, fuse with RRF, rerank 50 with a cross-encoder and pass the top 5 to the LLM, in about 100 ms before generation. I track recall@50 and recall@5 on labelled real questions, and a model upgrade is a full re-embed into a new index behind an alias.”

Why not just use cosine similarity with a threshold? Scores are relative to the model: multilingual-e5 puts most pairs between 0.7 and 1.0. I rank by similarity, and calibrate any cut-off on labelled pairs for that model version.

Why rerank if vector search works? A bi-encoder squeezes each passage into one vector before seeing the question; a cross-encoder reads both together and orders the top 50 far better. It costs a model call per candidate, so it only sees a short list.

How would you handle Hinglish or Hindi queries over English documents? A multilingual model tested per language and on romanised Hinglish, BM25 for brand and model names, and transliteration in query understanding. If recall stays weak, fine-tune on logged query–click pairs.

What changes when you switch embedding models? Everything is re-embedded, since the spaces are incompatible: a new index beside the old, a backfill (2.5 billion tokens is about 7 GPU-hours or $50 through an API), dual writes, a recall@k comparison, then an alias swap with the old index kept for rollback.

How do you stop RAG leaking documents a user can’t see? ACL groups on every chunk, synced from the source, filtered inside retrieval for both BM25 and kNN, never after generation. A chunk with no ACL is private.

Where to go next

• Vector databases: ANN indexes, filtering, memory arithmetic and products.
• Types of search and how search engines work: lexical, semantic and hybrid search, precision and recall.
• Search end to end: RRF and NDCG by hand, the ranking funnel and blue-green rebuilds.
• Search case studies: Amazon’s two-tower semantic product search.
• Elasticsearch and PostgreSQL: where hybrid search often lives first.
• LLM inference serving and design RAG search over company documents: the generation half and the full design.

References and further reading

1. Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)arxiv.orgThe word2vec CBOW and skip-gram models, training on 1.6 billion words in under a day, and the King − Man + Woman ≈ Queen example.
2. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers and Gurevych, EMNLP 2019)arxiv.orgWhy raw BERT [CLS] and averaged outputs score below averaged GloVe on similarity, and the 65 hours versus 5 seconds comparison for 10,000 sentences.
3. Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., EMNLP 2020)arxiv.orgThe dual encoder, in-batch and BM25 hard negatives, 78.4% versus 59.1% top-20 accuracy, the SQuAD exception, and indexing and query throughput.
4. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (Khattab and Zaharia, SIGIR 2020)arxiv.orgPer-token vectors, the MaxSim operator, 170× lower latency and 13,900× fewer FLOPs than a BERT reranker, and the index footprint.
5. Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)arxiv.orgCLIP: contrastive image–text training on 400 million pairs, putting images and captions in one space.
6. Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022)arxiv.orgTraining one embedding whose prefixes are usable on their own: up to 14× smaller at the same accuracy.
7. MTEB: Massive Text Embedding Benchmark (Muennighoff et al., EACL 2023)arxiv.org8 tasks, 58 datasets, 112 languages, and the finding that no single embedding method wins every task.
8. OpenAI API documentation: Vector embeddings guide and pricingdevelopers.openai.comtext-embedding-3 dimensions, 8,192-token input limit, the dimensions parameter, unit-length vectors, and per-token prices as listed in October 2026.
9. Sentence-Transformers documentation: Retrieve & Re-Ranksbert.netBi-encoder retrieval of about 100 candidates followed by cross-encoder re-ranking, and why cross-encoders cannot score a whole corpus.
10. intfloat/multilingual-e5-large model card (Hugging Face)huggingface.coThe query: and passage: prefixes, the 512-token limit, 100 languages with weaker low-resource coverage, and cosine scores clustered between 0.7 and 1.0.