Design RAG search over company documents

An “ask the company” assistant lets an employee type a question — “What is the LTA limit for a Scale III officer?” or “मेरी बची हुई छुट्टियाँ अगले साल carry forward होंगी?” — and get a short answer with links to the exact policy page it came from. The model knows nothing about your company. At question time you find a handful of relevant passages and tell the model to answer from them and cite them. That is retrieval-augmented generation (RAG), and the demo takes an afternoon. The production system is hard for reasons that have little to do with the model: 8 million documents in six systems that change all day, each with its own permission model; scanned circulars and Hindi text; questions that hinge on one exact circular number; and one rule that cannot bend — the assistant must never reveal a document the employee could not open in the source system. You will design it for a large Indian bank with 40,000 employees.

Understand the problem

Weak answers draw “vector database plus LLM” and stop. Say early where the difficulty is: retrieval quality and permissions. The model is a commodity you call through the LLM gateway. The original RAG paper by Lewis et al. (2020) paired a text generator with a retriever over a dense vector index, and named the two properties that matter most here: answers that show their provenance, and knowledge you can update without retraining — you re-index a document instead of fine-tuning a model.

Functional requirements

1. Users should be able to ask a question in English, Hindi or a mix of both and get a streamed answer with citations that open the exact page or section it came from.
2. Users should be able to find anything the bank has written down — Confluence, SharePoint, Google Drive, Jira, selected Slack channels and about 100,000 scanned circulars — with new and edited content searchable within minutes.
3. Users should be able to trust that answers only use documents they could open themselves in the source system: no answer, citation, snippet or even title from anything else.
4. Users should be able to rate an answer and flag a wrong or outdated source, so the team can measure and fix quality.

Non-functional requirements

• Zero permission leaks. This is a correctness rule, not a percentage. A revoked permission or a deleted document stops being used within 5 minutes at p99, and an urgent purge within seconds.
• Freshness. New or edited text searchable within 15 minutes at p95; scanned PDFs, which need OCR, within an hour.
• Latency. First token within 2.5 s at p95; a complete answer within 10 s.
• Quality. The right chunk among the top 20 candidates for at least 90% of golden-set questions (recall@20); at least 95% of answer sentences supported by the source they cite; “I couldn’t find that” instead of a guess.
• Scale. 8M documents, 40,000 employees, about 10 questions a second at peak and 30 in a burst.
• Availability. 99.9% during business hours (09:00–19:00 IST). If the model is down, fall back to ranked search results.
• Residency and cost. Content, indexes and model calls stay in Indian regions; under $1 per employee per month, all-in.

Out of scope

• Model serving — batching, KV cache, GPUs: Design an LLM inference service.
• Provider routing, token budgets and prompt caching: the LLM gateway. We call it like any other team.
• Chat UI, conversation storage and resumable streams: Design ChatGPT. We reuse its SSE contract.
• Taking actions such as filing a Jira ticket or applying for leave: the AI agent platform.
• Training or fine-tuning on company data, and cleaning up the corpus itself (retention, classification). We honour labels; we don’t create them.

The set-up

Plan the approach

Say the shape in one sentence: “a write path that turns documents into permission-tagged chunks in a search index, and a read path that retrieves, reranks and generates from that index.” Satisfy each requirement simply, then spend most of the time in the deep dives; that is where this problem is won.

Core entities

• Source. A connected system and scope (the SharePoint site “HR Policies”), a credentials reference, a sync cursor and its lag.
• Document. doc_id (source plus the source’s own ID), title, URL, version, effective date, language, content hash, access-control list (ACL) and a deleted flag.
• Chunk. The unit of retrieval: text, a short context header, section path, page or anchor for the citation, token count, content hash, embedding-model version, the vector, and a copy of its document’s ACL.
• Principal. A user or group, mapped to one corporate identity. A user’s principal set is the user plus every group they belong to, directly or through nesting.
• Trace. One question: rewritten queries, candidates with scores, the chunks used, prompt and model versions, the answer, citations, latency and feedback.

One chunk, as the index stores it:

{
  "chunk_id": "sp:hr-policies/lta-2025.pdf#v7:c12",
  "doc_id": "sp:hr-policies/lta-2025.pdf",
  "doc_version": 7,
  "title": "Leave Travel Allowance Policy 2025",
  "section_path": "4 Eligibility > 4.2 Officers",
  "page": 4,
  "lang": "en",
  "context_header": "From the bank's 2025 LTA policy, effective 1 April 2025; limits for officers in Scales I-IV.",
  "text": "Officers in Scale III and above may claim LTA once in a block of two calendar years ...",
  "content_hash": "b3:9f1c07",
  "embedding_model": "emb-1024-v3",
  "vector": [0.0121, -0.0443, 0.0087],
  "acl_allow": ["grp:all-officers", "grp:hr-ops"],
  "acl_deny": [],
  "acl_version": 1932,
  "effective_from": "2025-04-01"
}

Copying the ACL onto every chunk is deliberate denormalisation: the search engine can then filter by permission inside the same query that ranks by relevance. The price is that a permission change on one document means rewriting all of its chunks.

API

Ask. One POST that returns a server-sent events stream, as in Design ChatGPT (see SSE and WebSockets for why SSE).

POST /v1/ask
Authorization: Bearer <SSO access token>
Accept: text/event-stream
Content-Type: application/json

{
  "conversation_id": "c_81f2",
  "question": "Scale III officer ka LTA limit kitna hai, aur claim kab tak submit karna hai?",
  "filters": { "sources": ["sharepoint", "confluence"] },
  "answer_language": "auto"
}

event: sources
data: {"trace":"t_5c9e","sources":[{"tag":"S1","title":"Leave Travel Allowance Policy 2025","url":"https://sp.bank.internal/hr/lta-2025.pdf#page=4","effective":"2025-04-01"},{"tag":"S2","title":"HR FAQ: travel claims","url":"https://wiki.bank.internal/hr/faq#travel"}]}

event: delta
data: {"text":"Scale III officers can claim LTA once in a block of two calendar years [S1]. "}

event: delta
data: {"text":"Submit the claim within 60 days of returning from the trip [S2]."}

event: done
data: {"trace":"t_5c9e","cited":["S1","S2"],"grounded":true,"usage":{"input":4710,"output":338}}

The sources event lists only chunks the user may open. When retrieval finds nothing good enough, the stream ends with "grounded": false and a refusal instead of a guess.

The rest of the surface:

GET  /v1/search?q=...&k=10                 ranked, permission-filtered chunks; the no-LLM fallback and the eval harness
POST /v1/answers/t_5c9e/feedback            body: rating up|down, reason wrong|outdated|incomplete, optional source tag
POST /v1/sources                            admin: register a connector (type, scope, credentials reference)
GET  /v1/sources/src_sp_hr/status           admin: lag, cursor, error counts, documents indexed
POST /v1/sources/src_sp_hr:resync           admin: full crawl and reconcile
POST /v1/documents/{doc_id}:purge           compliance: tombstone now, delete everywhere after

Back-of-envelope estimates

These are assumptions; say so out loud. (Method: back-of-envelope estimation.)

Corpus. 4M SharePoint and Drive files, 1.5M Confluence pages, 2M Jira issues, 0.4M Slack threads and 0.1M scanned circulars: 8M documents. At an average of 5 pages each that is 40M pages, and at about 500 tokens a page, 40M × 500 = 20B tokens of text.

Chunks. 400-token chunks with a 50-token overlap advance 350 tokens at a time: 20B ÷ 350 ≈ 57M. Short documents — a Jira issue, a Slack thread — still make one whole chunk each, so call it 60M chunks. Each chunk is embedded with a 50-token context header, so the embedding model reads 60M × 450 = 27B tokens.

Raw vectors. Use 1,024 dimensions — one provider’s large embedding model shortened with its dimensions parameter, or an open multilingual model of the same width hosted in India. In float32 a vector is 1,024 × 4 B = 4,096 B, so 60M × 4,096 B ≈ 246 GB.

HNSW graph. With M = 16, the bottom layer of an HNSW graph keeps up to 2 × M = 32 neighbour IDs per vector at 4 bytes each: 128 B. For the upper layers, the HNSW paper’s memory estimate adds mL × M links per vector, with mL = 1/ln(M) ≈ 0.36 (a generous bound, since only one vector in 16 reaches layer 1): 0.36 × 16 × 4 B ≈ 23 B. That is about 150 B a vector, 60M × 150 B ≈ 9 GB — under 4% of the float32 vectors. At 1,024 dimensions the vectors, not the graph, dominate memory.

Quantization. Half precision stores 2 bytes a dimension: 123 GB. int8 stores 1: 61 GB. Binary stores 1 bit: 1,024 ÷ 8 = 128 B a vector, 7.7 GB — now smaller than the graph. Add the 9 GB graph to each: 255, 132, 70 and 17 GB per copy of the index (drawn in the index deep dive). Keep two copies in the primary region and one in the disaster-recovery region.

One-time embedding cost. At one provider’s October 2026 list price of $0.13 per million tokens for its large embedding model: 27,000 million × $0.13 = $3,510, about ₹3.1 lakh at ₹88 to the dollar. Its small model at $0.02 would cost $540. Writing the context headers with an LLM costs more: Anthropic estimated $1.02 per million document tokens, so 20B tokens ≈ $20,400 — six times the embeddings, and still a rounding error next to a year of answers.

Daily churn. Assume 1% of documents change a day: 80,000 documents × 7.5 chunks each (60M ÷ 8M) = 600,000 chunks. If chunk hashing shows only a third really changed, 200,000 × 450 = 90M tokens ≈ $12 a day; re-embedding every chunk of every changed document would be $35. The embedding bill is not why you hash chunks; index churn and the LLM-written headers are.

Questions. Assume 40% of employees use it on a working day and ask 5 questions each: 16,000 × 5 = 80,000 questions a day. Squeezed into a 9-hour working day: 80,000 ÷ 32,400 s ≈ 2.5 questions/s; a 3× peak hour gives 7.5, so plan for 10/s, and 30/s in a burst — a new circular lands, or it’s appraisal week. Each question runs about 3 query variants against 2 retrievers: 60 index queries/s at peak. For a search cluster that is nothing. Throughput is not this design’s problem.

Tokens and cost per answer. System prompt and rules 700 tokens, conversation summary 300, 8 chunks × 450 = 3,600, question 100: 4,700 input tokens, and about 350 output tokens. At one provider’s October 2026 price for a mid-tier model, $2 per million input and $10 per million output tokens:

• Generation: 4,700 × $2/M + 350 × $10/M = $0.0094 + $0.0035 = $0.0129
• Query rewrite on a small model ($0.10 / $0.50): 500 × $0.10/M + 80 × $0.50/M ≈ $0.0001
• Reranker and query embedder on two GPUs at an illustrative $2.50 an hour: $120 a day ÷ 80,000 questions ≈ $0.0015
• Total ≈ $14.50 per 1,000 questions, $1,160 a day, about $25,500 a month over 22 working days — $0.64 per employee. In-region processing adds 10% to the model calls on that price list.

The retrieved chunks are 3,600 of the 4,700 input tokens, 77%. The number of chunks is your biggest cost lever: 20 chunks instead of 8 makes the prompt 10,100 tokens and the answer cost 1.8× as much. A cached system prompt saves about 10%, which the gateway handles.

Latency budget. Most of it is the model; most of the rest is the query rewrite.

High-level design

Build one requirement at a time. By the end you have a working assistant; the deep dives make it accurate, safe and affordable.

Users should be able to ask a question and get a cited answer

Start with the read path and assume an index already exists, built by a nightly batch job for now. An Ask service verifies the employee’s SSO token, asks a Retriever for the eight most relevant chunks, builds a prompt and calls the model through the gateway, relaying tokens to the browser as SSE.

The prompt is where grounding starts. Each chunk goes in as a numbered source with its title, section and effective date, and the rules say: answer only from these sources, put a tag such as [S1] after each claim, and say so when they don’t answer the question. Afterwards the Ask service checks that every tag exists and turns it into a link. Streaming, stop and resume work as in Design ChatGPT.

Users should be able to find anything the bank has written down

Replace the batch loader with an ingestion pipeline. One connector per source type receives the source’s webhooks, polls a “changes since cursor” API as a backstop, and fetches each changed document with its permissions. It stores the raw file in blob storage and publishes a small change event — doc_id, version, and upsert, delete or ACL change — to a Kafka topic keyed by doc_id, so one document’s events stay in order.

Workers parse (HTML, DOCX, slides, the PDF text layer), OCR scanned pages in English and Hindi, detect the language, chunk and embed in batches. An index writer upserts chunks into the search index and records versions, chunk hashes and ACLs in a doc store.

Two details keep it healthy. Priority lanes: deletes and permission changes travel on their own topic ahead of edits, and OCR runs in its own worker pool, so a 400-page scanned manual never delays a Slack thread. And OCR is the slow step: at an assumed 2 s per page per core, the backlog of about 400,000 scanned pages is 800,000 core-seconds, about 222 core-hours — three and a half hours on a 64-core pool, then a trickle. As in any queue-driven pipeline, every stage must be idempotent.

Users should be able to trust that answers only use documents they could open

Add two things. Connectors translate each source’s permissions into a common form — allow and deny lists of principal IDs — and the index writer copies them onto every chunk. A Principals service turns the employee into a principal set: their user ID plus every group, resolved through nesting. The Retriever adds that set as a filter to both the keyword query and the vector query.

Say the rule out loud: the model must never see a chunk the user cannot open, because anything in the prompt can come out in the answer. So the filter runs before prompt assembly, never after generation, and the same principal set filters snippets, titles, suggestions and any cache. Doing that without wrecking recall — and keeping up when permissions change — is the fourth deep dive.

Users should be able to rate an answer and flag a bad source

Every answer gets a trace ID. The Ask service writes a trace — question, rewrites, candidates with scores, the eight chunks used, prompt and model versions, answer, citations and timings — and feedback joins it by ID. Traces quote documents, so they inherit the documents’ ACLs, are readable only by the asker and a small audited quality team, and expire after 90 days. Answers flagged “wrong source” become candidate golden-set questions.

You now have an assistant that is fresh, permission-filtered and cited — but not yet good. Fixed chunks lose context, vector search alone misses circular numbers, and 246 GB of float32 vectors won’t sit comfortably on one machine. The deep dives fix that.

Deep dives

How do we split documents so the right passage can be found?

You must split. A 200-page credit manual squeezed into one vector is an average of everything in it and matches no question well; embedding models cap their input (8,192 tokens for one provider’s models); and the prompt has room for about eight passages, not eight manuals. How you split decides whether the passage that answers a question even exists as something you can retrieve.

Bad: fixed windows. Cut every 400 tokens with a 50-token overlap, ignoring structure. You gain simplicity and predictable sizes, so cost estimates are easy. You pay in meaning: cuts land mid-paragraph and mid-table, a rule is separated from its exception, and many chunks lose their subject. “Officers in Scale III and above may claim once in a block of two calendar years” — which allowance, in which policy, from which year? The overlap only patches the edges.

Good: structure-aware chunks. Parse each document into a tree — headings, sections, lists, tables — and pack whole sections into chunks of at most 400 tokens, splitting long sections at paragraph boundaries. Never split a table row; repeat the header row in every piece of a long table. Keep the section path (“LTA Policy 2025 › 4 Eligibility › 4.2 Officers”) as metadata and prepend it to the text you embed and index. Sources get their own rules: a Slack thread is one chunk; a Jira issue is summary, description and the resolving comment; a scanned circular splits by page and by the headings OCR recovers. You gain self-contained chunks and citations that land on a section. You pay for a parser per format and uneven sizes, and documents without headings fall back to fixed windows anyway.

Great: contextual headers and parent–child retrieval. Two additions on top of structure:

• Contextual header. An LLM reads the document and writes 50–100 tokens that place each chunk in it — “From the bank’s 2025 LTA policy, effective 1 April 2025; this section sets claim limits for officers in Scales I–IV” — prepended before embedding and before keyword indexing. In Anthropic’s published experiments this cut top-20 retrieval failures by 35% with contextual embeddings alone and by 49% when the keyword index used the headers too (5.7% to 2.9%). You pay one LLM call per chunk: about $20,000 once for our corpus, then about $80 a day for the 200,000 changed chunks (80M tokens at Anthropic’s $1.02 per million).
• Parent–child. Search small, precise chunks but give the model their parent section, or the neighbouring chunks, so the answer has the surrounding conditions. You pay for larger prompts — and if you shrink children to 200 tokens, twice the vectors.

The choice: structure-aware 400-token chunks with a contextual header on every chunk, and parent expansion only for long policy manuals and SOPs, where an answer usually depends on the conditions around it. Chunking must also be deterministic — the same document always splits the same way — or every edit reshuffles the boundaries and the freshness deep dive’s hashing stops saving anything.

How do we find the right chunks for a question?

Two retrieval families fail in opposite ways. Dense retrieval embeds the question and finds the nearest chunk vectors (see embeddings and semantic search). It matches meaning — “money for a holiday trip” finds the LTA policy, and a good multilingual model matches “छुट्टी” to “leave” — but it is weak on exact strings such as a circular number (HO/OPS/2025/118), a Jira key (PAY-4812), a form code or a branch code. Sparse retrieval with BM25 over an inverted index is the opposite: rare exact tokens win, but synonyms, paraphrases and cross-language matches are lost.

Bad: vectors only. The demo default. You gain one index and good paraphrase matching. You pay on every question that names an identifier, and in a bank that is a large share: circulars, products, forms and tickets all have codes.

Good: hybrid search fused with reciprocal rank fusion. Run BM25 and kNN in parallel, both with the permission filter, take the top 50 from each, and merge. Raw scores aren’t comparable — BM25 is unbounded, cosine similarity sits between −1 and 1 — so fuse ranks. Reciprocal rank fusion (Cormack, Clarke and Büttcher, 2009) gives each document

RRF(d) = Σ over retrievers r of 1 / (k + rank_r(d))

with k = 60, a value the authors fixed in a pilot study and found near-optimal but not critical. Elasticsearch’s RRF retriever uses the same formula with rank_constant defaulting to 60, and pgvector’s documentation points to RRF for hybrid search in Postgres. You gain robustness with nothing to tune: a chunk both retrievers like rises above one only one of them likes. You pay for two queries per question, and RRF throws away how confident each retriever was — a perfect BM25 hit on a circular number earns no more than a lukewarm rank 1 from vectors.

Before retrieval, a small model rewrites the question: a follow-up (“and for Scale IV?”) becomes a standalone question, acronyms are expanded, and a Hindi question gets an English variant so BM25 has policy terms to match. You gain recall; you pay about 400 ms and another call, so skip it when the question already stands alone.

Great: hybrid, then a cross-encoder reranker. The embedding model is a bi-encoder: question and chunk are encoded separately, which is what makes indexing possible, but it compares them only through two vectors. A cross-encoder reads question and chunk together and scores their relevance — much more accurate, and far too slow to run over 60M chunks. So run it on the top 50 fused candidates and keep 8. Anthropic’s experiments reranked 150 candidates down to 20 and, combined with contextual chunks, cut retrieval failures by 67% (5.7% to 1.9%). You pay about 150 ms on a GPU for 50 pairs (our assumption), a GPU to run, and a reranker that must understand Hindi as well as English. After reranking, apply cheap business rules: prefer the newest effective version of a policy, prefer the policy library over Slack, and demote anything marked superseded.

How do we store and search 60 million vectors quickly?

There are three families of vector index (more in vector databases):

• Flat (exact). Compare the query with every vector. Perfect recall, no build, no extra memory — and hopeless at our size: scanning 246 GB per query at an assumed 100 GB/s of memory bandwidth takes about 2.5 s. Flat is right for small collections: a 200,000-chunk silo is 820 MB, about 8 ms a scan.
• IVF (inverted file). Cluster the vectors with k-means into lists and scan only the nprobe lists nearest the query. pgvector suggests √rows lists above a million rows — √60M ≈ 7,700 — and √lists ≈ 88 probes to start. You gain low memory overhead and fast builds. You pay in recall at the same speed (neighbours just across a cluster boundary are missed), a training step, and clusters that drift as content changes, so you retrain.
• HNSW (hierarchical navigable small world). A layered proximity graph: sparse upper layers act as express lanes, and a search descends greedily, then explores a candidate list of size ef_search on the bottom layer, where each node keeps up to 2 × M links — the 150 bytes per vector computed earlier. You gain the best speed–recall trade-off and inserts without training. You pay for graph memory, slow builds and awkward deletes: Faiss’s HNSW cannot remove vectors at all, and pgvector’s HNSW vacuums are slow unless you reindex first.

The knobs, with pgvector’s defaults (v0.8.7): m = 16 links, ef_construction = 64 for build quality, hnsw.ef_search = 40 at query time. One trap: an HNSW scan returns at most ef_search results, so if the reranker wants 50 candidates, the default 40 silently gives you fewer. Set it to about 100 and measure recall against exact search on a sample, as pgvector’s docs suggest.

Memory is what decides the hardware, and quantization is the main lever.

Half precision costs almost no recall. int8 and binary cost more, which you buy back by rescoring: fetch the top 100 by the compressed vectors, then reorder them by the full-precision vectors read from SSD. pgvector shows exactly this pattern for binary quantization. Shrinking dimensions is the other lever: the same provider’s model emits 3,072 dimensions by default, and asking for 1,024 cuts storage three-fold before any quantization.

Then decide where the index lives.

The pick: a search engine (see Elasticsearch) holding text, BM25 postings, an int8 HNSW field and the ACL fields together. Shard by chunk_id into 6 primary shards of about 12 GB of vectors each, keep two replicas in the primary region and one in the DR region, and store float32 vectors on SSD for rescoring. The small, highly sensitive collections get their own flat indexes, which the next deep dive explains.

How do we never show a document the employee cannot open?

Start by mapping permissions. Every source has its own model — SharePoint inheritance with items that break it, Confluence space permissions plus page restrictions, Drive sharing and links, Jira project roles and issue security levels, Slack public and private channels. Each connector translates its model into one form, allow and deny lists of principal IDs, and maps every account to one corporate identity by employee ID. Three rules keep this safe: an account that cannot be mapped grants nothing; a document whose permissions could not be read gets an empty allow list until fixed (fail closed); and a file shared “with anyone who has the link” counts as visible only to its explicit principals.

Expand groups on the user side, not the document side. If “all-officers” were expanded into 25,000 user IDs on every chunk, one person joining would rewrite millions of chunks. Instead chunks store group IDs, and the Principals service computes each employee’s full set — nested directory groups, Confluence groups, Slack channels, Jira roles — caches it for 5 minutes, and drops the entry on any membership event. A typical employee has a few dozen principals, an administrator a few hundred: still a small filter.

Bad: search, then filter. Retrieve the top 40 from the whole index and drop what the user can’t open. Recall collapses for anyone who can see only a slice of the bank. pgvector’s documentation gives the arithmetic: with a filter matching 10% of rows and the default ef_search of 40, an HNSW query returns about 4 matching rows. A branch officer who can open 5% of chunks keeps 40 × 5% ≈ 2. Worse, the filter is now a step every code path must remember — citations, snippets, the cache — and one forgotten path is a leak. The worst version filters after generation, when the model has already read the text.

Good: filter inside the search. Put the ACL filter in both queries. On the keyword side it is an ordinary filter clause. On the vector side it needs filtered ANN, which skips disallowed nodes while walking the graph and keeps going until it has k allowed results; pgvector 0.8 added iterative index scans for this, bounded by hnsw.max_scan_tuples (20,000 by default). You gain a true top-k for every user and one rule in one place. You pay latency for very selective users — to collect 40 allowed results at 5% the walk visits roughly 40 ÷ 5% = 800 nodes — and a document’s permission change must re-tag every one of its chunks.

Great: filter inside the search, check again before the prompt, and wall off the most sensitive collections.

• Late check. Just before prompt assembly, check the eight chosen chunks against the ACL store’s current version of each document and the user’s fresh principal set: eight key lookups, about 5 ms. The ACL store is updated straight from permission events within seconds, while re-tagging chunks in the index can take minutes.
• Silos. Board papers, HR investigation files and treasury deal notes go in separate small indexes, searched only for members of the owning group. A shared ANN index also lets one group’s vectors affect another’s recall and speed — pgvector recommends partitions or separate tables for tenant isolation for the same reason — and a 200,000-chunk silo is small enough for an exact flat search.

You gain defence in depth and a hard wall where it matters most. You pay with more moving parts, a fan-out when a user belongs to several silos, and duplicate copies of documents shared across walls. More on the trade-offs in multi-tenancy and isolation.

Revocation is a race; size the window. Two kinds of change reach the system differently. When Ravi leaves the treasury desk, the directory emits a membership event and the Principals service drops his cached set — exposure is the event delay, or the 5-minute cache lifetime if the event is lost. When a document is restricted, the index must re-tag every chunk, and that is where the late check earns its keep.

Webhooks get lost, so back them with a delta poll every few minutes and a nightly reconciliation that compares every document’s permissions in the source with the ACL store. For a compliance emergency the purge API tombstones a document everywhere within seconds.

Finally, the side doors. An answer cache shared between users leaks unless its key includes the user’s principal set and the documents’ versions; simplest is no cross-user answer caching at all (the gateway’s semantic cache stays off for this app). Traces and logs quote documents, so they carry the same restrictions.

How do we keep the index fresh when documents change, move or get deleted?

Bad: a nightly full re-crawl. Simple and self-healing — whatever went wrong yesterday is fixed tonight. But fetching 8M documents a night runs into every source’s API rate limits, answers can be a day stale, and a withdrawn circular keeps answering questions until tomorrow.

Good: change feeds and incremental updates. Treat each source as a database with a change stream, in the spirit of change data capture: webhooks for prompt notice, plus a “changes since cursor” poll every few minutes because webhooks get dropped. Each event carries the document’s version, and a worker ignores any version not newer than the one indexed, so replays and duplicates are harmless (see idempotency). You gain minutes instead of hours. You pay for per-source quirks, and you still need to catch what the feeds miss.

Great: diff at the chunk level, tombstone first, reconcile always.

• Hash every chunk — normalised text plus section path — and on a new version re-chunk and compare. Embed only new hashes, delete vanished ones, keep the rest. A typo fixed in a 60-chunk manual re-embeds one chunk, not sixty, and touches one entry in the index instead of sixty deletes and inserts. This is why the chunker must be deterministic.
• Tombstone deletes. Mark the document deleted in the doc store within seconds and drop tombstoned doc_ids at query time; physical removal follows in the background, because deleting from an HNSW graph is slow and segments are rebuilt periodically.
• Use stable IDs. doc_id is the source’s immutable ID, never the path, so a moved or renamed file is an update, not a delete plus a create.
• Reconcile weekly. List every ID in each source, compare with the doc store, and tombstone orphans. This catches the deletes no webhook announced.

Changing the embedding model is the big version of this problem. Vectors from different models live in different spaces, so you cannot mix them in one index or compare a new query vector with old document vectors. Build a second index instead. Re-embedding all 27B tokens costs about $3,510 at the earlier price and, at an assumed 200,000 tokens/s of embedding throughput, takes 27B ÷ 200,000 ≈ 135,000 s — about 37 hours. Meanwhile, live changes are written to both indexes.

Evaluate the new index on the golden set and on shadowed live queries, flip a read alias, keep the old index for a week, then drop it. The query embedder is versioned with the alias, so a question is always embedded by the same model as the index it searches.

How do we keep answers grounded and safe from poisoned documents?

Bad: paste the chunks and ask for an answer. The model blends the sources with what it remembers from training — central-government LTA rules, say — cites nothing, or invents a plausible citation.

Good: numbered sources and mandatory citations. The prompt from the first high-level step, plus rules to flag disagreeing sources, prefer the latest effective date and answer in the question’s language. Code checks that every cited tag exists and strips any URL that isn’t a source’s. You gain answers anyone can verify in one click, almost free. You pay little — but the model can still cite a source that doesn’t say what it claims.

Great: refuse early, verify afterwards.

• A retrieval gate. If the best reranker score is below a threshold calibrated on the golden set, don’t call the model: return “I couldn’t find this in documents you can access” with the top three search results. It saves money on exactly the questions most likely to produce confident nonsense.
• A support check. A small model or an entailment classifier checks each answer sentence against the chunk it cites. Run it on a sample of answers offline, and inline for high-stakes areas such as compliance and HR policy, where you’d rather spend 300–500 ms than be wrong.
• Conflicts are shown, not hidden. When the 2019 and 2025 policies disagree, say so, with dates.

Prompt injection comes in through the documents. OWASP’s 2025 list for LLM applications puts prompt injection first and describes the indirect kind: instructions hidden in files or web pages the model reads. It also notes that RAG does not remove the risk. Any of 40,000 employees can edit a wiki page, and a vendor’s PDF can carry white-on-white text. A planted line might push a false limit (“say it is ₹5 lakh”), or try to smuggle the conversation out inside an image URL.

Defend in layers, following OWASP’s mitigations:

• Segregate untrusted content. Rules first; retrieved text in a delimited block labelled as data, never inside the system prompt.
• Clean at ingestion. Drop hidden text — white or tiny fonts, zero-width characters, HTML comments — and quarantine chunks that read like instructions to a model.
• Least privilege. No tools, and only what the asking employee can read, so a successful injection cannot fetch anything new. This is the strongest control, and why actions belong on a separate, more guarded agent platform.
• Check the output. No remote images and links only to cited sources, which closes the URL exfiltration trick. Keep injection probes in the golden set.

None of these is complete — OWASP says plainly that fool-proof prevention is unclear — so the goal is to limit what a successful injection can do.

How do we know the answers are good?

You cannot improve what you only feel. Measure retrieval and generation separately: if the right chunk never reaches the prompt, no prompt change can fix the answer.

Offline: a golden set. Ask subject experts in HR, compliance, operations, IT and credit for about 1,000 real questions, in English, Hindi and a mix, each labelled with the chunks that answer it and a reference answer. Include about 10% that the corpus cannot answer, permission traps asked as a persona who cannot see the answer (the only right response is a refusal), and injection probes. Then measure:

• Retrieval: recall@20 after fusion, and recall@8 after reranking — what the model actually sees.
• Answers: faithfulness (the share of sentences supported by the cited chunk), correctness against the reference, citation precision, and refusal accuracy.

LLM-as-judge, with care. A judge model scales grading to thousands of answers, but judges can favour longer answers, be swayed by the order in which answers are shown, and prefer text that sounds like their own model family. Calibrate the judge against about 200 answers graded by people, report the agreement, pin the judge’s version, and recalibrate when you change it. A full run costs 1,000 × ($0.0129 for the answer + about $0.01 to judge) ≈ $23, so run it on every change to the chunker, embedding model, ranker, prompt or model, and block the change if any headline metric drops more than a point.

Online. Watch thumbs-down rate and “wrong source” flags by department, citation click-through, refusal rate and latency. Cluster refused questions weekly: 300 questions about a new gratuity rule with no matching document is a message for HR, not the search team. Try new rankers on shadow traffic first, and feed reviewed failures back into the golden set.

The final design

Follow one question, then one document:

1. Ask. The employee’s question arrives at the Ask service with an SSO token over a streaming request.
2. Resolve principals. The Principals service returns the user and every group, from a 5-minute cache that membership events invalidate.
3. Retrieve. The Retriever rewrites the question if needed, runs BM25 and int8 HNSW queries with the ACL filter inside both — plus the silo indexes the user belongs to — and fuses the top 50 from each with RRF.
4. Rerank. A cross-encoder scores the top 50 and keeps 8. If even the best score is below the threshold, the Ask service refuses and shows search results instead.
5. Check and assemble. The eight chunks are checked against the current ACLs in the Doc + ACL DB and laid out as numbered, untrusted sources under the system rules.
6. Generate. The LLM gateway sends the prompt to a model hosted in India; tokens stream back through the Ask service, which validates citations and turns them into links.
7. Learn. The trace and any feedback are stored, and the evaluation jobs replay the golden set on every change.
8. Ingest. Connectors turn webhooks and delta polls into events on the change log; parse, OCR, chunk and embed workers process them in priority lanes; the index writer diffs chunk hashes, applies tombstones, and updates the search index and the Doc + ACL DB.

Check it against the requirements: about 1.4 s to first token at p50; edits searchable in minutes and revocations effective in about 30 seconds thanks to the late check; 70 GB of int8 index per copy instead of 255 GB; and about $14.50 per thousand answers, or $0.64 per employee a month.

What interviewers look for by level

Follow-up questions

What if a better embedding model comes out?

Don’t switch on a benchmark. Run your golden set against a sample index built with the new model, and switch only if recall@20 improves enough to pay for the migration: about $3,510 of embeddings, a day and a half of backfill and ten days of double memory, using the dual-index plan above. If the new model emits more dimensions, check memory first; shortening its vectors with a dimensions-style option may keep most of the gain.

What if employees ask in Hindi or Hinglish?

Use a multilingual embedding model so “छुट्टी” and “leave” land close together, and let the rewrite step add an English variant, since most policies are written in English. BM25 needs a language-aware analyzer per field, and transliterated words such as “chhutti” need a synonym list, because neither analyzer will match them to “छुट्टी”. Answer in the language of the question, keep citations pointing at the original documents, and put Hindi and mixed questions in the golden set — they fail differently.

What if the answer is in a table, a 300-page manual or a bad scan?

Tables are chunked as tables: Markdown with the header row repeated in every piece, and a caption naming the table. Long manuals get a section tree, so a chunk can bring its parent section or neighbours along. Scans go through OCR with per-page confidence; low-confidence pages are flagged in the citation (“scanned page, may contain errors”) and queued for re-OCR or manual review. If a regulator’s circular exists only as a bad scan, it is worth a human transcribing it once.

What if a question needs facts from several documents?

“Which of last quarter’s circulars changed the KYC rules, and have our SOPs been updated for each?” needs several retrievals. Let a planner model split the question into sub-questions, retrieve for each, and answer from the union — within a hard budget of, say, three rounds, because each round adds about a second and a model call. Permissions and citations work exactly as before. Open-ended loops with tools belong to an agent platform.

What if each department or subsidiary wants its own assistant?

Most departments only need a different scope: same index, an extra filter on source or collection, and a different system prompt. Subsidiaries with legal walls — an insurance arm, a securities arm — get separate indexes, keys and possibly separate deployments, which is the silo model from multi-tenancy and isolation. You pay for duplicated infrastructure and lose cross-subsidiary answers; you gain a wall nobody can misconfigure with one filter.

What if the monthly bill triples?

Look at tokens per answer first: chunks are 77% of the input, so cut 8 chunks to 6 where the golden set allows. Route simple, single-source questions to the small model, which is 20 times cheaper per token. Keep the system prompt identical across requests so the provider’s prompt cache applies. Refuse early when retrieval is weak. Then check for abuse — a script asking thousands of questions — and set per-team budgets in the LLM gateway. See cost-aware design.

How to say it in the interview

“I’ll design an ‘ask the company’ assistant for a bank with 40,000 employees over 8 million documents in Confluence, SharePoint, Drive, Slack, Jira and scanned circulars. The hard requirements are that it never uses a document the employee couldn’t open, that changes show up within minutes, and that every answer cites its sources.

There are two paths that meet at one search index. Ingestion: connectors turn webhooks and delta polls into events on a Kafka log keyed by document; workers parse and OCR, chunk along headings into about 400 tokens with a short LLM-written context header, embed, and write chunks with their document’s allowed groups. Chunks are hashed, so an edit re-embeds only what changed, and deletes are tombstoned first.

That’s about 60 million chunks. At 1,024 dimensions float32 vectors are 246 GB, and HNSW links add only 9 GB, so I store int8 vectors — 70 GB a copy — and rescore from full vectors on SSD.

Query: resolve the user’s groups, rewrite the question, run BM25 and vector search with the permission filter inside both, fuse with reciprocal rank fusion, rerank 50 down to 8 with a cross-encoder, recheck those eight against the live ACL store, and stream an answer that must cite them. If retrieval is weak, it refuses. Retrieved text is treated as untrusted data, the assistant has no tools, and links are limited to the cited sources.

It costs about $14.50 per thousand answers, around 64 cents per employee a month, with the first token in about 1.4 seconds. A golden set of a thousand questions — including permission traps and injection probes — gates every change to the chunker, models or prompt.”

Concepts used in this design

• Embeddings and semantic search — what a vector captures, and what it misses.
• Vector databases — HNSW, IVF, quantization and filtered search.
• Elasticsearch — one engine for BM25, vectors, filters and RRF.
• How search engines work and types of search — inverted indexes, BM25 and hybrid search.
• Change data capture — treating sources as change streams.
• Message queues and Kafka — the change log and its ordering per document.
• Idempotency — replaying change events safely.
• Multi-tenancy and isolation — shared indexes with filters versus silos.
• OAuth 2.0 and OpenID Connect — the SSO token behind each question.
• Caching 101 — and why answer caches must respect permissions.
• Real-time connections: SSE and WebSockets — streaming the answer.
• Observability — traces for every answer.
• Cost-aware design — tokens per answer as the main cost lever.
• Design an LLM gateway, Design an LLM inference service and Design ChatGPT — the systems this one calls.

References and further reading

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)arxiv.orgThe original RAG paper: a generator plus a retriever over a dense vector index, motivated by provenance and by updating knowledge without retraining.
2. Efficient and robust approximate nearest neighbor search using HNSW graphs (Malkov and Yashunin)arxiv.orgHNSW layers, mL = 1/ln(M), up to 2·M links on layer 0, 4-byte links, and the 60–450 bytes of graph per element estimate.
3. pgvector README (v0.8.7)github.comHNSW and IVFFlat options and defaults, filtering after the index scan and iterative scans, halfvec and bit types, binary quantization with re-ranking, and partitioning for tenant isolation.
4. Faiss wiki: Guidelines to choose an indexgithub.comFlat versus IVF versus HNSW, the (d × 4 + M × 2 × 4) bytes-per-vector HNSW formula, IVF list counts by dataset size, and HNSW’s lack of removal.
5. Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (Cormack, Clarke and Büttcher, SIGIR 2009)plg.uwaterloo.caThe RRF formula, score = Σ 1/(k + rank), with k = 60 fixed in a pilot study and found near-optimal but not critical.
6. Elasticsearch reference: Reciprocal rank fusionelastic.coThe RRF retriever that fuses a BM25 query and a kNN query, with rank_constant defaulting to 60 and rank_window_size.
7. OpenAI API: Vector embeddings guidedevelopers.openai.comDefault 1,536 and 3,072 dimensions, the dimensions parameter for shorter vectors, and the 8,192-token input limit.
8. OpenAI API pricing (October 2026)developers.openai.comList prices used here: embeddings at $0.02 and $0.13 per million tokens, chat models at $0.10/$0.50 and $2/$10 per million input/output tokens, and a 10% uplift for in-region processing of newer models.
9. Introducing Contextual Retrieval (Anthropic, September 2024)anthropic.comPrepending 50–100 tokens of context to each chunk; top-20 retrieval failures down 35%, 49% and 67% with contextual embeddings, contextual BM25 and reranking; about $1.02 per million document tokens.
10. OWASP Top 10 for LLM Applications 2025: LLM01 Prompt Injectiongenai.owasp.orgIndirect prompt injection through files and web pages, why RAG does not remove it, and mitigations such as segregating untrusted content and least privilege.