Design RAG search over company documents: the 60-second story

Design RAG search

“Scale III ka LTA limit kitna hai?” A cited answer, in seconds.

The demo takes an afternoon. Production means 8M documents in six systems, and never using one you couldn’t open.

246 GB

of float32 vectors for 60M chunks

Assume 8M documents, split into 60M chunks of 1,024 dimensions. Ten questions a second is easy; this isn’t. Store them as int8 and it’s 70 GB a copy.

Bad: Every 400 tokens

Cuts land mid-table; chunks forget which policy they’re from.

Good: Along headings, plus context

Whole sections, tables intact, each with a 50–100-token header naming its policy.

Run BM25 and vector search and fuse their ranks, so chunks both find rise. A cross-encoder then reranks the top 50 to 8.

Search the top 40, then filter, and ≈ 2 survive. Filter inside the search instead; the walk skips chunks they can’t open until it has 40.

Assume 80,000 documents change a day. One typo fix in a 60-chunk manual. Re-embed all sixty?

The trick

Split deterministically and hash each chunk. Embed only new hashes, delete vanished ones. One chunk, not sixty.

Any of 40,000 employees can edit a wiki page, and RAG doesn’t remove prompt injection. The planted line arrives as just another source.

Remember

Two paths meet at one index. Chunk by structure, search hybrid then rerank, filter permissions inside the search, and cite every claim or refuse.