RAG

How would you design a retrieval-augmented chatbot for enterprise search?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 13 min read

How would you design a retrieval-augmented chatbot for enterprise search? Build a system that indexes company documents as permissioned chunks, retrieves only passages the asking user may see, and has a language model answer from them with citations. The step candidates most often skip is the permission filter, and where it runs decides whether the chatbot leaks.

In this article

Before you start
Why interviewers ask this
A retrieval-augmented chatbot from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

Why interviewers ask this

A chatbot that answers from company documents fails in ways a demo hides: it returns a page the user is not allowed to read, misses the one chunk that holds the answer, or states something the documents never said. Interviewers use this question to see whether a candidate designs for those failures or only draws the happy path of embed, search and generate.

A weak answer lists "vector database plus LLM." A strong answer walks the pipeline stage by stage and says what each stage does to the evidence. A Staff-level answer adds three things: permissions enforced inside retrieval, a way to measure retrieval separately from generation, and the cost of each stage. Related questions are collected in RAG interview questions and answers.

A retrieval-augmented chatbot from first principles

Permission filter: A permission filter is a check that limits search results to documents the asking user is allowed to read, applied inside the search itself, before any passage reaches the language model, so the answer never draws on text the user should not see.

In plain words

Picture a research librarian who only fetches from shelves your key opens. You ask a question. The librarian pulls a few relevant pages. A writer drafts an answer from only those pages and points to each one. The analogy breaks in one place. A real librarian knows when a page is irrelevant. A search system returns the nearest pages even when none is good. So the design needs a way to say "not found."

In one precise sentence, a retrieval-augmented chatbot is a retriever over a document index, filtered by the user's permissions, feeding a few ranked passages into a language model that answers from them. Lewis et al. (2020) named and popularized the RAG pattern: a language model's built-in knowledge combined with a searchable external index. That is why knowledge can be updated by changing the index instead of retraining.

Step 0: Clarify requirements and rewrite each follow-up

A chatbot design starts by clarifying requirements and by turning each chat turn into a standalone search query. Ask first about corpus size, the latency target, how fresh answers must be, and which document types exist, because each answer changes the design. A chatbot also differs from one-shot search: a follow-up such as "what about last year?" means nothing to a search index alone.

Each follow-up is rewritten into a standalone query from the chat history before retrieval, usually by a language model call, so the query reads "what was revenue last year?" (illustrative). Put only the last few turns, or a short summary of older ones, in the prompt, because history competes with evidence for space. The permission filter still uses the identity of the user asking, never anything inferred from the history.

Step 1: Ingestion turns documents into permissioned chunks and indexes them

Ingestion is the batch stage that reads documents from their source systems, cuts them into chunks, and stores each chunk with its text, a vector and its permissions. A splitter cuts text into chunks of a few hundred tokens; see chunking strategies for RAG for the options. Each chunk keeps metadata: its source, its last-modified time and its access control list (ACL), the list of users and groups who may read the document.

Two indexes are built over the same chunks: a keyword index scored with Best Match 25 (BM25), a formula that ranks chunks by how often and how rarely the query's words appear in them, and a vector index of embeddings. Anthropic (2024) also prepends a short generated context line to each chunk before indexing.

Sizing shows why the vector index is approximate. An illustrative corpus of 1,000,000 documents at 10 chunks each gives 10,000,000 chunks; with 1,024-number vectors in 32-bit floats that is 40.96 GB raw, or 10.24 GB with 8-bit numbers.¹ Brute-force comparison costs 10,240,000,000 multiply-adds per query, so a nearest-neighbor index is used.¹

Step 2: Retrieval applies the permission filter before the model sees any text

Retrieval returns candidate chunks for a question, and it must apply the user's permissions inside the search query. The common error is post-filtering: take the top results, then drop the ones the user cannot read. It wastes the candidate list, and any step between search and filter may already have handled forbidden text.

Take an illustrative user who can read 10% of the corpus. Of 50 nearest chunks, about 5 are visible; to end with 50 visible chunks you would need to fetch about 500 first.¹ Pushing the filter into the index query, using the chunk's ACL metadata as a condition, returns the full list from readable chunks only. A very selective filter can lower recall in an approximate index, so test that case. Use the identity of the person asking, not a shared service account.

Then run hybrid search. Vector search finds paraphrases but can miss exact codes; BM25 finds those exactly. Microsoft documents that reciprocal rank fusion (RRF) scores each document 1/(rank + k) per list and sums them, and that k set to a small value such as 60 works well in experiments. In the illustrative example below, a BM25 list ranks documents A, B, C, D, E and a vector list ranks C, A, F, B, G.¹

A document found by both searches outranks documents found by only one, even when no search ranks it first in both.
Document	BM25 rank	Vector rank	Fused score (k = 60)
A	1	2	0.03252
C	3	1	0.03227
B	2	4	0.03175
F	not listed	3	0.01587
D	4	not listed	0.01563

Document A scores 1/61 + 1/62 = 0.03252, and documents C and B also appear in both lists, so all three beat any single-list document.¹

Step 3: Reranking and prompt assembly keep a short, well-ordered context

A reranker is a model that reads the question and one candidate chunk together and outputs a relevance score, more accurate than comparing precomputed vectors but too slow for the whole corpus. Run it on the fused short list of 50 candidates (illustrative) and keep the best 8. At 400 tokens per chunk (illustrative), the reranker reads 20,000 tokens of text per question and the prompt receives 3,200 tokens of evidence.¹

Keep the list short on purpose. Liu et al. (2023) found that models use information at the start or end of a long context better than information buried in the middle, so more chunks can lower answer quality. Put the strongest chunks first.

Anthropic measured these stages across several test datasets: the share of questions whose answer chunk missed the top 20 was 5.7% at baseline, 3.7% with contextual embeddings, 2.9% with contextual BM25 added, and 1.9% with reranking on top, a 67% reduction.² Those figures come from one vendor's test datasets, so measure your own corpus. For stage costs see where RAG latency and cost go.

In Anthropic's test, each added stage lowered the share of questions whose answer chunk missed the top 20.CC BY 4.0 — source: How would you design a retrieval-augmented chatbot for enterprise search?

Text description of the diagram

The top-20 retrieval failure rate was 5.7% at baseline and 3.7% with contextual embeddings.
The rate was 2.9% with contextual BM25 added and 1.9% with reranking added.

Step 4: Generation answers only from the passages and cites them

Generation is the stage where the language model writes the answer from the assembled passages. The prompt tells the model to use only the supplied passages, to cite the chunk behind each claim, and to say it found nothing when the passages do not answer the question. Citations give the reader a way to check, which is the provenance benefit Lewis et al. (2020) describe for the pattern.

Treat retrieved text as untrusted input. A document can contain instructions aimed at the model, and the OWASP Top 10 for LLM Applications, from the Open Worldwide Application Security Project (OWASP), lists prompt injection, sensitive information disclosure and vector and embedding weaknesses among its risks. Keep system instructions separate from passages, give the model no tools it does not need, and see securing a RAG system for the full threat list.

Grounding reduces unsupported answers but does not remove them: retrieval can still miss, and the model can still misread what it was given. That is why the last step is measurement.

Step 5: Evaluation and refresh keep the system honest

Evaluation measures the retrieval stage and the generation stage separately, so a bad answer can be traced to its cause. For retrieval, build a set of real questions, each labeled with the chunk that answers it, and track whether that chunk appears in the top results. For generation, check that every claim is supported by a cited passage. How do you evaluate a RAG system? covers the metrics.

Refresh is the other half. When a document changes, re-chunk and re-embed only that document; when it is deleted or its permissions change, update or remove its chunks promptly, because a stale ACL in the index is a leak. Log questions that return no readable passages: they show missing documents or missing access.

Each stage narrows or checks the evidence, and the permission filter sits inside retrieval, not after it.CC BY 4.0 — source: How would you design a retrieval-augmented chatbot for enterprise search?

Step-by-step description

Requirements are set; follow-ups become standalone queries.
Documents become permissioned chunks in two indexes.
Search applies the user's permission filter.
A reranker keeps the best few chunks.
The model answers and cites the chunks.
Retrieval and answers are measured; refresh keeps the index current.

Each follow-up is rewritten into a standalone query from the chat history before retrieval.
Ingestion stores each chunk with its text, vector and permissions.
Retrieval applies the user's permission filter inside the search and fuses keyword and vector results.
A reranker keeps a few chunks, the model answers with citations, and evaluation measures retrieval and answers separately.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.