In this article
Before you start
This page sits in the generative AI system design interview guide. You can read it from zero.
Why interviewers ask this
The consequence comes first: a design round about a company's documents, policies or support articles needs a plan for getting those facts to the model, and a strong answer starts from RAG. Interviewers use the question to separate a candidate who recites a definition from one who knows what breaks.
A weak answer says RAG makes the model smarter. A strong answer names the steps (index once, then retrieve, augment, generate) and the benefit: the model uses data it was never trained on. A Staff-level answer adds that answer quality is capped by retrieval, and says how to measure retrieval separately from generation. The walkthrough of designing Perplexity shows the same pattern at the scale of a search product.
RAG from first principles
- Ingestion
- Ingestion is the preparation stage of a retrieval pipeline that splits documents into chunks, turns each chunk into an embedding and stores the embeddings in a vector database, so later questions can be searched quickly.
In plain words
Think of an open-book exam. The student does not memorize the textbook. When a question arrives, they find the right pages and answer from them. RAG works the same way: the model is the student, your documents are the book, and a search step finds the pages.
The analogy breaks in one place. A student picks pages with understanding. The retriever picks them by similarity, so it can pick the wrong ones. The model also keeps nothing: the next question starts with a blank desk.
In one precise sentence, RAG adds a retrieval step before generation so the model reads evidence instead of relying only on what its training stored. Lewis et al. (2020) introduced the name. Their models pair a pre-trained generator (the parametric memory) with a dense vector index of Wikipedia (the non-parametric memory), reached through a neural retriever.
Step 1: Ingestion cuts documents into chunks, embeds them and stores them
Ingestion prepares the documents once. Amazon Web Services (AWS) guidance describes the process as four steps, where the first runs once and the other three run as often as needed.
First, split each document into chunks. Second, turn each chunk into an embedding, a numeric representation of its meaning. Third, store the embeddings in a vector database. A vector database is a database of embeddings, sometimes called a vector store or vector index, that can return the entries closest to a query.
Take an illustrative 12,000-word handbook cut into chunks of 200 words with a 40-word overlap, so each chunk starts 160 words after the previous one. That gives 75 chunks.1 The overlap keeps a sentence cut at one boundary whole in the next chunk.
Input: documents. Output: an index of vectors, each tied to its text and source. You now have a searchable library.
Step 2: Retrieval finds the passages closest to the question
At question time the system embeds the question with the same model and asks the index for the top-k chunks. This is semantic search: it matches meaning, not only shared words. AWS describes retrieval as typically powered by vector embeddings, over a curated knowledge source.
Here is why the choice matters. In an illustrative toy of eight policy chunks and six questions, plain word overlap finds the right chunk for five questions. The sixth, "Can I do my job from my house?", shares no word with the remote-work chunk, so recall at 3 is 0.833.1 An embedding search is built to catch that paraphrase. The toy does not simulate one, so it shows only the limit of word matching.
If the right chunk is not in the top k, the model never sees the fact. That is the answer to the lead's follow-up: retrieval sets the ceiling on answer quality.
Step 3: Retrieved passages are added to the prompt
The system builds a prompt from three parts: an instruction, the retrieved passages and the question. The instruction says to answer only from the passages and to say so when they do not contain the answer.
Retrieval keeps this prompt small. Assume an illustrative 1.3 tokens per word. Four 200-word chunks add 1,040 tokens. With a 25-token question and a 60-token instruction, the prompt holds 1,125 tokens against 15,685 for the whole handbook, about 7.2% of it.1 A smaller prompt costs less per question and gives the model less to wade through.
Text description of the diagram
- A top-4 prompt holds 1,125 tokens against 15,685 for the whole handbook.
- The question alone is 25 tokens.
Input: question and top-k chunks. Output: one grounded prompt.
Step 4: The model answers from the passages
The LLM reads the prompt and writes the answer from the supplied text. Because each passage has a source id, the answer can name where each fact came from. That is provenance, one of the open problems Lewis et al. (2020) set out to address, along with updating a model's world knowledge.
No weights change. The data stays outside the model, which references an authoritative source before it generates (AWS, introduction). To update what the system knows, re-embed the changed documents. Indexing runs once, when the documents are prepared. Retrieve, augment and generate run for every question, so each answer is built fresh from the passages the retriever returns at that moment.
Step-by-step description
- Documents are chunked, embedded and stored once.
- The question finds the top-k chunks.
- The chunks join the prompt.
- The model answers from the passages.
- Indexing runs once.
- Retrieve, augment and generate run for every question.
What is RAG useful for?
RAG is useful for four reasons. It gives the model data absent from training: AWS notes that models trained on public text often lack awareness of proprietary data, business rules or recent changes. It reduces hallucinations, meaning confident but unsupported answers, and helps enforce internal policies, per the AWS grounding guidance. It makes knowledge updates cheap, because you edit the index and not the model. And it allows citations. In the original paper, RAG models produced more specific, diverse and factual text than a parametric-only baseline. The word is "reduces", not "eliminates": a wrong chunk still misleads the model, so retrieval quality still matters.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What is RAG, and why is it useful?
RAG, retrieval-augmented generation, lets a language model answer from documents it was never trained on. Once, I split the documents into chunks, embed them and store them in a vector index. For each question, I embed the question, retrieve the top few chunks, put them in the prompt with an instruction to answer only from that context, and the model writes the answer, ideally citing chunk ids. It is useful because private and recent data stay outside the weights, so I update the index instead of retraining. The failure mode is retrieval: if the right chunk is not in the top k, the model cannot answer correctly, so I measure retrieval recall separately from answer quality.
What makes it Staff-level is the last sentence: it names the failure mode and says how you would measure it.
The rubric
This rubric is a study aid for self-review, not any company's scoring.
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Definition | "It makes the model smarter." | Retrieval adds external text to the prompt before generation. | Adds that no weights change and data stays outside the model. |
| Mechanism | Names embeddings only. | Index once, then retrieve, augment, generate per question. | Explains chunking, overlap and top-k choices. |
| Usefulness | "More accurate." | Private or recent data, fewer hallucinations, citations. | Adds cheap updates by re-indexing and policy control. |
| Trade-offs | None given. | Wrong chunk gives a wrong answer. | Measures retrieval recall apart from answer quality. |
| Delivery | Long and unordered. | Definition, steps, benefit, in order. | Ends on the failure mode, ready for the follow-up. |
Three variants of this question
What is RAG in simple words?
RAG is an open-book exam for a language model: a search step finds the relevant pages from your documents, and the model answers from those pages instead of from memory.
When would you use RAG instead of fine-tuning?
Use RAG when the knowledge is private, changes often or needs citations, because the data stays outside the model and an update is a re-index. Fine-tuning changes the model's weights, which suits teaching a style or format rather than adding facts.
Do you need a vector database for RAG?
Not by definition. AWS says retrieval is typically powered by embeddings, and a vector database is the usual store for them. What the definition requires is a retrieval step, so keyword or combined search can serve as well, and you should choose by how well the retriever finds the right chunk.
Common mistakes
- "RAG retrains the model on our documents." (wrong) The pipeline updates no weights. (right) "The documents stay outside the model and are retrieved per question." The original paper did train the retriever and generator jointly; modern usage means the pipeline.
- "RAG eliminates hallucinations." (wrong) Sources say it reduces them. (right) "It reduces them, but a wrong or missing chunk still causes bad answers."
- "RAG is only for enterprise chatbots." (wrong) The paper targeted knowledge-intensive tasks such as open-domain question answering. (right) "It fits any task that needs an external knowledge source."
- "Just paste the whole handbook in the prompt." (wrong) That sends 15,685 tokens instead of 1,125 on every question in the worked example.1 (right) "Retrieve only the best chunks."
Follow-up questions
What happens when retrieval returns the wrong passage?
The model either answers from irrelevant text or says it cannot find the answer. Fixes include better chunking, combining keyword and semantic search, reranking the short list, and an instruction to refuse when the passages do not contain the answer.
How do you know a RAG system works?
Measure retrieval and generation separately. For retrieval, check recall at k on questions with known source chunks. For generation, check that each answer is supported by the retrieved text.
How is agentic RAG different from RAG?
Standard RAG retrieves once per question. Agentic RAG lets the model decide when and what to retrieve, and retrieve again if the first result falls short.
How do you handle tables and charts in RAG documents?
Plain text chunking breaks tables and loses charts, so the pipeline needs layout-aware parsing. The page on designing RAG over real documents covers it.
How would you build an answer engine on RAG?
Add web retrieval, ranking and citations to the same four steps. The Perplexity design walkthrough works through it.
Where does an LLM gateway fit in a RAG system?
The gateway sits in front of model calls and handles routing, limits and logging for the generate step. See the LLM gateway interview questions.
- RAG retrieves relevant passages from external data and adds them to the prompt before the model answers.
- Index once; retrieve, augment and generate for every question.
- It is useful for private or recent data, fewer hallucinations, citations and cheap updates.
- No weights change; update by re-indexing the changed documents.
- Retrieval sets the ceiling, so measure recall at k apart from answer quality.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401v4) — arXiv (Lewis et al.; NeurIPS 2020), 12 Apr 2021
- Understanding Retrieval Augmented Generation — Amazon Web Services Prescriptive Guidance (undated page; read 2026-10-26), 26 Oct 2026
- Introduction to RAG options on AWS — Amazon Web Services Prescriptive Guidance (undated page; read 2026-10-26), 26 Oct 2026
- Grounding and RAG — Amazon Web Services Prescriptive Guidance (undated page; read 2026-10-26), 26 Oct 2026
