In this article
Foundations
The foundations questions open most rounds, and the later groups build on them.
Text description of the diagram
- This page collects eighteen retrieval-augmented generation interview questions grouped by topic.
- The order follows the pipeline, from definitions to the design questions of senior rounds.
How do you explain RAG simply?
RAG gives a language model a second memory it can search. Documents are indexed ahead of time; for each question the system fetches the most relevant passages and hands them to the model, which writes its answer from them. The original paper combines the model's own "parametric" memory with a "non-parametric" one, a vector index of Wikipedia (Lewis et al., 2020).
Why it is asked: it checks that you can explain the idea in two sentences. Read the full explanation: retrieval-augmented generation, the complete guide.
What problem does RAG solve?
RAG addresses three weaknesses of a model that answers from training alone: made-up facts, knowledge that is out of date, and reasoning that cannot be traced to a source. The 2023 survey names exactly these, and says retrieval from external databases improves accuracy and lets knowledge be updated (Gao et al., 2023). It does not remove errors; it makes them easier to check.
Why it is asked: it separates motivation from buzzwords. Read the full explanation: the RAG guide.
When would you not use RAG?
Skip RAG when the whole knowledge base fits in the prompt. Anthropic says a knowledge base smaller than 200,000 tokens, about 500 pages, can simply be included in the prompt (Anthropic, September 2024). Also skip it when the task needs a change of style or skill, not new facts; that is a job for fine-tuning.
Why it is asked: it tests judgement about simpler options. Read the full explanation: the RAG guide.
What is the difference between RAG and fine-tuning?
RAG changes what the model can see at question time; fine-tuning changes the model's weights. Use RAG for facts that change or must be cited, because updating the index is cheap. Use fine-tuning for a consistent format, tone or task behaviour. They combine well: a tuned model reading retrieved evidence.
Why it is asked: it is the most common comparison question. Read the full explanation: fine-tuning versus prompt engineering.
Retrieval
Walk me through a RAG architecture.
Two pipelines share one index. Indexing, done ahead of time, splits documents into chunks, embeds each chunk and stores the vectors. Querying embeds the question, retrieves the nearest chunks, places them in the prompt and generates. Mention where each can fail: bad chunks, weak embeddings, a retriever that misses, a model that ignores the evidence.
Why it is asked: it is the standard opening for a design round. Read the full explanation: RAG architecture.
What is an embedding, and why does it work for search?
An embedding is a list of numbers a model produces for a text so that texts with similar meaning get nearby vectors. Search then becomes a nearest-neighbour lookup. It works because training pushes related texts together, not because anyone assigned meaning to each number. Say which distance you use, and that it must match the one the model was trained with.
Why it is asked: it checks first-principles understanding. Read the full explanation: what an embedding is.
Why use an approximate index, and what does it cost?
Comparing a question with every stored vector is exact but slow at scale, so production indexes skip most vectors and accept that some true neighbours are missed. The Hierarchical Navigable Small World (HNSW) graph method reports logarithmic scaling of search cost (Malkov and Yashunin, 2016). The price is recall, and you tune the speed-recall trade with the index settings.
Why it is asked: it tests whether you know retrieval is a trade, not a lookup. Read the full explanation: similarity search.
Why add keyword search to vector search?
Vector search matches meaning but can miss exact strings such as error codes, product names or rare terms; keyword search does the opposite. Hybrid search runs both and merges the lists, usually by rank because the scores are on different scales. The dense retrieval paper itself reports a dense retriever beating a keyword baseline by 9 to 19 points in top-20 accuracy (Karpukhin et al., 2020), yet neither wins everywhere.
Why it is asked: it tests retrieval design beyond one method. Read the full explanation: hybrid search.
How do you choose chunk size?
Chunk size trades precision against context. Small chunks match a question tightly but can lose the surrounding facts; large chunks keep context but dilute the match and use more of the prompt. There is no universal best size, so test a few on your own questions and measure whether the right passage is retrieved. Split on natural boundaries such as headings, and consider a little overlap.
Why it is asked: it reveals whether you evaluate or guess. Read the full explanation: RAG architecture, where a real knowledge base is counted into chunks.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Quality
What does a reranker do, and when is it worth it?
A reranker rescores the shortlist the retriever returned by reading the question and each passage together, then reorders it. It pays off when the right passage is usually in the shortlist but not near the top. It costs one model run per candidate, so keep the shortlist small. Anthropic's test saw the top-20 failure rate fall from 2.9% to 1.9% when reranking was added (Anthropic, 2024).
Why it is asked: it tests latency-versus-quality judgement. Read the full explanation: cross-encoder rerankers.
What is contextual retrieval?
Contextual retrieval prepends a short description of where a passage sits in its document to the passage before it is embedded, so a fragment keeps the subject and period it refers to. Anthropic reports the top-20 retrieval failure rate fell from 5.7% to 3.7% with contextual embeddings, and to 2.9% with a contextual keyword index as well (Anthropic, September 2024).
Why it is asked: it checks awareness of current practice. Read the full explanation: the RAG guide.
Why does the model ignore good passages in a long prompt?
Models do not use every part of a long input equally. The Lost in the Middle study found performance "often highest when relevant information occurs at the beginning or end of the input context", and lower when it sits in the middle (Liu et al., 2023). So send fewer, better passages and consider putting the strongest at the edges of the context.
Why it is asked: it separates retrieval problems from generation problems. Read the full explanation: context rot.
How do you reduce hallucination in a RAG system?
First make retrieval reliable, since a missing passage invites invention. Then instruct the model to answer only from the supplied passages and to say when they do not contain the answer, and require citations to specific passages. Finally measure faithfulness, the share of claims the passages support, so you can see regressions. RAG lowers the risk; it does not eliminate it.
Why it is asked: it is the question every product team has. Read the full explanation: Ragas and faithfulness.
Evaluation
How do you evaluate a RAG system?
Evaluate retrieval and generation separately. For retrieval, use a labelled set of questions and check whether the right passage is returned and how high it ranks (recall at k, mean reciprocal rank). For generation, check that each claim is supported by the passages and that the answer addresses the question. The Ragas paper adds reference-free scores for three dimensions: context relevance, faithfulness and answer relevance (Es et al., 2023).
Why it is asked: it is the most reliable senior-level filter. Read the full explanation: RAG evaluation, the Ragas page and the large language model (LLM) evaluation hub.
Can a language model judge be trusted to score answers?
A model judge is a useful screen but a biased one, so calibrate it. Compare its scores with human labels on a sample, fix the prompt until they agree, and keep a human-reviewed set for release decisions. Ragas relies on model-computed metrics and says they avoid the need for human reference answers, which is a convenience, not proof of accuracy.
Why it is asked: it tests scepticism about automated metrics. Read the full explanation: the Ragas page.
Design
When would you use agentic RAG or GraphRAG?
Use agentic RAG when questions need several dependent lookups and a single retrieval pass cannot gather the evidence; accept extra model calls on every question. Use GraphRAG when questions concern the whole collection, such as themes across thousands of documents, which no single passage answers. Start with plain RAG and add either only after a measured failure.
Why it is asked: it tests restraint with complexity. Read the full explanation: agentic RAG and GraphRAG.
How would you design RAG over private company documents?
Index with permissions attached to each chunk, filter the retrieval by the asking user's access before anything reaches the model, keep an update path for changed documents, and log which passages produced each answer. Then measure retrieval quality per document type. The access check must happen at retrieval time, because the model cannot be trusted to withhold what it has been shown.
Why it is asked: it is a standard enterprise design prompt. Read the full explanation: the GenAI system design hub.
How do you answer a RAG question when you have not built one?
Say so in one sentence, then show the structure: the two pipelines, one failure at each stage, and how you would measure it. Offer the smallest build you would try first, such as fifty documents, one embedding model and a labelled set of twenty questions. Interviewers reward a clear plan and honest limits over bluffing. More practice questions are in the AI engineer question set.
Why it is asked: it shows how you reason without experience. Read the full explanation: agentic AI interview questions for the neighbouring topic.
- Explain RAG as a searchable second memory, in two sentences.
- Walk the pipeline in order and name one failure at each stage.
- Under about 500 pages of material, putting everything in the prompt can beat retrieval.
- Evaluate retrieval and generation separately, and calibrate any model judge.
- Add complexity (reranking, agents, graphs) only after a measured failure.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401 v4) — arXiv (Lewis et al.), v4 revised 2021-04-12, 22 May 2020
- Dense Passage Retrieval for Open-Domain Question Answering (arXiv 2004.04906 v3) — arXiv (Karpukhin et al.), v3 revised 2020-09-30, 10 Apr 2020
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (arXiv 1603.09320 v4) — arXiv (Malkov and Yashunin), v4 revised 2018-08-14, 30 Mar 2016
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172 v3) — arXiv (Liu et al.), v3 revised 2023-11-20, 6 Jul 2023
- Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv 2312.10997 v5) — arXiv (Gao et al.), v5 revised 2024-03-27, 18 Dec 2023
- Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217 v2) — arXiv (Es et al.), v2 revised 2025-04-28, 26 Sep 2023
- Introducing Contextual Retrieval — Anthropic, 19 Sep 2024
