Retrieval-augmented generation: the complete guide

In this article

1. Start here: the learning path
2. How does RAG work?
3. Why use RAG instead of a longer prompt?
4. Embeddings and similarity search
5. Hybrid search
6. Reranking
7. Contextual retrieval
8. Evaluating a RAG system
9. Agentic RAG and GraphRAG
10. Long context and context rot
11. Questions interviewers ask on RAG
12. All 17 guides in RAG and retrieval
13. Follow-up questions
14. Sources

Retrieval-augmented generation

Retrieval-augmented generation is a method in which a language model is given passages fetched from an external knowledge base at question time and writes its answer from them, so the answer can use facts that were never in its training data and can point back to its sources.

Retriever

A retriever is the component of a retrieval-augmented system that takes a question and returns the passages most likely to contain the answer, usually by comparing vectors or keywords against an index built in advance.

Knowledge base

A knowledge base is the collection of documents a retrieval-augmented system searches, such as manuals, tickets or policies, split into passages and indexed before any question arrives.

Start here: the learning path

This page is the overview of how RAG works. Next, learn the RAG architecture: the two pipelines that the topics below change. Then take the topics in the order this page covers them.

1. What are the different chunking strategies for RAG?
2. How do you evaluate a RAG system?
3. How do you debug a RAG system that gives wrong answers?
4. How do you secure a RAG system that handles sensitive data?
5. What are grounding and citations, and how do you verify them?
6. What is GraphRAG, and how does it work?
7. Where RAG latency and cost go, stage by stage
8. RAG in a regulated bank: how to design it
9. How would you design a retrieval-augmented chatbot for enterprise search?
10. Transformer neural network: how attention works and where it is used
11. What are ranking and retrieval metrics?
12. What are some techniques for evaluating RAG systems?
13. What is agentic RAG? How it differs from standard RAG
14. What is GraphRAG and when do you use it over vector RAG?
15. Why do LLMs hallucinate, and how do you reduce it?

How does RAG work?

RAG works in two stages that share one index: an indexing stage that runs ahead of time, and a query stage that runs for every question. The original paper describes the design as combining "pre-trained parametric and non-parametric memory for language generation", where the non-parametric memory is "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever" (Lewis et al., 2020). In plain words, the model's weights are one memory, and a searchable store of documents is a second one.

In the indexing stage, documents are cut into passages called chunks, each chunk is turned into a vector (a list of numbers) by an embedding model, and the vectors are stored in an index. In the query stage, the question is embedded with the same (or a paired query) embedding model, the index returns the nearest passages, those passages are placed in the prompt next to the question, and the model writes the answer. Both pipelines are laid out in the RAG architecture.

Text description of the diagram

• Retrieval-augmented generation lets a language model answer from passages retrieved from a knowledge base.
• Architecture, embeddings, hybrid search and reranking shape retrieval; evaluation measures the system; agents and graphs add control at extra cost.

Why use RAG instead of a longer prompt?

Teams use RAG when the collection is too large to send with every question, or when sending all of it on every call costs too much. A model that answers only from training has three documented weaknesses, and retrieval addresses all three: the 2023 survey of the field says language models "encounter challenges like hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes", and that RAG answers them by drawing on external databases (Gao et al., 2023, revised March 2024).

RAG is not always needed. Anthropic's guidance on retrieval says that if a knowledge base is smaller than 200,000 tokens, about 500 pages of material, the whole of it can be placed in the prompt with no retrieval at all (Anthropic, September 2024). Above that size, retrieval is the practical route. A long prompt also has its own failure: models use information at the start and end of a long input better than information in the middle (Liu et al., 2023).

Embeddings and similarity search

An embedding is a list of numbers that a model produces for a piece of text so that texts with similar meaning get vectors that sit close together, and similarity search is the lookup that finds the stored vectors nearest to the question's vector. These two ideas are the engine of the retriever, and many interview questions about RAG are really questions about them: which model, how many dimensions, which distance, exact or approximate.

Study the two ideas in this order. Start with the embedding: a vector whose meaning no one chose by hand. Then comes similarity search, where an approximate index trades some recall for speed. Vector databases and recall at k extend the same ideas in embeddings and vector search.

Hybrid search

Hybrid search runs a keyword search and a vector search on the same question and merges the two ranked lists, because each one finds what the other misses. Keyword search catches exact strings such as error codes and product names. Vector search catches paraphrases that share no words with the question.

The interview follow-up is how to merge two lists whose scores are not comparable, and the standard answer is reciprocal rank fusion, which scores each passage by 1/(k + rank) in each list, where k is a constant that softens the lead of the top ranks, and adds the scores, so only positions matter and raw scores never need to be compared. Use hybrid search when your questions mix exact identifiers with natural language, which is common in support and code search.

Reranking

A reranker is a model that rescores the shortlist the retriever returned, reading the question and each passage together to put the best ones first. It exists because the fast first stage is tuned to get the right passage somewhere in the shortlist (recall), not to put it first (precision), so the right passage is often in the shortlist but not at the top.

A reranker is slow because it must run once per question and passage pair, so it scores only a shortlist, usually tens to a few hundred candidates (Anthropic's test reranked the top 150 and kept 20), never the whole collection. A cross-encoder reranker cannot precompute anything, because its score depends on the question. In Anthropic's test, adding reranking on top of contextual retrieval (applied to both the embeddings and the keyword index) cut the top-20 retrieval failure rate, the share of relevant passages missing from the top 20 results (1 minus recall at 20), from 2.9% to 1.9% (Anthropic, 2024); the chart in the next section shows every stage.

Contextual retrieval

Contextual retrieval is a technique that prepends a short, model-written description of where a passage sits in its document to the passage before it is embedded and before it is added to the keyword index, so a passage such as Anthropic's example, "The company's revenue grew by 3% over the previous quarter", is no longer stripped of the company and period it refers to. Anthropic reports that contextual embeddings reduced the top-20 retrieval failure rate by 35%, from 5.7% to 3.7%; combined with a contextual keyword index the reduction was 49%, to 2.9%; and with reranking added it was 67%, to 1.9% (Anthropic, 19 September 2024).

These are one vendor's results on its own datasets, so treat the sizes as indicative and measure on your documents. The reasoning is that each stage targets a different kind of miss.

Text description of the diagram

• Contextual embeddings reduced the top-20 retrieval failure rate from 5.7% to 3.7%.
• With a contextual keyword index it fell to 2.9%, and with reranking added to 1.9%.

Evaluating a RAG system

A RAG system is evaluated in two halves, because it can fail in either: the retriever can miss the evidence, or the generator can ignore or distort it. A good score on the final answer alone hides which half is broken. RAG evaluation therefore measures retrieval (did the right passage come back, and how high) and generation (is every claim supported by the retrieved text).

Ragas scores the two halves with metrics such as faithfulness and context recall, using a language model as the judge. Broader methods, from benchmarks to human review, are covered under LLM evaluation.

Agentic RAG and GraphRAG

Agentic RAG is a form of RAG in which the model controls retrieval, deciding whether to search, what to search for, and when it has enough, instead of running one fixed retrieve-then-generate pass. It helps on questions that need several lookups, and it costs extra model calls on every question. Agentic RAG is worth it only when the extra calls fit the latency and cost budget.

GraphRAG is a different extension: it builds a knowledge graph from the documents and answers from communities in that graph and their summaries, which suits questions about the whole collection that no single passage can answer (Edge et al., 2024). Neither is the starting point. Build the plain pipeline first and add these only when a measured failure demands them.

Long context and context rot

A larger context window does not make retrieval unnecessary, because models use the middle of a long input poorly. The Lost in the Middle study found that performance is "often highest when relevant information occurs at the beginning or end of the input context" and drops when the relevant text is in the middle of a long input (Liu et al., 2023).

This is why retrieval keeps the prompt short and relevant even when the window is huge. A related effect, context rot, is the wider loss of accuracy on long inputs. When you design a full product around retrieval, such as an enterprise question-answering service, GenAI system design adds access control, scale and cost.

Questions interviewers ask on RAG

These are ten themes that recur in RAG rounds, in our own words, with what each one tests. We make no claim about how often each is asked. Worked answers to the common ones are in RAG interview questions and answers.

Follow-up questions

How does RAG work in AI?

RAG indexes documents ahead of time as vectors, then for each question retrieves the nearest passages and gives them to a language model with the question, which writes the answer from them. The model supplies the language, and the index supplies the facts. Both pipelines are laid out in the RAG architecture.

How does RAG work with LLMs?

The LLM receives a prompt that contains the retrieved passages and an instruction to answer from them, so it needs no retraining to use new documents. Updating the knowledge means updating the index, not the model weights. The prompt is assembled in the query pipeline of the RAG architecture.

How do you do reranking in RAG?

Retrieve a generous shortlist with a fast method, then score each shortlisted passage together with the question using a reranker and keep the top few. A cross-encoder reranker costs one model run per passage and question pair.

What is RAG architecture?

RAG architecture is the layout of the indexing pipeline and the query pipeline around one model: chunk, embed and store on one side, embed, retrieve and generate on the other. Both sides are laid out in the RAG architecture.

Who invented RAG?

Patrick Lewis and colleagues coined the term retrieval-augmented generation in a 2020 paper on knowledge-intensive natural language processing tasks, first posted on 22 May 2020 (arXiv 2005.11401). Retrieval-augmented language models existed before it, so say the paper named and popularised this combination rather than invented retrieval for language models.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401 v4) — arXiv (Lewis et al.), v4 revised 2021-04-12, 22 May 2020
2. Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv 2312.10997 v5) — arXiv (Gao et al.), v5 revised 2024-03-27, 18 Dec 2023
3. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172 v3) — arXiv (Liu et al.), v3 revised 2023-11-20, 6 Jul 2023
4. From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv 2404.16130) — arXiv (Edge et al.), v2 revised 2025-02-19, 24 Apr 2024
5. Introducing Contextual Retrieval — Anthropic, 19 Sep 2024