How do you evaluate a RAG system?

In this article

1. Before you start
2. Why interviewers ask this
3. RAG evaluation from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A candidate who scores only the final answer cannot say which half of the system broke, and that is the follow-up interviewers ask. They put the question as "how would you evaluate this RAG system", "which metrics would you track" or "users say the answers are wrong, what do you do". A weak answer names one metric or says "check the answers by hand". A strong answer separates the retriever from the generator, names a metric for each, builds an evaluation set, and says how many questions are enough. The follow-up is usually which half broke and how you know. This page builds each part from zero.

RAG evaluation from first principles

RAG evaluation

RAG evaluation is the measurement of a retrieval-augmented generation system on a fixed set of questions, with separate scores for whether the retriever found the right passages and whether the generator used them correctly.

Mean reciprocal rank

Mean reciprocal rank (MRR) is the average, over all questions, of one divided by the position of the first correct passage in the retrieved list, counting zero when none was retrieved.

In plain words

Think of an open-book exam. A student can fail in two ways. The wrong pages were on the desk, or the right pages were misread. A teacher who marks only the answer cannot tell which. So the teacher also checks the desk. RAG evaluation checks the desk (retrieval) and the writing (generation).

In one precise sentence: RAG evaluation scores retrieval against known relevant passages, scores generation by whether the answer sticks to those passages, and crosses the two to find each failure's cause. The Ragas paper (2023, revised 2025) frames the same split: whether the retriever finds relevant, focused context, whether the model uses it faithfully, and the quality of the answer.

Step 1: Build an evaluation set

An evaluation set is a fixed list of questions, each with the passage or passages that contain the answer and, where possible, a reference answer. Without it, every number is an anecdote. Sample passages from your own documents, write a question each passage answers, and record the source passage. That record is the gold passage retrieval is judged against.

The output is a table of question, gold passage identifier and reference answer. Add real user questions too, because questions written from a passage repeat its words and flatter keyword search. The general discipline is in the guide to evaluating language models. What you now know: the gold passage is what makes retrieval measurable.

Step 2: Score the retriever

Score retrieval by asking whether the gold passage is in the top k results and how high it sits. Recall at k is the share of questions whose gold passage appears in the top k. This is not the recall of an approximate vector index, which is measured against exact search. Mean reciprocal rank rewards ranking it first.

On the ten-question example the gold passage is first for 3 questions, in the top 3 for 7 and in the top 5 for 8, so recall at 1, 3 and 5 is 0.30, 0.70 and 0.80, and the mean reciprocal rank is 0.4867.¹ Two questions never retrieve their gold passage, which no later step can repair from the gold passage.

The Benchmarking Information Retrieval (BEIR) suite (2021) tests retrievers on 18 datasets. Without gold passage labels you can still score the retriever: the Ragas documentation computes context recall as reference-answer claims supported by the retrieved context, divided by all reference claims (Ragas docs, read October 2026). What you now know: retrieval quality is two numbers, how often and how high.

Step 3: Score the generator

Score generation on the passages the retriever actually supplied, not on the whole corpus. Is every claim in the answer supported by the supplied passages? That is faithfulness. Does the answer address the question? That is answer relevance.

Faithfulness is a ratio: the claims in the answer that the retrieved context supports, divided by all claims in the answer (Ragas faithfulness documentation, read October 2026). An answer with 4 claims, 3 of them supported, scores 0.75. Most generation metrics are scored by a large language model (LLM) acting as a judge. An LLM judge is a model asked to grade another model's output against a rubric. Automated RAG Evaluation System (ARES) (2023, revised 2024) trains small judges on synthetic data. What you now know: generation is scored per answer against the context it was given.

Step 4: Attribute every failure to retrieval or generation

Cross the two scores. For each question ask: did the gold passage arrive, and was the answer correct? That gives four cells. The mistake of scoring only the final answer hides this table, and the table decides what to fix.

On the example, 6 questions are found and correct, 2 are found but wrong, 2 are missed and wrong, and none is missed yet correct.¹ The two found-but-wrong questions point at the generator or at ranking: the right passage arrived but was misused or buried. The two missed-and-wrong questions point at the retriever: better chunking, hybrid search or a reranker. A missed-but-correct answer would mean either the model answered from its own memory or another retrieved passage also held the answer (the gold label is incomplete); both are signs the test is not measuring retrieval cleanly. What you now know: the split turns one accuracy number into a fix list.

Step 5: Check the judge and the sample size

Two checks keep the numbers honest. First, compare the language-model judge with a human on a sample. The LLM-as-a-judge paper (2023) reports that strong judge models reach over 80% agreement with human preferences on chat answers (its chat-answer benchmarks), the same level as between humans, and names three biases: position, verbosity and self-enhancement.

Second, count the questions. Ten questions are a demonstration, not a measurement: 6 correct of 10 has a 95% confidence interval of about 0.31 to 0.83, while 60 of 100 narrows it to about 0.50 to 0.69.¹ To report accuracy within plus or minus 5 points at 95% confidence in the worst case you need about 385 questions.¹ What you now know: a score without a sample size is not yet evidence.

1 · Eval set2 · Retriever3 · Generator4 · Split5 · TrustReplay

Step-by-step description

1. Questions with gold passages.
2. Score recall and rank.
3. Score faithfulness and relevance.
4. Cross the two scores per failure.
5. Check the judge and the sample size.

• The evaluation set records a gold passage for each question.
• The failure split tells you which half to fix.

Worked example

The log in the table below is illustrative: ten questions written for this page, each with a gold passage and the five passages the retriever returned. It is not a measurement of any real system.¹

Text description of the diagram

• Recall at 1, 3 and 5 is 0.30, 0.70 and 0.80.
• Two questions never retrieve their gold passage.

The reciprocal ranks are 1, 0.5, 0.3333, 0, 1, 0.2, 1, 0.3333, 0 and 0.5, which add to 4.8667; divided by 10 that is 0.4867.¹ Precision at 3 is lower than recall at 3 only because each question has a single gold passage, so at most one of the three results can be relevant. This is why precision at k is a weak metric when there is one right passage, and recall and reciprocal rank are used instead.

The retrieval half of the scoring fits in a few lines:

ranks = [r.index(gold) + 1 if gold in r else None
         for gold, r, ok in log]
recall_at_3 = sum(1 for x in ranks if x and x <= 3) / len(log)
mrr = sum(1 / x for x in ranks if x) / len(log)

Trade-offs and edge cases

Reference-free versus reference-based. Reference-based scoring compares an answer with a written reference; it is more trustworthy but costs human work. Reference-free scoring uses a judge and the retrieved context, so it scales, but it inherits the judge's biases. A common compromise is a small human-labelled core set plus a large automatic set, which is the idea behind the ARES design: its abstract says only a few hundred human annotations are needed during evaluation.

Metrics that look good together can hide a failure. High faithfulness with low recall means the model is accurately summarising the wrong passages. High recall with low faithfulness means the right passages arrive and are ignored. Always read the pair.

Text description of the diagram

• Retriever metrics are recall at k and mean reciprocal rank.
• Generator metrics are faithfulness and answer relevance.

Agents and multiple steps. In agentic RAG the system retrieves several times, so you also score each retrieval step and the final state, not only the last answer. Pipeline changes. Adding hybrid search or a cross-encoder reranker should move recall and mean reciprocal rank first; if it does not, the change did not help retrieval.

How to answer it in an interview

The spoken structure is: the evaluation set, one metric per half, then the failure split and the sample size.

Common mistakes

"I measure accuracy of the final answer." (wrong) One number cannot say whether retrieval or generation failed. (right) "I score the retriever and the generator separately and cross them."

"High faithfulness means the system is good." (wrong) The model can be faithful to the wrong passages. (right) "I read faithfulness together with recall."

"The judge model is the ground truth." (wrong) Judges show position, verbosity and self-enhancement bias. (right) "I compare the judge with human labels on a sample."

"Fifty hand-picked questions are enough." (wrong) Easy, self-written questions flatter the system and the interval is wide. (right) "I mix real user questions in and report the sample size."

Follow-up questions

What are the metrics for RAG evaluation?

For retrieval, recall at k and mean reciprocal rank; for generation, faithfulness and answer relevance; for end to end, answer correctness against a reference. The Ragas paper and ARES both organise metrics around context relevance, faithful use of context and answer quality.

How do you create a RAG evaluation dataset from documents?

Sample passages from your corpus, have a model or a person write a question each passage answers, and store the passage identifier as the gold source. Then add real user questions and a share of questions the corpus cannot answer, so you can test refusals.

How do you evaluate a RAG agent?

Score each retrieval step as above, then check the trajectory: which searches ran, whether the final context contained the gold passage, and whether the answer is correct. Compare cost too, since an agent may retrieve many times.

How many test questions do you need?

It depends on the precision you need. For plus or minus 5 points in the worst case, about 385; with 10 questions, the 95% interval spans about half the scale.¹ If you want to detect a small change, you need more.

Can you trust a language model as the judge?

Partly. Strong judges agree with humans over 80% of the time in the cited study, and they show position, verbosity and self-enhancement bias. Validate on a human-labelled sample. See also AI engineer interview questions.

Sources

1. Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217 v2) — arXiv (Es, James, Espinosa-Anke, Schockaert), 28 Apr 2025
2. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (arXiv 2311.09476 v2) — arXiv (Saad-Falcon, Khattab, Potts, Zaharia), 31 Mar 2024
3. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv 2104.08663 v4) — arXiv (Thakur, Reimers, Ruckle, Srivastava, Gurevych), 21 Oct 2021
4. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685 v4) — arXiv (Zheng et al.), 24 Dec 2023
5. Context recall, Ragas documentation — Ragas (page updated December 2025; read 2026-10-24), 24 Oct 2026
6. Faithfulness, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026