What are some techniques for evaluating RAG systems?

In this article

1. Before you start
2. Why interviewers ask this
3. RAG evaluation from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A RAG demo can look right on three hand-picked questions, so interviewers ask about evaluation to see whether you can tell a system that works from one that sounds fluent. The consequence of skipping it is a system that fails quietly: users get confident, wrong answers and nobody knows whether the search or the model is at fault.

A weak answer names one tool and one score ("I run Ragas and look at the number"). A strong answer separates retrieval from generation and names a metric for each. A Staff-level answer adds how the labels were built, how the automated judge was checked against humans, and what the numbers cannot show. The broader question, with a different angle, has its own page: how to evaluate a RAG system. More questions of this kind are in the RAG interview questions.

RAG evaluation from first principles

LLM judge

An LLM judge is a large language model (LLM) prompted to grade another model's output, for example whether each claim in an answer is supported by a passage, which replaces slow human grading with an automated score.

In plain words

Think of an open-book exam. Two things can go wrong: the student opens the wrong page, or the student opens the right page and misreads it. Grading only the final answer tells you the student failed, not which mistake happened. The analogy breaks in one place: a RAG system can also open the wrong page and then write something fluent that sounds sourced.

In precise terms, RAG evaluation does three things. It scores the retriever on labeled relevant chunks. It scores the generator on grounding and correctness. Then it scores the whole pipeline on the same questions. The original RAG paper (Lewis et al., 2020) combines a retrieval step with a generator, which is why the two parts are measured apart.

1 · Split stages2 · Retrieval3 · Answers4 · JudgesReplay

Step-by-step description

1. One labeled question set, two stages.
2. Score the chunks that came back.
3. Score faithfulness and correctness.
4. Check the judge against humans.

• Retrieval and generation are scored separately on one labeled question set.
• Retrieval is scored on labeled chunks and answers are scored for faithfulness and correctness.
• Judges are calibrated against a human-labeled sample.

How do you build the labeled question set?

A labeled question set is a list of test questions, each with the chunks that answer it marked by a person. Build it from real user queries, for example sampled from logs with sensitive data removed, because invented questions may not match what users really ask. Ask people to mark which chunks are relevant for each question, and write a gold answer where the cost allows.

Add synthetic questions generated from your documents to cover rare topics. The automated evaluation framework (ARES) of Saad-Falcon et al. shows the idea, with lightweight judges built on a language model (LM): "by creating its own synthetic training data, ARES finetunes lightweight LM judges" (Saad-Falcon et al., 2023). Keep a human-checked share of the labels, because every later score depends on them.

Input: documents and user queries. Output: the one labeled set that every step below reuses.

Step 1: Score retrieval and generation separately

Retrieval and generation are scored separately because one low end-to-end score cannot say which stage failed. The Ragas paper (Es et al., 2023) makes the same split: whether retrieval finds focused context, and whether the model uses that context faithfully.

Cross the two results per question and the failure types appear. In an illustrative set of 50 questions, 30 are retrieved and faithful, 8 retrieved but unfaithful, 7 missed but faithful and 5 missed and unfaithful.¹ The retrieval hit rate (a hit is at least one relevant chunk in the top k) is 0.76 and the faithfulness rate 0.74, yet the 7 missed-but-faithful answers (14% of the set) are grounded in the wrong text. Faithful does not mean correct, and a faithfulness score alone would call them fine.

Input: a labeled question set. Output: a per-stage result for each question. You now know why two scores exist; the next step defines the first.

Step 2: Retrieval metrics on labeled chunks

Retrieval metrics compare the returned chunks with chunks a person marked relevant for each question. Precision at k is the share of the top k results that are relevant. Recall at k is defined above. Mean reciprocal rank (MRR) averages one divided by the rank of the first relevant result. Normalized discounted cumulative gain (nDCG) rewards relevant results more when they rank higher: each relevant result at rank r adds 1/log2(r + 1). nDCG divides that sum by the sum for the ideal ordering. Precision, recall and nDCG come from classic information retrieval (Manning, Raghavan and Schütze, 2008, ch. 8).

Take five toy questions with k = 3 (illustrative labels). The means are precision at 3 of 0.4, recall at 3 of 0.7333, MRR of 0.6067 and nDCG at 3 of 0.5632.¹ For question 1 the top 3 hold relevant chunks at ranks 1 and 3: the sum is 1 + 0.5 = 1.5. The ideal is 1 + 1/log2(3) = 1.6309, so nDCG is 1.5 / 1.6309 = 0.9197.¹ Question 3 scores zero because its first relevant chunk sits at rank 5, outside the top 3.¹

Text description of the diagram

• Recall at 3 is 1 for question 2 while nDCG at 3 is 0.6309.
• Question 3 scores zero on both metrics.

Question 2 shows the gap: recall at 3 is 1.0 and nDCG at 3 is 0.6309, because the one relevant chunk is second.¹ Chunk size and overlap move these scores, so rerun them after any change to chunking.

Step 3: Generation metrics: faithfulness, relevance, correctness

Generation is scored on three questions: is the answer grounded, does it address the question, and is it right. Faithfulness splits the answer into claims and checks each against the retrieved context; an answer with 4 claims of which 3 are supported scores 3 / 4 = 0.75.¹ Answer relevance asks whether the answer addresses the question, with no reference text. Correctness compares the answer with a reference answer.

Context relevance scores the retrieved passages themselves. Ragas frames it as finding "relevant and focused context passages", and ARES scores "context relevance, answer faithfulness, and answer relevance". Without gold answers, use the reference-free route of Ragas; with them, add correctness, which catches the faithful-but-wrong case of Step 1.

Retrieving more is not an automatic fix. Liu et al. (2023) report that models use information less well when it sits in the middle of a long context, so a higher k can add noise and cost. The cost side is covered in where RAG latency and cost go.

Step 4: Calibrate the LLM judges against humans

Automated judges are trusted only after a human-labeled sample checks them. Zheng et al. (2023) report that strong LLM judges reach over 80% agreement with human preferences, and also document position, verbosity and self-enhancement biases. The automated evaluation framework of Saad-Falcon et al. (2023) trains lightweight judges and uses a small human-labeled set with prediction-powered inference to correct the judges' errors.

You can do a simple version yourself. If a judge agrees with humans on 85 of 100 labeled answers (illustrative), the 95% Wilson score interval, a standard interval for a share, runs from 0.7672 to 0.9069; with the same rate on 400 labeled answers it narrows to 0.8117 to 0.8817.¹ A wide interval means the judge needs more labels before you trust a small score change.

The Staff-level answer

What makes it Staff-level: it names the labeled set, checks the judge against humans, and states the one failure the metrics hide.

The rubric

Three variants of this question

Which retrieval metric would you pick for a RAG system?

Pick recall at k when the generator reads all k chunks, since presence matters most, although position in the context still affects how well the model uses it. Pick nDCG or MRR when order matters, for example when you truncate the context or show sources in rank order.

How do you evaluate RAG without gold answers?

Use reference-free measures: claim-level faithfulness and answer relevance with an LLM judge, as in Ragas, plus a small human-labeled sample to check the judge. Retrieval metrics still need relevance labels for the chunks.

What if recall at k is high but users still get wrong answers?

Check faithfulness and correctness next. High recall means the evidence was retrieved, so the fault is in how it was used: too much noise at a large k, evidence buried mid-context, or a missing reference check. Also confirm the labels themselves are right.

Common mistakes

• "I evaluate the final answer only." (wrong) That hides the failing stage. (right) "I score retrieval and generation separately, then end to end."
• "The faithfulness score is high, so the answers are correct." (wrong) Faithfulness checks support from the retrieved context, not truth. (right) "I cross faithfulness with retrieval hits and add correctness where I have references."
• "The LLM judge is objective." (wrong) Judges show position, verbosity and self-enhancement bias. (right) "I calibrate the judge on human labels and report an interval."
• "Higher recall at k is always better." (wrong) A larger k adds noise and cost. (right) "I tune k against answer quality and cost, not recall alone."

Follow-up questions

What is the difference between faithfulness and answer relevance?

Faithfulness asks whether the answer's claims are supported by the retrieved context. Answer relevance asks whether the answer addresses the question; a faithful answer can still be off-topic.

How many labeled questions do you need?

Enough that the interval around your score is narrower than the change you want to detect. In the worked example, going from 100 to 400 labeled answers narrows the judge-agreement interval from about 0.14 wide to about 0.07 wide.¹

What changes after you change the chunk size?

Chunk boundaries move, so relevance labels attached to old chunks may no longer match. Re-map the labels, rerun recall at k and nDCG, then recheck faithfulness, since each chunk now carries different context.

How do you evaluate a RAG system in production?

Keep the pre-deployment test set as a regression check and rerun it after every change to chunking, retrieval or the model. On live traffic, collect user feedback, run A/B comparisons of variants, and sample answers for judge and human review. Watch for drift, since new documents and new kinds of questions slowly make the test set unrepresentative.

Does evaluation cover security and cost?

Not by itself. Accuracy metrics do not test leakage of sensitive data; see securing a RAG system. Track latency and cost alongside quality.

What if the system uses a graph instead of plain chunks?

The same two-stage split applies, with retrieval labels attached to graph facts or paths; see GraphRAG.

Sources

1. Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217) — arXiv (Es, James, Espinosa-Anke, Schockaert), 26 Sep 2023
2. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (arXiv 2311.09476) — arXiv (Saad-Falcon et al.), 16 Nov 2023
3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al.), 9 Jun 2023
4. Introduction to Information Retrieval, chapter 8: Evaluation in information retrieval — Cambridge University Press (Manning, Raghavan, Schütze), 7 Apr 2008
5. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401) — arXiv (Lewis et al.), 22 May 2020
6. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023