Evaluating a RAG system stage by stage: what to measure at each step

In this article

1. Before you start
2. Why interviewers ask this
3. Evaluating a RAG system from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the generative AI system design interview guide path and assumes only the three ideas below.

Why interviewers ask this

A RAG answer that is wrong tells you nothing about why, and the interviewer wants to see whether you can find out. Questions of this kind are reported by candidates in design rounds: evaluate each component of a RAG design, diagnose a quality drop after a data update, or explain why a system that passes its tests fails on real users.

A weak answer names one end-to-end score, such as "we check answer accuracy." A strong answer separates the stages, names a metric for each, and says which stage a given symptom points to. The paper that introduced the retrieval-augmented generation assessment (RAGAs) metric suite makes the same split: the RAGAs authors (Es et al., 2023) name three dimensions: whether retrieval finds relevant, focused passages, whether the model uses them faithfully, and the quality of the generation itself (arXiv abstract). A Staff-level answer adds the pipeline between the two ends and the gap between a test set and production traffic.

Evaluating a RAG system from first principles

Faithfulness

Faithfulness is a score from 0 to 1 that measures how factually consistent an answer is with the retrieved context: the share of the answer's claims that the retrieved context supports. It checks the answer against the retrieved text, not against the real world.

Context recall

Context recall is a retrieval score that measures how much of what a correct answer needs was actually retrieved: the share of the reference answer's claims that the retrieved context supports. It needs a reference to compare against.

In plain words

Picture a kitchen that sends out a wrong dish. The fault could be the shopping, the choice of ingredients from the pantry, the cooking or the plating. Tasting only the final dish cannot tell you which. So you check each station with its own test. The analogy breaks in one place: in a RAG system the stations are software, so each test can run automatically on stored examples.

A RAG system is a pipeline that first retrieves passages from a private collection of documents and then asks a language model to write an answer using them. The RAGAs paper puts it briefly: "RAG systems are composed of a retrieval and an LLM based generation module." Evaluating stage by stage means each module and each step inside it gets its own metric.

Step 1: Ingestion and chunking decide what can ever be found

A chunk is a passage cut from a source document and stored in the index as one searchable unit. If the answer to a question is split across two chunks, or a table was flattened into nonsense, no later stage can repair it.

Input: your documents. Output: chunks. To test this stage, take test questions whose answer you know sits in a specific passage and check whether the full answer text lives inside one chunk, and whether tables and headings survived extraction. This answer-in-one-chunk check is a simple test we suggest, not a standard metric, and it needs no language model. The page on RAG over real documents with tables and charts covers how extraction fails. You now know what the later stages can reach at best.

Step 2: Retrieval is scored with context recall and context precision

Retrieval is scored by asking whether the right chunks came back and whether the useless ones stayed out. Two scores from the Ragas documentation cover this.

Context recall is the "did we miss anything" score. The Ragas context recall documentation, read October 2026 says that "calculating context recall always requires a reference to compare against" . The reference is split into claims, and then, in the page's words, "each claim is analyzed to determine whether it can be attributed to the retrieved context or not" . The formula is supported reference claims divided by all reference claims, and an ID-based variant uses reference chunk identifiers found among the retrieved ones.

Context precision is the "did we bring junk" score. The Ragas context precision documentation, read October 2026 says it "evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones". Classic Recall@K and Precision@K count hits in the top K results and ignore order.

Input: a question and its retrieved list. Output: two numbers that point at the retriever.

Step 3: Reranking is scored by where the relevant chunks land

A reranker is a model that takes the retrieved chunks and reorders them so the most useful come first, usually reading the question and each chunk together. Because the reranker keeps the same set of chunks and only changes their order, set-based scores such as Recall@K cannot see its work. You need order-aware scores.

Context precision is order-aware by design. Mean reciprocal rank (MRR) averages one divided by the rank of the first relevant chunk. Normalised discounted cumulative gain (nDCG) rewards relevant chunks more the higher they sit, then divides by the best possible ordering. Compare the scores before and after the reranker on the same questions: if the numbers do not rise, the reranker is not earning its latency. The worked example below shows all three on one toy list.

Step 4: Generation is scored against the retrieved context, not the truth

Generation is scored with faithfulness, which checks the answer against the retrieved context, and with answer relevance, which checks that the answer addresses the question.

The Ragas faithfulness documentation, read October 2026 defines it this way: a response counts as faithful when "all its claims can be supported by the retrieved context." The method is to identify every claim in the response, then check each against the context; the score is the number of supported claims divided by the total number of claims , from 0 to 1. The same page's example has a context saying Einstein was born on 14 March 1879. An answer giving the wrong day makes one of two claims unsupported, so faithfulness is 1/2 = 0.5.

A language model does the claim-splitting and checking. An LLM judge is a language model used to grade another model's output, so the judge's quality moves the score. Answer relevance, a metric the RAGAs paper also uses, asks whether the answer addresses the question and not whether it is factual. Key point: an answer can be faithful to a wrong or stale chunk.

Step 5: Citations and the end-to-end answer close the loop

Citation checking asks whether each cited passage supports the claim it is attached to. Faithfulness does not do this: it tests claims against the whole retrieved context, so a claim can be supported by chunk 3 while the answer cites chunk 1.

Input: an answer with markers and the chunks they point to. Output: a per-claim verdict. The method is to take each claim and its cited passage alone and ask a judge, or a person on a sample, whether that passage entails the claim; then count claims with no citation at all. Last comes the end-to-end check: a small set of questions with reference answers and a score for final correctness. It is run last because the stage scores explain it; it cannot explain them.

1 · Ingestion2 · Retrieval3 · Reranking4 · Generation5 · CitationReplay

Step-by-step description

1. Check answers sit whole in one chunk.
2. Score context recall and precision.
3. Check relevant chunks move up.
4. Score faithfulness and relevance.
5. Check each claim against its source.

• Retrieval is scored with context recall and context precision.
• Generation is scored with faithfulness and answer relevance.
• Citation support is checked claim by claim.

Worked example

Take one toy question for which three chunks in the collection are relevant, and a top-five list that contains two of them. The relevance flags are illustrative. Three possible orderings of the same two relevant chunks show what order-aware scores see and set-based scores miss.¹

For ranks 1 and 3, precision at rank 1 is 1.0 and precision at rank 3 is 0.6667; averaging them over the two relevant positions gives 0.8333.¹ Context precision is 1.0 when both relevant chunks are ranked 1 and 2 and 0.325 when they are ranked 4 and 5. Recall@5 cannot tell these lists apart, because it counts only how many relevant chunks are in the top five. This is the Ragas formula: the sum of precision@k at each relevant rank, divided by the number of relevant chunks in the top K (Ragas context precision documentation, read October 2026).

Text description of the diagram

• Context precision is 1.0 when both relevant chunks are ranked 1 and 2.
• Context precision is 0.325 when both relevant chunks are ranked 4 and 5.

Now context recall: if a reference answer holds eight claims and seven are attributable to the retrieved context, context recall is 7/8 = 0.875 (illustrative counts).¹

Last, a diagnosis. Suppose a chatbot's faithfulness falls from 0.88 to 0.71 after a knowledge base update, measured on fifty answers of eight claims each (illustrative). That is 400 claims, with unsupported claims rising from 48 to 116, which is 68 more.¹ The stage scores locate the cause.

Recall barely moved, so the right chunks are still retrieved. Precision collapsed, so more irrelevant or duplicate chunks now sit among them, perhaps old and new versions of the same document. Faithfulness is scored against the whole retrieved context, so duplicates alone do not make claims unsupported; the drop comes when the generator, distracted by off-topic or conflicting chunks, writes more claims the context does not support. Before blaming retrieval, confirm the judge and the context window were unchanged and rerun the Step 1 chunk check on the updated documents. Then inspect the retrieved lists for duplicates and deduplicate or filter by document version.

Trade-offs and edge cases

Each score answers a different question, and only some need a reference answer.

Text description of the diagram

• Context recall always needs a reference to compare against.
• Faithfulness checks claims against the retrieved context.

Reference-free does not mean label-free. The RAGAs paper introduced "a suite of metrics which can be used to evaluate these different dimensions without having to rely on ground truth human annotations" , but context recall always needs a reference. Context precision can be judged against a reference answer, reference chunk IDs or, in a reference-free variant, the model's own response (Ragas context precision documentation, read October 2026). Annotating reference chunks is slow, which is why the language-model version of recall uses the reference answer as a proxy for them.

The judge is part of the measurement. Faithfulness and recall are computed by a language model that splits text into claims and checks each one, so a different judge model gives different numbers. Keep the judge fixed between runs you compare, spot-check a sample by hand, and send judge calls through the same controlled path as production traffic; the LLM gateway interview questions cover that layer.

Faithful can still be wrong. If the context is stale, a faithful answer repeats the stale fact. Faithfulness is silent here; recall against a current reference is what catches it.

Fact-heavy domains. For a help desk or historical question answering (QA), the Ragas documentation also offers a context entities recall score, which measures what fraction of the reference's entities was recalled. The Ragas context entities recall documentation, read October 2026 calls the metric "useful in fact-based use cases like tourism help desk, historical QA, etc."

Agent loops. When the system retrieves several times per question, as in agentic RAG, score every retrieval call, not just the last one.

Where the test questions come from

A RAG test set is a fixed list of questions, each with a reference answer and, ideally, the chunks that hold it. There are three common sources. A hand-written golden set, written by people who know the documents, is small but trusted. Sampled real queries, taken from logs with personal data removed and labelled by a person, show what users actually ask. Language-model-generated synthetic questions, produced from the documents, give cheap coverage, but a person should check them, because generated questions are often easier and worded more like the documents than real ones.

Keep the set small enough to label carefully and large enough that one question does not swing a score; the right size depends on how much a score must move to matter, so do not copy a number from another team. Freeze the set and its version between runs, add new failures as new questions, and never edit old ones silently. Beyond the set, watch online signals such as thumbs-down, rephrased queries, abandoned sessions and escalations to a person, and send the failures back into the set.

How to answer it in an interview

The spoken structure: name the stages, give one metric per stage, then say what faithfulness cannot see.

Common mistakes

• "Our faithfulness is high, so the answers are correct." (wrong) Faithfulness only checks claims against the retrieved context. (right) "Faithfulness says the answer follows the context; recall says whether the context held the right facts."
• "One overall accuracy score is enough." (wrong) It hides which stage failed. (right) "I report retrieval, ranking, generation and citation scores next to the end-to-end number."
• "Precision@K tells me how good the ranking is." (wrong) Precision@K ignores order within the top K. (right) "I use context precision, mean reciprocal rank or nDCG to see order."
• "Reference-free metrics need no labels at all." (wrong) Context recall always needs a reference. (right) "Some scores are reference-free; recall is not, so I keep a small labelled set."

Follow-up questions

How do you evaluate RAG systems specifically?

Evaluate retrieval and generation separately, then the end-to-end system. Retrieval gets context recall and precision, generation gets faithfulness and answer relevance, and the end-to-end check compares final answers with references.

When RAG aces evals but fails on real queries, how do you close the gap?

Treat the test set as the suspect. Sample real queries, label a slice, and compare their wording and difficulty with your test questions; add the failures to the test set. Track online signals such as thumbs-down and rephrased queries, and run the stage scores in a release gate so each change is compared against the same fixed questions before release.

How do you verify that answer citations actually support the claims?

Check claim by claim: pair each claim with its own cited passage, ask a judge or a person whether the passage entails the claim, and count claims with no citation. Faithfulness alone does not do this, because it uses the whole retrieved context.

How do you diagnose a faithfulness drop after a knowledge base update?

Compare recall and precision before and after on the same questions. In the worked example, flat recall and falling precision pointed at noisy or duplicated chunks, not missing ones; the numbers there are illustrative.

Your RAG system returns relevant documents but users still cannot find the answer. What do you change?

The fault is usually in the answer layer rather than retrieval, though chunk size or ranking can also cause it: add generation that states the answer directly with citations, and score answer relevance and citation support. The Perplexity design walkthrough shows an answer engine built this way.

How does evaluation change when the front end is a voice agent?

Speech recognition becomes a stage before retrieval, so score its transcript errors separately, because a misheard question lowers every later score. See how to design a voice agent.

Sources

1. Faithfulness - Ragas documentation — Ragas (vendor documentation, undated, accessed 2026-10-26), 26 Oct 2026
2. Context Precision - Ragas documentation — Ragas (vendor documentation, undated, accessed 2026-10-26), 26 Oct 2026
3. Context Recall - Ragas documentation — Ragas (vendor documentation, undated, accessed 2026-10-26), 26 Oct 2026
4. RAGAs: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217) — arXiv (Es, James, Espinosa-Anke, Schockaert), 26 Sep 2023
5. Context Entities Recall - Ragas documentation — Ragas (vendor documentation, undated, accessed 2026-10-26), 26 Oct 2026