What is Ragas, and how does it evaluate a RAG system?

In this article

1. Before you start
2. Why interviewers ask this
3. Ragas from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

The large language model (LLM) evaluation guide explains why a fixed test set and a defined metric come before any tool, and this page builds on it.

Why interviewers ask this

A RAG system has two parts that can fail independently, and a single "answer quality" score cannot tell you which. Interviewers ask about Ragas, and about metrics like it, to see whether you can split a pipeline into retrieval and generation and name a measurement for each. The question usually arrives as "how would you evaluate a RAG system" or "your RAG answers are wrong, how do you debug it". The RAG evaluation page takes the wider view of the same question.

A weak answer says "compare with a reference answer". A strong answer names the four things to measure, says which are judged by a model, and says what each low score would make you change.

Ragas from first principles

In plain words

Ragas

Ragas is a library of evaluation metrics and tooling for language-model applications, with a set of metrics aimed at retrieval-augmented generation. Its paper (Es et al., September 2023) describes a framework for evaluation that does not rely on ground-truth human annotations. The library later added more metrics, some of which need a reference.

Context precision

Context precision is a score from 0 to 1 for how high the relevant retrieved chunks rank: the mean of precision at each rank where a relevant chunk appears. An irrelevant chunk costs points only when it sits above a relevant one.

Picture a student writing an essay from a stack of library books. Faithfulness asks whether every sentence in the essay can be found in the books on the desk. Context precision asks whether the useful books were on top of the stack. Context recall asks whether all the books needed were on the desk at all.

The analogy breaks in one place: the student chooses the books, but in RAG the retriever chooses them before the writer starts.

In one precise sentence, Ragas turns a question, the retrieved passages, the answer and optionally a reference answer into scores between 0 and 1 for the retriever and the generator separately. The steps below build the scores in order.

Step 1: Collect a sample with four fields

Each Ragas test case holds what the metrics need: the question, the response the system gave, the retrieved passages and, for some metrics, a reference answer. The metric pages (read 2026-10-24) name these inputs user_input, response, retrieved_contexts and reference.

Different metrics need different fields. Faithfulness needs the question, the response and the retrieved contexts, and no reference. Context recall needs a reference answer. So some Ragas checks can run on live traffic, which has no reference, and others need a labelled set. The paper's three metrics (faithfulness, answer relevance and context relevance) need no reference, which is what it means by reference-free evaluation. The library later added metrics such as context recall and factual correctness that do compare against a reference.

The input is a logged interaction. The output is a case with the fields filled in.

Step 2: Split answers into claims and have a judge check each

Three of the metrics on this page (faithfulness, context recall and noise sensitivity) work on claims, which are single checkable statements pulled from a text, usually by a language model. Faithfulness splits the response into claims and asks, for each, whether the retrieved context supports it. Context recall splits the reference answer and asks the same question. The metrics overview says LLM-based metrics can be somewhat non-deterministic but are closer to human evaluation than string-similarity metrics, which are deterministic.

A claim is the unit of scoring because a whole answer is rarely wholly right or wholly wrong. An answer with five true sentences and one invented one should score high but not perfect, and claim counting gives exactly that.

The input is a text. The output is a list of claims, each marked supported or not.

Step 3: Score the retriever with context precision and context recall

Context precision and context recall judge the retrieved passages only. The context precision page defines it as the sum over ranks of precision at k times a relevance flag, divided by the number of relevant items in the top K. Its without-reference example shows order mattering: relevant chunk first scores about 1.0, an irrelevant one first about 0.5.

The context recall page defines the LLM-based version as reference claims supported by the retrieved context over total reference claims. Precision asks how high the relevant chunks rank; recall asks whether the needed information is present at all. Returning the whole corpus gives perfect recall, with precision depending entirely on ranking, since a chunk below the last relevant one costs nothing. A reranker that only reorders the same set changes precision and leaves recall unchanged. In the usual pipeline, though, the reranker keeps the top k of many candidates for the model, so context recall at a fixed k can rise too.

The input is the ranked passages and the reference. The output is two retriever scores.

Step 4: Score the generator with faithfulness and response relevancy

Faithfulness and response relevancy judge the answer. The response relevancy page defines it as the mean cosine similarity between the embedding of the user input and the embeddings of N questions generated from the response, with N defaulting to 3. It checks how directly the response addresses the question, not whether it is correct.

Faithfulness checks the other thing: whether the answer stays inside the retrieved context. The noise sensitivity page adds a third check: the share of response claims that are wrong against the reference and also supported by a relevant retrieved chunk (the default mode; mode="irrelevant" counts those supported only by irrelevant chunks), that is, errors the retrieved passages led the model into. Lower is better. An answer can be relevant and unfaithful, which is how a fluent hallucination looks.

The input is the response and its context. The output is the generator scores.

Step 5: Read the scores together

Read the scores as a diagnosis, not as an average. A low faithfulness with good retrieval points at the generator: the context contained the answer and the model drifted from it. A low context recall points at the retriever: the answer was never on the desk. Low precision with good recall points at ranking, where a reranker or a relevance filter helps. Low response relevancy points at an answer that does not address the question.

Averaging the scores hides this, because a strong retriever can pay for a weak generator in the mean. The scores also sit on different scales, so do not pick the lowest number. Compare each score with its own baseline on the same test set, read the failing cases, and fix retrieval first when recall is low, since missing context is a common cause of low faithfulness. Then re-run the same test set.

The input is the scores from steps 3 and 4. The output is the part of the pipeline to change.

1 · Sample2 · Claims3 · Retriever4 · Generator5 · DiagnoseReplay

Step-by-step description

1. Question, response, passages, reference.
2. A judge checks each claim.
3. Context precision and context recall.
4. Faithfulness and response relevancy.
5. Read each score against its baseline.

• Each test case holds the question, the response, the retrieved passages and sometimes a reference answer.
• Faithfulness, context recall and noise sensitivity split a text into claims and have a judge check each one.
• Context precision and context recall judge the retrieved passages.
• Faithfulness and response relevancy judge the answer.
• Each score is read against its own baseline on the same test set; low context recall is fixed first.

Worked example

One illustrative question is answered from five retrieved chunks, of which three are relevant. The response has seven claims: five are supported by the context and two are not. The two incorrect claims (wrong against the reference) are among the five supported ones, because they repeat a wrong figure from one of the three relevant chunks. The reference answer has five claims, four of them supported by the retrieved context. All figures come from a script that implements the formulas as written on the Ragas pages¹.

If the relevant chunks sit at ranks 1, 2 and 4, precision at those ranks is 1, 1 and 0.75, so context precision is their mean, 0.9167¹. Faithfulness is 5 of 7 claims, 0.7143. Context recall is 4 of 5 reference claims, 0.8. Noise sensitivity is 2 of 7 claims, 0.2857. For response relevancy, three questions generated from the response have cosine similarities of 0.9962, 0.953 and 0.9731 with the user input under toy three-number embeddings, a mean of 0.9741¹.

Text description of the diagram

• Faithfulness is 5 of 7 claims, 0.7143.
• Context precision is 0.9167 and context recall is 0.8.
• Response relevancy is 0.9741 with toy embeddings.
• Noise sensitivity is 2 context-supported incorrect claims of 7, 0.2857, and is better when lower.

The ordering rows show why precision is rank-aware. The same three relevant chunks score 1.0, 0.9167 or 0.4778 depending only on where they rank¹. The two irrelevant chunks ranked last in the 1, 2, 3 case cost nothing, so precision there is 1.0, not 0.6. Nothing was added or removed from the retrieved set, which is why the fix for low precision can be a reranker rather than a new index; see the cross-encoder reranker page.

Text description of the diagram

• Relevant chunks at ranks 1, 2 and 3 give context precision 1.0.
• Relevant chunks at ranks 1, 2 and 4 give 0.9167.
• Relevant chunks at ranks 3, 4 and 5 give 0.4778.

The documentation's own small examples are simpler. A faithfulness example (read 2026-10-24) with 2 claims of which 1 is supported scores 0.5, an identifier-based precision example with 2 of 4 retrieved identifiers matching scores 0.5 (plain set precision, not rank-aware; the rank-aware formula would give 0.75 on that example), and an identifier-based recall example finding 1 of 4 reference identifiers scores 0.25.

Trade-offs and edge cases

Ragas trades cost and determinism for coverage. LLM-based metrics need model calls for each case, so a large test set costs money and time, and the scores can shift between runs because the judge is itself a model. The Ragas documentation says such metrics may use one or more LLM calls to arrive at a score. Non-LLM variants of precision and recall compare identifiers or strings instead, at the cost of agreeing less with human judgement.

Four cases need care. First, a judge is a model: its agreement with people must be checked, as the LLM evaluation guide explains with sample sizes. Second, scores depend on the claim splitting, so two judges can disagree on a borderline claim. Third, a high score on a small test set says little; a set of 50 cases leaves a margin of about 11 points at an 80 percent score, a figure computed in that guide. Fourth, a metric measures one aspect: the overview says each metric should target one specific aspect, so a good score on all four still leaves latency, cost and safety unmeasured.

For production, log the same four fields as traces so that failures become test cases; the LLM observability page covers the logging side.

Text description of the diagram

• Low faithfulness points at the generator.
• Low context recall points at the retriever.
• Low context precision points at ranking.
• Low response relevancy points at an answer that does not address the question.

How to answer it in an interview

The spoken structure is: split the pipeline in two, name one metric per question, then say the judge needs validating and each score is read against its own baseline and the failing cases.

Common mistakes

• "Faithfulness measures whether the answer is correct" (wrong). Faithfulness checks consistency with the retrieved context, which can be wrong too (right: say what each metric compares against; factual correctness is the separate metric that compares the answer with a reference).
• "A high faithfulness means the answer is right" (wrong). An answer faithful to a bad passage is still wrong (right: pair it with context recall and a reference check).
• "I average the four scores into one" (wrong). The mean hides which part failed (right: read them together against each metric's own baseline).
• "Ragas needs no labels" (wrong). Some metrics need a reference answer; the paper's three metrics are reference-free, and later library metrics such as context recall are not (right: say which inputs each metric needs).
• "Higher is better for every Ragas score" (wrong). Noise sensitivity is better when lower (right: check the direction of each metric).

Follow-up questions

How do you use Ragas for RAG evaluation?

Build a test set of questions with the retrieved passages and the response, add a reference answer for the metrics that need one, run the metrics, then read the scores together. Faithfulness needs no reference: a judge model splits the response into claims and checks each against the retrieved passages. Response relevancy needs none either: it generates questions from the response and compares their embeddings with the original question. Context recall needs a reference answer.

What are the Ragas metrics for RAG evaluation?

The core ones are faithfulness, context precision, context recall, response relevancy and noise sensitivity; the library also lists context entities recall and multimodal variants. Precision and recall judge the retriever, faithfulness and relevancy judge the generator, and noise sensitivity is better when lower. Precision asks how high the relevant chunks rank, while recall asks how much of what was needed was retrieved at all.

What does a Ragas RAG evaluation example look like?

The worked example scores one question over five chunks: context precision 0.9167, faithfulness 0.7143, context recall 0.8, noise sensitivity 0.2857 and response relevancy 0.9741 on toy embeddings. Each number is a count or mean you can recompute by hand.

Can you trust an LLM to score faithfulness?

Trust it as far as you have measured it. Strong model judges have been reported to agree with human preferences over 80 percent of the time, with position, verbosity and self-enhancement biases, per the LLM-as-a-judge paper by Zheng et al. (2023), so score a sample with people first.

How many examples do you need to evaluate a RAG system?

Enough that the difference you care about exceeds the noise. About 100 cases give a margin near 8 points at an 80 percent score, and telling 80 from 85 percent takes about 906 per system on independent samples, or about 312 when both versions are scored on the same set and disagree on 10 percent of items, as the LLM evaluation guide computes.

Which Ragas metric would you fix first?

The one that is worst against its own baseline on the same test set, after reading the failing cases, because scores sit on different scales. Low context recall comes first: improve retrieval or chunking, or retrieve more candidates and rerank to k. Low faithfulness with good recall means constraining the generator, and low precision means reranking. For the model behind the generator, see how a transformer generates text.

Sources

1. Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217) — arXiv (Es, James, Espinosa-Anke, Schockaert), 26 Sep 2023
2. Faithfulness, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026
3. Context precision, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026
4. Context recall, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026
5. Response relevancy, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026
6. Noise sensitivity, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026
7. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685) — arXiv (Zheng et al.), 9 Jun 2023
8. Factual correctness, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026
9. Metrics overview, Ragas documentation — Ragas (page shows no date; read 2026-10-24), 24 Oct 2026