How do you debug a RAG system that gives wrong answers?

In this article

1. Before you start
2. Why interviewers ask this
3. RAG debugging from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of the retrieval-augmented generation guide, and it assumes you know what the pipeline does: split documents into pieces, store them, fetch the best pieces for a question, and let a model answer from them (Lewis et al., 2020).

Why interviewers ask this

A wrong answer from a RAG chatbot is the problem teams face after launch, so the question separates candidates who have operated one from those who have only built a demo. Candidates report being asked to walk through debugging a chatbot that is often wrong, or a pipeline that began hallucinating after a data refresh.

A weak answer jumps to a fix: a better prompt, a bigger model, a larger context. A strong answer starts from one failing question, locates the evidence, and rules out stages in order. A Staff-level answer adds measurement: it labels a sample of failures by stage, so the team knows which fix can move the failure rate and which cannot. Barnett et al. (2024) report seven failure points drawn from three case studies, and argue that a RAG system is only validated during operation.

RAG debugging from first principles

Retrieval failure

A retrieval failure is a wrong answer caused by the passages handed to the model: the evidence was absent from the corpus, or it was present but not among the chunks fetched for the question.

Generation failure

A generation failure is a wrong answer produced even though the evidence was in the model's prompt: the model ignored it, misread it, or added claims the passages do not support.

Gold chunk

A gold chunk is the passage in your corpus that contains the correct answer to a test question, found by a person or a trusted label, and used as the reference when you check retrieval.

In plain words

Think of a student taking an open-book exam. A wrong answer has three possible causes. The book never contained the fact. Or the student opened the wrong page. Or the student had the right page open and misread it. A better pen fixes none of these. The analogy breaks in one place. The "student" here is a model. It may answer from memory when the page is unhelpful. So a confident wrong answer can come from missing evidence.

In one precise sentence, a RAG failure is localised by comparing three things. They are the gold chunk, the retrieved chunks and the claims in the answer.

Step 1: Capture the trace and find the gold chunk

Step 1 turns "the bot is wrong" into one reproducible case. Pick a failing question and save its full trace: the question as the retriever saw it, the ranked chunks with scores, the exact prompt sent to the model, and the answer. Without the prompt text you are guessing.

Step 1 finds the gold chunk by searching the raw corpus with plain text search. Input: the question and the correct answer. Output: either a chunk, or the finding that no document contains the answer. If the answer is not in the corpus, the failure is upstream of retrieval: a missing, unparsed or outdated document, and no prompt change can repair it. You now know whether the gold chunk exists, and Step 2 asks whether the system found it.

Step 2: Check whether retrieval fetched the gold chunk

Step 2 compares the gold chunk with the top-k. The top-k is the k highest-scoring chunks the retriever returns and passes to the model; a common default is a handful of chunks, such as five (illustrative). Recall@k (also called hit rate) is the share of test questions whose gold chunk appears in the top-k. Evaluation tools such as Ragas also report context precision, which asks whether the retrieved chunks are relevant, and context recall, which asks whether the retrieved context covers the reference answer.

If the gold chunk is in the top-k, retrieval did its job and the suspect moves to the model. If it is not, ask where it ranked. A gold chunk at rank 12 with k set to 5 is a ranking problem, fixed by a reranker, a larger k, or hybrid search that mixes keyword and vector scores. A gold chunk that never appears in the first 100 results points to the chunk itself: the answer may be split across two chunks, buried under a long heading, or embedded poorly. Chunking choices can cause these misses.

This is the check from the lead: reading retrieved text first prevents hours of prompt edits for a problem that sits in the index. You now know which side of the retrieval boundary the failure is on.

Step 3: Check whether the model used the context

Step 3 applies only when the gold chunk is in the prompt. Read the answer claim by claim and mark each one supported or unsupported by the retrieved text. Faithfulness is the share of the claims in an answer that the retrieved context supports; the Ragas documentation defines it as supported claims divided by total claims, on a scale from 0 to 1.

An unsupported claim with the evidence in context is a generation failure. Common causes are a long prompt in which the model underuses material, a question that needs two chunks combined, or a prompt that does not say "answer only from the passages". Liu et al. (2023) study how language models use long contexts, so position and length of the context are worth testing. Move the gold chunk to the top, drop distractor chunks, and see whether the answer changes.

Step 4: Check what changed since it last worked

Step 4 handles regressions, such as a score that fell after a knowledge base update. Compare the old and new index on the same test questions: did the chunk boundaries move, was a different embedding model used for new documents than for old ones, did duplicates or superseded versions of a document enter, and did the parser drop tables or headings?

Run the Step 2 check on both versions. If recall@k fell, the cause is in ingestion or retrieval. If it stayed flat while faithfulness fell, look at the prompt template or the model version. A score drop after an update points to the index, chunking or the prompt. Contradictory answers from different documents are also a corpus finding: two valid-looking chunks disagree, so the fix is version and authority metadata, not a smarter model.

1 · Find the gold chunk2 · Check retrieval3 · Check generation4 · Check what changedReplay

Step-by-step description

1. Search the raw corpus for the answer.
2. Is the gold chunk in the top-k?
3. Is every claim supported by context?
4. Compare the old and new index.

• Step 1 finds the gold chunk by searching the raw corpus.
• Step 2 checks whether retrieval fetched the gold chunk.
• Step 3 checks whether the model used the context.
• Step 4 checks what changed since the system last worked.

Worked example

The numbers below are illustrative, built to show the method. A team keeps a test set of 1,000 questions and 300 fail, a 30% failure rate. They triage a sample of 100 failures with the four steps above.¹ The result: 20 gold chunks were not in the corpus, 30 were in the corpus but not in the top-k, 35 were in the prompt and ignored or misread, and 15 answered from a stale or conflicting chunk instead of the gold chunk.¹

The gold chunk was in the top-k for only 35 of the 100 failures, so for the other 65 the cause sits in the corpus, in retrieval or in the sources.¹ Tuning the prompt first would have worked on 35 failures at best.

Now the regression. Faithfulness of 0.88 means roughly 88 of 100 claims are supported; 0.71 means roughly 71 of 100, an absolute drop of 0.17.¹ Suppose faithfulness is 0.92 when the gold chunk is in context and 0.50 when it is not. If the share of questions with the gold chunk in the top-k falls from 0.90 to 0.50 after the update, the overall score moves from about 0.88 to 0.71 with no change to the model.¹ The diagnosis is then a retrieval regression, found by splitting the score on one question: was the gold chunk retrieved?

Text description of the diagram

• In the illustrative triage, 20 gold chunks were not in the corpus and 30 were not retrieved.
• 35 failures had the gold chunk in the prompt and 15 came from a stale or conflicting source.

Trade-offs and edge cases

The order of the checks is a trade-off between cost and certainty. Searching the corpus and reading the top-k are cheap and decisive, so they come first; judging faithfulness claim by claim is slower and often done by a model, which adds its own error, so spot-check by hand. A RAG evaluation setup automates the labelling once the method works.

Edge cases: the answer may need several chunks, so "gold chunk" becomes a set, and a question about relationships across documents may need a different design such as GraphRAG. Retrieval can look right while the answer is stale, as in the 15 illustrative failures above. Corrective RAG (CRAG) adds a retrieval evaluator that decides what to do when retrieved text looks poor (Yan et al., 2024). That helps with bad retrieval but does not fix a corpus that lacks the answer.

Text description of the diagram

• An answer that says no information was found points to the corpus or retrieval.
• Right context with a wrong answer points to generation.
• A score drop after an update points to the index, chunking or the prompt.

How to answer it in an interview

The spoken structure: trace one case, check the corpus, check retrieval, check generation, then measure the buckets before choosing a fix.

Common mistakes

• "I'd improve the prompt first." (wrong) That reaches only failures where the evidence was in context. (right) "I read the retrieved chunks first, because prompt changes cannot fix missing evidence."
• "RAG removes hallucination." (wrong) The model can still add unsupported claims; the original aim was to reduce them (Barnett et al., 2024). (right) "RAG reduces hallucination, and I measure faithfulness to check it."
• "More context is always safer." (wrong) Long contexts are used unevenly. (right) "I test fewer, better-ranked chunks against a larger k."
• "The score dropped, so the model changed." (wrong) The index may have changed. (right) "I split the score by whether the gold chunk was retrieved."

Follow-up questions

How do you tell if a wrong answer is a retrieval failure vs a generation failure?

Check whether the gold chunk is in the top-k. If it is absent, the answer is a retrieval failure. If it is present and the answer still has claims the passages do not support, the answer is a generation failure.

How would you handle the problem of a model hallucinating when no information is found?

Make "I could not find this" a valid output: instruct the model to answer only from the passages, add a retrieval-score threshold below which it declines, and test with questions that have no answer in the corpus. CRAG-style evaluators decide what to do when retrieval looks poor.

Your RAG chatbot's faithfulness score drops from 0.88 to 0.71 after a knowledge base update. How do you diagnose the cause?

Run the same test questions on the old and new index and compare recall@k. A fall there means ingestion or chunking changed; no fall means the prompt template or model changed. The worked example above shows this split.

Your enterprise RAG system returns contradictory answers from different source documents. How do you resolve conflicts?

First confirm both chunks are retrieved, then compare dates, versions and owners. Add that metadata to the index, filter or prefer the authoritative version, and when two current sources disagree, have the answer name both with citations rather than choose silently.

Why do LLMs hallucinate even with RAG?

The retriever may miss the right passage, the model may underuse a long context, or may fill gaps from memory. Each case is a separate failure stage in the RAG interview questions you may be asked about.

What is a corrective RAG system?

Corrective RAG adds a step that scores retrieved documents and triggers a fallback when they look weak (Yan et al., 2024). It improves robustness but still depends on a corpus that holds the answer.

Sources

1. Seven Failure Points When Engineering a Retrieval Augmented Generation System (arXiv 2401.05856v1) — arXiv (Barnett et al.), 11 Jan 2024
2. Faithfulness (Ragas documentation) — Ragas, 26 Oct 2026
3. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401) — arXiv (Lewis et al.), 22 May 2020
4. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023
5. Corrective Retrieval Augmented Generation (arXiv 2401.15884) — arXiv (Yan et al.), 29 Jan 2024