In this article
  1. Before you start
  2. Why interviewers ask this
  3. Agentic RAG from first principles
  4. Worked example
  5. Trade-offs and edge cases
  6. How to answer it in an interview
  7. Common mistakes
  8. Follow-up questions
  9. Sources

Before you start

Why interviewers ask this

Agentic RAG is the question that tests whether you add complexity only when it pays. A candidate who answers "use agents for everything" has not costed the loop, and a candidate who answers "standard RAG is enough" has not met a question that needs two searches. Interviewers use it in system design rounds because the right answer is a trade-off: accuracy on hard questions against latency, cost and new failure modes. A weak answer names frameworks. A strong one draws the loop, says when it stops, and names which questions should skip it. The wider design questions that surround this one are in the GenAI system design hub.

Agentic RAG from first principles

Agentic RAG
Agentic RAG is a retrieval-augmented generation pipeline in which a language model acts as a controller that decides whether to retrieve, writes the search queries, judges the retrieved passages and repeats or stops, so the number of retrieval steps varies with the question.
Retrieval grader
A retrieval grader is a model or scoring component that reads the retrieved passages for a question and returns a verdict on whether they are good enough to answer from, which decides whether the loop retries or answers.
Query rewriting
Query rewriting is the step that turns a user's question into one or more search queries, by rephrasing it or splitting it into sub-questions, so that the retriever searches for what is actually needed.

In plain words

Standard RAG is a vending machine. You press a button and one fixed thing comes out. Agentic RAG is a research librarian. The librarian decides whether the library is needed, searches, reads the results, and searches again if they were wrong. The analogy breaks in one place. A librarian knows when they are lost, but a model's judgement that "this is good enough" can be wrong, so the loop needs a hard stop.

Stated precisely, agentic RAG wraps retrieval in a decision loop that follows the pattern of ReAct, which mixes reasoning with actions that gather outside information. The original RAG paper (Lewis et al., revised April 2021) joins a language model to a retriever over a document index. That single pass is what agentic RAG builds on.

Step 1: Route the question

The first step decides whether to retrieve at all. The input is the user's question; the output is one of three choices: answer directly, retrieve once, or retrieve and iterate. Adaptive-RAG (Jeong et al., March 2024) trains a small classifier to predict the complexity of a query and pick no retrieval, single-step or iterative retrieval, and reports gains in both efficiency and accuracy. Self-RAG (Asai et al., October 2023) likewise trains one model to retrieve only when needed. A greeting or a question the model already knows should cost one call, not eight. What you now know: the cheapest answer is the one that skips the loop.

Step 2: Write the search query

The second step turns the question into a search. The input is the question, plus anything already retrieved; the output is one or more queries. Query rewriting handles two failures of the raw question: it may be vague ("how did that go?") or it may hide several facts. A multi-hop question, one whose answer needs facts from more than one passage, is split: "Who founded the company that makes product X?" becomes "which company makes product X" and then "who founded that company". Later rounds use the failed attempt as a hint for a better query. What you now know: the model chooses what to look for, not just what to read.

Step 3: Retrieve and grade the results

The third step runs the search and checks the answer's raw material. The input is the query; the output is a set of passages and a verdict. Corrective RAG (Yan et al., revised October 2024) uses a lightweight evaluator that scores the quality of retrieved documents and returns a confidence level that selects the next action. The grader can be a model call, a score from a reranker such as a cross-encoder, or a simple rule on similarity scores; a cheaper grader makes each round cheaper. The retrieval itself uses the usual embeddings and vector search. What you now know: the loop only works if the verdict is reliable.

Step 4: Retry or fall back

The fourth step handles a failed verdict. The input is the verdict and the failed query; the output is a new action: rewrite and search again, search a different source, or give up politely. Corrective RAG falls back to large-scale web search when the retrieved documents are judged poor, because a static corpus can be incomplete. In an enterprise setting the fallback is usually another index or a clear "not found". Each retry needs a reason that differs from the last, otherwise the loop repeats the same failing query. What you now know: a retry must change something, or it only burns calls.

Step 5: Answer from the evidence and stop on a budget

The last step writes the answer from passages that passed the grade and ends the loop. The input is the passed evidence; the output is the answer with citations to the passages. Set a hard limit on rounds, tokens or time, and decide in advance what to do when the limit is hit: answer with what you have and say so. Without a limit, the cost of the loop is unbounded. In the model below, a loop capped at three rounds makes eight model calls where standard RAG makes one, and about five on average when the grader passes most first tries.1 What you now know: the budget is the design decision that makes the loop safe to ship.

Agentic RAG is a loop of route, write, retrieve, grade and stopBoxes for the agentic RAG loop: a router decides whether to search, a query writer drafts the search, retrieval and grading check the results, a failed grade sends the loop back to the writer, and a passed grade goes to the answer.Routesearch, oranswer?Writequeryrewrite orsplitRetrieve,gradegoodenough?failnew tryRetry orfall backnew queryor sourcepassAnswerpass orbudget hitDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/agentic-rag.html

The loop retries until the grader passes the evidence or the budget runs out.CC BY 4.0 — source: What is agentic RAG? How it differs from standard RAG
Step-by-step description
  1. Decide whether to retrieve at all.
  2. Draft or rewrite the search from the question.
  3. Fetch passages and judge whether they answer it.
  4. A failed grade sends the loop back to the writer.
  5. Answer from passed evidence; a step budget ends the loop.
  • The first step decides whether to retrieve at all.
  • The second step turns the question into a search.
  • A grader checks the retrieved passages, and a failed grade returns the loop to the writer.
  • The last step writes the answer from passed evidence, and a step budget ends the loop.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.

Start Free Trial →

Worked example

The model compares standard RAG with an agentic loop under illustrative inputs: each model call takes two seconds, each round retrieves five chunks of 300 tokens, and the grader passes a round with probability 0.6. Prompt overheads are 150 tokens for the generator, 350 for the router, 400 for the query writer and 200 for the grader, each plus its chunks where it reads them. Standard RAG makes one call: 150 + 1,500 chunk tokens = 1,650 input tokens. The loop makes one routing call, then a query-writing call and a grading call per round, and a final answering call, so it makes four calls after one round, six after two and eight after three.1

At two seconds a call that is two seconds against sixteen in the three-round worst case, counting sequential model calls only and leaving retrieval time out. One round sums to 350 + 2,100 + 1,650 = 4,100 input tokens, where 2,100 is the writer's 400 plus the grader's 200 and 1,500 chunk tokens; each extra round adds another 2,100. Reading five chunks in every grading call, the input tokens grow from 1,650 to 8,300 in the worst case, about five times the standard pipeline.1

A three-round agentic answer makes eight model calls against one for standard RAG, and about five calls on average when most first tries pass.
FactStandard RAGAgentic, 1 roundAgentic, 3 rounds
Model calls148
Latency at two seconds a call2 s8 s16 s
Input tokens1,6504,1008,300
Input tokens vs standard1.0x2.48x5.03x

The expected value matters more than the worst case. If a round passes the grader 60% of the time, the expected number of rounds with a cap of three is 1.56, and the expected number of calls is 5.12.1 Real systems show the same direction: Anthropic reports that agents typically use about four times the tokens of chat interactions, and multi-agent systems about fifteen times (Anthropic, June 2025).

Each retrieval round adds two model calls to an agentic RAG answerBar chart of model calls per question in an illustrative cost model: standard RAG makes one call, and an agentic loop makes four, six or eight calls for one, two or three retrieval rounds.Model calls (calls)02468Standard1 round2 rounds3 roundsPipeline1468Diagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/agentic-rag.html
Every round adds a query-writing call and a grading call, so a three-round answer costs eight calls.CC BY 4.0 — source: What is agentic RAG? How it differs from standard RAG
Text description of the diagram
  • Standard RAG makes one model call per question.
  • An agentic loop makes four calls after one round, six after two rounds and eight after three rounds.

Trade-offs and edge cases

Agentic RAG is worth its cost only when the question needs it. Anthropic contrasts traditional RAG, which uses static retrieval, with a multi-step search that dynamically finds relevant information, and reports that a multi-agent system outperformed a single agent by 90.2% on its internal research evaluation, at the token cost above. For a frequently asked, single-fact question the loop only adds delay, which is the point of routing in Step 1.

Standard, agentic and graph RAG each suit a different kind of questionTable comparing standard RAG, agentic RAG and GraphRAG on control flow, the questions each suits, extra cost and main risk: agentic RAG loops and costs more calls, GraphRAG builds a graph before any query.QuestionStandardAgenticGraphRAGControl flowOne fixedpassModel loops,decidesGraph builtfirstSuitsSingle factsMulti-hopquestionsCorpusthemesExtra costNoneCalls perroundIndex-timemodel callsMain riskMissesglobalquestionsLoops,wronggradesHeavy indexbuildDiagram: Interview Clozo, CC BY 4.0 ·interview.clozo.ai/blog/agentic-rag.html
Pick the pipeline by question type: single facts, multi-hop questions or whole-corpus themes.CC BY 4.0 — source: What is agentic RAG? How it differs from standard RAG
Text description of the diagram
  • Standard RAG runs one fixed retrieve-then-generate pass.
  • Agentic RAG lets the model loop and decide, and suits multi-hop questions.
  • GraphRAG builds an entity graph before queries and suits whole-corpus questions.

Four edge cases decide most follow-ups. First, a poor grader breaks the loop: a grader that passes bad passages gives a confident wrong answer, and one that rejects good ones wastes calls. Second, the loop can repeat itself, so keep a record of past queries and reject a repeat.

Third, retrieval quality still sets the ceiling; Anthropic's contextual retrieval post (September 2024) reports that adding chunk-level context cut failed top-20 retrievals from 5.7% to 3.7%, and to 1.9% when combined with keyword search and reranking, before any loop was added. Fourth, long contexts have a weakness: models do best when the relevant text is at the start or end of the input (Liu et al., 2023), so rank the passages before you place them.

How to answer it in an interview

Interviewer asks

What is agentic RAG, and when would you use it over standard RAG?

A strong answer

Agentic RAG puts a model in control of retrieval: it decides whether to search, writes the query, grades the results and retries or stops. I would use it for multi-hop questions, where one search cannot find every fact, and keep standard RAG for single-fact questions. The cost is extra model calls: a three-round loop is about eight calls instead of one, so I would route simple questions around it, cap the rounds, and measure answer quality and cost together.

Say it in this order: definition, when to use it, the cost, the controls. If you have time, mention the grader as the weak point.

Common mistakes

  1. Using the loop for every question. (wrong) "Every query goes through the agent." (right) "Route simple questions to one retrieval, or none, and loop only on hard ones."
  2. Having no stop condition. (wrong) "It retries until the grader is happy." (right) "It stops after a fixed number of rounds and answers with what it has."
  3. Trusting the grader blindly. (wrong) "The model judges relevance, so the evidence is good." (right) "I evaluate the grader against labelled examples before relying on it."
  4. Retrying the same query. (wrong) A retry that repeats the failed search. (right) Each retry rewrites the query or changes the source.
  5. Ignoring the cost. (wrong) "Agentic RAG is more accurate, so it is better." (right) "It is more accurate on some questions and costs several times as many calls."

Follow-up questions

Agentic RAG vs traditional RAG?

Traditional RAG runs one fixed pass: retrieve, then generate. Agentic RAG lets the model decide whether to retrieve, rewrite the query, grade the results and loop, so it handles multi-hop questions better and costs more calls on every question.

Agentic RAG vs GraphRAG?

Agentic RAG and GraphRAG solve different problems. GraphRAG uses a language model to build an entity knowledge graph and pregenerated summaries of related groups of entities, and answers whole-corpus questions such as "what are the main themes" by merging partial answers; the paper reports gains in comprehensiveness and diversity on corpora of about one million tokens (Edge et al., revised February 2025). Agentic RAG adds a decision loop at query time instead. They can be combined.

How to deploy agentic RAG for customer service automation?

Start with routing so common questions skip the loop, cap the rounds, and define a fallback that hands over to a person when the grader fails twice. Log every query, retrieval and grade so a wrong answer can be traced. Keep a human review path for actions with side effects; see human in the loop for AI agents.

No ranking is possible without testing on your own documents, so judge a platform by its controls: a round limit, access control on retrieval, traces of every step, an evaluation harness and a clear fallback. Compare them on a labelled question set from your own corpus. For tracing, see observability for language-model apps.

How do you evaluate an agentic RAG system?

Measure retrieval, grounding and the answer separately. Ragas is a reference-free framework that scores how well retrieval finds focused context, how faithfully the model uses it and the quality of the answer (Es et al., revised April 2025). Add cost per answer and rounds per question, because a good answer at eight calls is a different product from one at one call. Related reading: AI agents, tool use and the Model Context Protocol and machine learning system design.

Key takeaways
  • Agentic RAG puts a model in control of retrieval: route, write, grade, retry, stop.
  • Use it for multi-hop questions; route single-fact questions around it.
  • Each round adds two model calls; the three-round loop makes eight against one.
  • Cap the rounds and define what happens at the cap.
  • Evaluate the grader, retrieval and the answer separately.

Sources

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401 v4) — arXiv (Lewis et al.), 12 Apr 2021
  2. ReAct: Synergizing Reasoning and Acting in Language Models (arXiv 2210.03629 v3) — arXiv (Yao et al.), 10 Mar 2023
  3. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (arXiv 2310.11511) — arXiv (Asai et al.), 17 Oct 2023
  4. Corrective Retrieval Augmented Generation (arXiv 2401.15884 v3) — arXiv (Yan et al.), 7 Oct 2024
  5. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (arXiv 2403.14403 v2) — arXiv (Jeong et al.), NAACL 2024, 28 Mar 2024
  6. From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv 2404.16130 v2) — arXiv (Edge et al.), 19 Feb 2025
  7. Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217 v2) — arXiv (Es et al.), 28 Apr 2025
  8. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172 v3) — arXiv (Liu et al.), 20 Nov 2023
  9. Introducing Contextual Retrieval — Anthropic, 19 Sep 2024
  10. How we built our multi-agent research system — Anthropic, 13 Jun 2025