In this article
Before you start
This page belongs to the retrieval-augmented generation (RAG) learning path; the search step it builds on is explained in vector search and embeddings.
Why interviewers ask this
Multi-hop retrieval is the first place where a plain retrieve-then-read pipeline visibly breaks, so a RAG design round often reaches it. The question separates candidates who treat retrieval as a single lookup from candidates who see that the next thing to look up can depend on what has already been worked out.
A weak answer says "retrieve more documents" or "use a better embedding model." A strong answer says why one search cannot find evidence that is only named inside another document, describes the loop that alternates reasoning and searching, and names its costs: extra model calls, extra delay and mistakes that compound. A Staff-level answer adds how to measure it, with retrieval scored separately from the final answer, and when a different design, such as a graph, fits better. The sections below build each of those pieces from zero.
Multi-hop retrieval from first principles
- Multi-hop query
- A multi-hop query is a question that can be answered only by finding several pieces of evidence, usually in different documents, and combining them by reasoning, so that no single passage contains the whole answer.
- Hop
- A hop is a retrieval round that returns passages moving the answer closer, usually because its query uses a fact found in the previous round; a two-hop question needs two such rounds.
- Interleaved retrieval
- Interleaved retrieval is a loop that alternates between a model writing one reasoning step and a retriever searching with that step, so each search is guided by the reasoning so far and each new result improves the next reasoning step.
In plain words
Think of a detective with a case file room. The question is "who owns the car that was parked outside?" The first file gives a plate number, not a name. The detective must write the plate number on a new request slip and fetch a second file, the registry. Asking for the right second file was impossible before reading the first.
The analogy breaks in one place: a detective knows what a registry is, while a retrieval loop must let the language model decide which fact to search for next. In one precise sentence, multi-hop retrieval makes each search query a function of the evidence already found, instead of a function of the original question only.
Trivedi et al. (2022) put the problem in one line: in multi-step question answering (QA), what to retrieve depends on what has already been derived. HotpotQA, a standard benchmark for such questions, is a Wikipedia-based dataset of 113k question-answer pairs whose questions require finding and reasoning over multiple supporting documents.
Step 1: Retrieve a base set with the question
The loop begins by retrieving a base set of paragraphs using the question as a query. This is ordinary first-stage search (the paper uses Best Matching 25 (BM25) keyword search; vector search works the same way): score the passages against the question and return the best ones.
Input: the question. Output: a handful of passages. Sometimes this base set already holds everything and the loop ends here. For a multi-hop question it usually holds only the first link of the chain. The paper's own example asks which country the roller coaster Lost Gravity was manufactured in. The page found with the question alone is about the coaster; it names the manufacturer but not the manufacturer's country. You now know what the first round gives you: a starting point, not an answer.
Step 2: Write one reasoning sentence and use it as the next query
The model reads the question and the passages so far, then writes one sentence of reasoning, and that sentence becomes the next retrieval query. Interleaved retrieval, in the words of Trivedi et al. (2022), interleaves retrieval with the sentences of a chain-of-thought, guiding the retrieval with the reasoning and using retrieved results to improve the reasoning.
In the coaster example the first sentence names the manufacturer, Mack Rides. That name was not in the question, so no search with the question could have used it. Searching with "Mack Rides" retrieves the page about the manufacturer. Input: question plus passages so far. Output: one sentence, used as the query. This is the missing piece: the second query is written by the model, from derived facts.
Step 3: Add the new passages and reason again
The passages from the second search are added to the evidence, and the model writes the next reasoning sentence with them in view. Each new sentence is checked against fresh text, which is why Trivedi et al. (2022) report that the approach reduces hallucination and gives more accurate reasoning.
Input: the evidence so far. Output: either another reasoning sentence, which triggers another search, or a final answer. The step is the same each round; only the evidence grows. Agent designs in the ReAct style interleave reasoning and actions such as search in the same way, as Yao et al. (2022) describe.
Step 4: Stop at an answer or at the iteration cap
The loop ends when the model writes a final answer or a maximum number of iterations is reached; otherwise it goes back to the reasoning step for another hop. Without the cap, a question the corpus cannot answer could loop without end and spend a model call each time round.
Input: the latest model output. Output: stop or continue. The cap is a design parameter you choose from the cost and the questions you expect; the worked example shows why it should be small. If the cap is reached without an answer, a good system says the evidence was not found rather than guessing, because a guess built on partial evidence reads as confident as a grounded answer. You now have the whole loop: search, reason, search again, and stop.
Step-by-step description
- The question is the first query.
- One sentence becomes the next query.
- Results join the evidence.
- Answer or cap: stop; otherwise loop back.
- The loop begins by retrieving a base set of paragraphs using the question as a query.
- Each reasoning sentence becomes the next retrieval query.
- The loop ends at a final answer or the maximum number of iterations.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Two things can be worked out by hand: what the loop does to the coaster question, and what extra hops cost. For the first, round one returns the coaster's page, which names the manufacturer, Mack Rides, but not its country; the reasoning sentence states that fact, round two searches with it and returns the Mack Rides page, and the answer, Germany, is read from it.
For the cost, assume (illustrative, not measured) that each hop independently finds its evidence with probability 0.8. All the evidence for a one-hop question is then found with probability 0.8, for two hops 0.64, for three 0.512 and for four 0.4096.1 With 0.9 per hop, four hops still succeed only with probability 0.6561.1 If each round costs two seconds (illustrative) and the final answer costs two more, four hops take 10 seconds and five model calls.1
| Hops | Chance all evidence is found at 0.8 per hop (illustrative) | Model calls | Seconds (illustrative) |
|---|---|---|---|
| 1 | 0.8 | 2 | 4 |
| 2 | 0.64 | 3 | 6 |
| 3 | 0.512 | 4 | 8 |
| 4 | 0.4096 | 5 | 10 |
The pattern is multiplication: each hop must work for the next to matter, so chances multiply and delays add. A cleaner first-stage search and a reranker lift the per-hop rate, which matters more than it looks.
Text description of the diagram
- At 0.8 per hop, four hops find all evidence with probability 0.4096 (illustrative).
- At 0.9 per hop, four hops find all evidence with probability 0.6561 (illustrative).
The Lost Gravity wording comes from the paper; the probabilities are an assumption. Real per-hop rates must be measured on your own questions.
Trade-offs and edge cases
An interleaved loop trades speed and simplicity for the ability to follow a chain. The comparison below holds the main differences.
Text description of the diagram
- A single-step search writes every query from the original question.
- An interleaved loop writes the next query from the evidence found.
- A single search fails when the evidence depends on an earlier answer.
A single-step search writes every query from the original question. An interleaved loop writes the next query from the evidence found. A single search uses one model call; a loop uses one per hop plus the answer. A single search fails when the evidence depends on an earlier answer; a loop fails when an early hop goes wrong.
Bigger top-k and better embeddings do not remove the core issue, because the second query cannot be written before the first hop returns. The benchmark work agrees: Tang and Yang (2024) find that existing RAG systems are inadequate for multi-hop queries and that embedding models perform unsatisfactorily at retrieving their evidence.
Edge cases: a bad first hop poisons everything after it, so check the first reasoning sentence against its passages. Reranking each round keeps junk out of the evidence; see the reranker page. Each hop also adds cost, covered in where RAG latency and cost go. When a system answers wrongly, test each hop separately, as in debugging a RAG system. If the relationships are known in advance, a graph can store them, which is the idea behind GraphRAG.
Interleaving or question decomposition?
Question decomposition is the main alternative to interleaving: the model splits the question into sub-questions and answers each with its own search. Self-Ask (Press et al., 2022) and DecomP (Khot et al., 2022) are decomposition methods, and Trivedi et al. (2022) discuss both alongside interleaving.
The difference is when the later queries get written. Decomposition plans the sub-questions up front, which is simple and lets independent sub-questions run side by side. Interleaving lets each step depend on the evidence found. Decomposition fails when a later sub-question cannot be written until an earlier one is answered: in the coaster example, "Which country is Mack Rides in?" can only be asked after the first passage reveals the manufacturer. For comparison questions, whose parts are independent, decomposition is a natural fit. In an interview, say which dependency your question has and choose from that.
How to answer it in an interview
How would you handle questions that need evidence from several documents?
A multi-hop question needs evidence from several documents, and the second search depends on what the first found, so one embedding of the original question is not enough. I retrieve a base set with the question, have the model write one reasoning sentence, and use that sentence as the next query. I repeat until the model can answer or I hit an iteration cap. The trade-offs are an extra model call and delay per hop, and errors that compound, so I cap hops, rerank each round and evaluate retrieval separately from the answer.
The spoken structure: say why one search fails, describe the loop, then name the cap, the cost and how you would measure it.
Common mistakes
- "I will raise top-k until the second document shows up." (wrong) The second query depends on the first answer. (right) "The next query is written from what the first hop found."
- "Multi-hop is just chain-of-thought." (wrong) Reasoning alone fails when the knowledge is missing or stale in the model. (right) "Reasoning and retrieval alternate, each guiding the other."
- "If the right documents are retrieved, the answer will be correct." (wrong) Retrieval and reasoning can each fall short. (right) "I score retrieval and the final answer separately."
- "More hops are always better." (wrong) Each hop adds delay and error risk, and loops are capped. (right) "I set a small cap and report when evidence is not found."
- "A high HotpotQA score shows my retriever works." (wrong) In the distractor setting the gold paragraphs are supplied, so retrieval is barely tested. (right) "I report the open-domain setting, or MultiHop-RAG, when I claim retrieval quality."
Follow-up questions
Why does one search miss the evidence for a multi-hop question?
One search embeds the question only, and the question never mentions the bridge fact, such as the manufacturer's name, that points to the second document. The passages nearest to the question are therefore about the first link only.
How many hops should a retrieval loop allow?
Allow as few as your questions need, usually a small cap. As the worked example shows, chances multiply and delays add with each hop, and the loop stops at a final answer or the cap. Pick the cap from measured per-hop success and the delay budget.
Is multi-hop retrieval just chain-of-thought?
No. Chain-of-thought alone relies on what the model already knows. In interleaved retrieval each reasoning sentence also triggers a search, and the results improve the next sentence.
What does the MultiHop-RAG benchmark show about multi-hop queries?
MultiHop-RAG is a RAG-specific benchmark of 2,556 multi-hop queries over a knowledge base of English news articles, in four types: inference, comparison, temporal and null. It scores retrieval with mean reciprocal rank at 10 (mean reciprocal rank (MRR)@10), mean average precision at 10 (mean average precision (MAP)@10) and hit rate at 10 and 4 (Hits@10, Hits@4), and scores answering by language models, finding both unsatisfactory. How to score such a system is covered in RAG evaluation.
How much does interleaving help in practice?
Trivedi et al. (2022) report that, using a large language model, it improves retrieval by up to 21 points and downstream QA by up to 15 points on four multi-step QA datasets, including HotpotQA, 2WikiMultihopQA and MuSiQue. Treat these as results on those datasets, not a promise for yours.
When is GraphRAG a better fit than a retrieval loop?
When the relationships between entities are known and stable, a graph can store the links so a query follows them directly. A loop is more flexible for unseen questions. More interview prompts are in RAG interview questions.
- A multi-hop query needs evidence from several documents, combined by reasoning.
- One search fails because the second query depends on what the first found.
- Interleaved retrieval alternates one reasoning sentence with one search, and ends at an answer or an iteration cap.
- Hops multiply risk and add delay, so keep the cap small and rerank each round.
- Score retrieval and the final answer separately.
Sources
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (arXiv 2401.15391) — arXiv (Tang and Yang), 27 Jan 2024
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (arXiv 2212.10509) — arXiv (Trivedi et al.), 20 Dec 2022
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering — ACL Anthology (Yang et al., EMNLP 2018), 25 Sep 2018
- Measuring and Narrowing the Compositionality Gap in Language Models (Self-Ask, arXiv 2210.03350) — arXiv (Press et al.), 7 Oct 2022
- ReAct: Synergizing Reasoning and Acting in Language Models (arXiv 2210.03629) — arXiv (Yao et al.), 6 Oct 2022
