What is the RAG take-home, and how do you do it well?

In this article

1. Before you start
2. Why interviewers ask this
3. The RAG take-home from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the interview process and loops path, alongside the refactor take-home.

Why interviewers ask this

A take-home shows what a live round cannot: how you scope, build, measure and explain over many hours, with no one watching. A RAG brief is a popular carrier because it has many places to cut corners, and the corners are visible in the submission. Candidates report briefs of very different size: one reported task left about twenty hours between setting the task and the review, and another was described as a build of a question-answering system for a large codebase with an evaluation suite and an application programming interface (API) endpoint. These are candidate reports, not a company's published format.

A weak submission is a demo: it answers three hand-picked questions and nothing is measured. A strong one shows what it retrieved, why the answer is supported, what it refuses to answer, and how you know. A Staff-level submission also states what the measurements do not cover. Formats move. Anthropic's take-home is a code-performance optimisation task, not a RAG brief, but its history shows the pattern: Anthropic's repository says its performance take-home went from four hours to two and was later given a different base after a newer model beat people on it.

The RAG take-home from first principles

Retrieval-augmented generation (RAG)

Retrieval-augmented generation (RAG) is a design that pairs a generator, a language model, with a searchable index of documents, so each answer is written from passages retrieved for that question instead of from the model's memory alone.

In plain words

Picture an open-book exam. The student cannot memorise the whole library. For each question they look up a few pages. They write the answer from those pages and note the page numbers. RAG is that student.

The analogy breaks in one place. The student knows when the book has nothing useful. A model often writes a confident answer anyway. Your system needs an explicit rule for "the documents do not say".

In one precise sentence: the original paper combines a pre-trained retriever with a pre-trained generator. The retriever uses Maximum Inner Product Search (MIPS), which finds the stored vectors with the largest dot product with the query vector. A take-home asks you to build a small, honest version of that and prove it works. The pipeline has four steps: an evaluation set written first, chunking and retrieval, an answer with citations and a fallback, and a score for retrieval and answer separately.

Step 1: Write the evaluation set before the pipeline

An evaluation set is a short list of questions with the answer each should get and the passage that supports it. Write it first, because it turns every later choice (chunk size, retriever, prompt) into something you can test instead of argue. Include three kinds of question: fully answerable, partly answerable, and unanswerable from the documents. Candidates report that a brief may supply only a small set, which is a starting point; yours should be larger.

Input: the supplied documents and the brief. Output: a file of questions, reference answers and source passages. Keep a few questions that need two passages joined, called multi-hop questions. Never tune on the whole set: hold some questions back, so a fix that helps one question does not quietly hurt the rest. You now know what "works" means, which the next steps can measure.

Step 2: Chunk the documents and retrieve the right passages

A chunk is a slice of a document, usually a few paragraphs, stored with its title and source so a citation can point back to it. Chunks are embedded and put in an index; at question time the question is embedded too and the closest chunks come back. Input: documents. Output: a ranked list of chunks per question.

One embedding search is a starting point, not a finish. Anthropic reports its baseline missed relevant chunks from the top twenty at a rate of 5.7%. Contextual embeddings cut the failure rate to 3.7%, adding contextual keyword search (Best Matching 25, or Best Matching 25 (BM25)) cut it to 2.9%, and with reranking the failures fall by 67% in total, to 1.9%.¹ Reranking is a second pass that re-orders a short list so the best chunks go first. Hybrid search combines an embedding search with a keyword search and merges the two result lists. Those figures come from Anthropic's own data, as of September 2024, so measure on your corpus instead of assuming they transfer.

Text description of the diagram

• Anthropic reports its baseline missed relevant chunks from the top twenty at a rate of 5.7%.
• Contextual embeddings cut the failure rate to 3.7%, adding contextual keyword search cut it to 2.9%, and with reranking the failures fall to 1.9%.

Step 3: Answer only from the passages, cite them and refuse when unsupported

The generator receives the question and the retrieved chunks and is told to answer only from them, to cite the chunk each claim came from, and to return a fixed fallback when the passages do not contain the answer. Input: a question and a few chunks. Output: an answer with citations, or the fallback.

Citation tracking is the reason the paper's design appeals: the authors note that retrieved knowledge can be inspected and revised, unlike knowledge buried in model weights. In practice, return the chunk text and source with each citation so a reviewer can check the claim in seconds. For a multi-hop question, such as one whose answer needs a fact from one document and a fact from another, retrieve again using what the first retrieval found, then cite both chunks. The fallback matters as much as the answer: the unanswerable questions in your set exist to test it.

Step 4: Score retrieval and the answer separately

Retrieval and generation fail for different reasons, so score them apart. Retrieval is judged by context recall (did the needed facts arrive?) and by context precision, which measures whether the useful chunks sit at the top; the Ragas documentation defines it by comparing each retrieved chunk with a reference answer, rank by rank. The answer is judged by faithfulness and by whether it addresses the question.

Take illustrative numbers. If a reference answer holds two facts and the retrieved chunks carry one, context recall is 0.5. If four chunks come back and the useful ones sit at ranks one and three, context precision is 5/6 (0.83); reranking them to ranks one and two lifts it to 1.0. An answer with three claims, two supported, has faithfulness 2/3 (0.67).¹ On an illustrative seven-question set with five passes, the pass rate is 5/7 (0.71), and the table shows where it fails.

Report the numbers, the failures and what you would try next.

1 · Evaluation set2 · Chunk and retrieve3 · Cited answer4 · Score both stagesReplay

Step-by-step description

1. Write the questions and expected answers first.
2. Index chunks and fetch the closest ones.
3. Answer from passages or return the fallback.
4. Measure retrieval and answer apart.

• A take-home pipeline has four steps: an evaluation set written first, chunking and retrieval, an answer with citations and a fallback, and a score for retrieval and answer separately.
• An evaluation set written first turns every later choice into something you can test.

The Staff-level answer

What makes it Staff-level: it begins with the consequence of a wrong answer, treats evaluation as the first deliverable, and separates what was measured from what is merely believed.

The rubric

Three variants of this question

Build question answering for a large codebase, with an evaluation suite and an endpoint

Reported by candidates as a build of a question-answering system for a large codebase. Chunk by function or file with its path attached, expose one endpoint that takes a question and returns an answer with file citations, and ship the evaluation suite as part of the repository.

Build a policy assistant with mandatory citations and safe fallbacks

Reported with a small evaluation set of answerable, partly answerable and unanswerable questions. Every answer needs a source; out-of-scope questions return a fixed refusal, and partly answerable ones should say which part the documents cover.

Answer multi-hop questions with citation tracking

Retrieve, read, and retrieve again with what you learned, then cite every passage used. Test with questions that cannot be answered from any single chunk.

Common mistakes

• "It answers well on my five test questions." (wrong) Five picked questions measure your taste. (right) "I wrote a labelled set first and report scores by question type."
• "Retrieval looked fine, so the answers are fine." (wrong) The generator can still ignore or invent. (right) "I score retrieval and faithfulness separately."
• "Embeddings alone are enough." (wrong) Anthropic's gains came from adding keyword search and reranking. (right) "I compared a plain embedding search with hybrid search on my set."
• "RAG means training the model on the documents." (wrong) Knowledge is revised by changing the index, not the weights. (right) "Updating a document means re-indexing it."
• "I used a model to grade answers, so they are verified." (wrong) A model used as a judge can be biased. (right) "I checked a sample of the judge's grades by hand."

Follow-up questions

Are take-home assignments still used?

Yes. Some employers still set take-homes, often for applied AI, machine learning and data roles, alongside live rounds. Formats keep changing: Anthropic's repository says its performance take-home was cut from four hours to two after Claude Opus 4 beat most humans on it, and the company later moved to a different base. Read the brief's time limit and tool rules closely.

How do you present an ML take-home?

Lead with the result and the evidence, then the code. A reviewer should understand your decisions without running anything. A four-line readme outline works:

• Problem and scope: what the system answers and what it leaves out.
• How to run it: the commands, in order.
• Evaluation results by category, with the failure cases.
• Limits and next steps. The project deep dive is the live version of this explanation.

How do you avoid data leakage in a take-home?

Keep the questions you tune on apart from the questions you report, and never let a reference answer appear in the indexed documents or the prompt. Say how you split them.

How should the system handle multi-hop questions?

A multi-hop question is one whose answer needs facts from two or more passages. Retrieve once, extract what is missing, retrieve again with it, and cite every passage used.

Can you use an AI assistant on a take-home?

Follow the brief's stated rules and ask the recruiter if they are silent. The AI-assisted coding interview page covers how such rounds are graded, and the AI engineer questions and applied AI engineer rounds pages show what follows a take-home. Lab-specific loops, such as the Anthropic process, are described on their own pages.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401, v4) — arXiv (Lewis et al., NeurIPS 2020), 12 Apr 2021
2. Contextual Retrieval in AI Systems — Anthropic Engineering, 19 Sep 2024
3. Anthropic's Original Performance Take-Home (README, read 2026-10-26) — Anthropic (GitHub repository README), 26 Oct 2026
4. Ragas documentation: Context Precision — Ragas, 26 Oct 2026
5. Ragas documentation: Faithfulness — Ragas, 26 Oct 2026
6. Ragas documentation: Context Recall — Ragas, 26 Oct 2026