Do you even need RAG? How to decide in an interview

In this article

1. Before you start
2. Why interviewers ask this
3. RAG decision from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits inside the generative AI system design interview guide, which covers the neighboring designs.

Why interviewers ask this

Candidates report this as a scoping question inside a RAG design prompt: a small set of one-page documents, such as invoices, and a request to design retrieval over them. The consequence of reflex is concrete. A candidate who starts drawing a chunker, an embedding model and a vector database for a corpus that fits in one prompt has skipped the decision the interviewer came to hear.

The question separates people who size the problem first from people who apply a template. A weak answer says "RAG always, because the model does not know our data." A strong answer measures the corpus in tokens, says what the questions need, and picks the simplest design that works. A Staff-level answer adds what breaks in each option: uneven recall in long prompts, exact-value questions in semantic search, and cost per question as volume grows. The bank of candidate-reported questions also pairs this scenario with "when would you choose RAG over fine-tuning", which is answered below.

RAG decision from first principles

Context stuffing

Context stuffing is the practice of putting the whole document set into the prompt of every request, so the model reads everything and no retrieval step exists.

Top-k retrieval

Top-k retrieval is a search step that scores every stored chunk against the question and returns the k highest-scoring chunks, however high or low the scores are.

Similarity threshold

A similarity threshold is a fixed score cutoff, such as 0.7, below which a retrieved chunk is dropped, so the number of results varies with the question.

In plain words

Imagine a manager who must answer questions about a filing cabinet. If the cabinet is one thin folder, the manager reads it before each question. If it is a warehouse, the manager sends an assistant to pull five folders first. That assistant is RAG. The analogy breaks here: a real assistant understands the question, while the retrieval step only compares numbers, so it can fetch the wrong folder and the manager will not know.

The context window is the maximum amount of text, counted in tokens, that a model reads in one request. Cosine similarity is a score between two embeddings that is higher when they point in the same direction. RAG is needed in only three cases. The corpus does not fit in the context window. Or the cost or delay of resending it on every question is too high. Or the answer needs freshness or traceable sources that retrieval supplies.

1 · Measure2 · Questions3 · Cost4 · ChooseReplay

Step-by-step description

1. Compare corpus tokens with the window.
2. Lookup, summary or total?
3. Tokens read per question.
4. Three outcomes, simplest first.

• A corpus under about 200,000 tokens fits in one prompt.
• Questions that need exact totals need a query over extracted fields.
• Full context resends the whole corpus on every question.
• Choose the simplest design that passes.

Step 1: Measure the corpus against the context window

Corpus size in tokens is pages times tokens per page, and the comparison to the context window is the first decision. Anthropic's September 2024 guidance of about 200,000 tokens for roughly 500 pages implies about 400 tokens a page.¹ One-page invoices are assumed here at 400 to 600 tokens each (illustrative), so a hundred of them is 40,000 to 60,000 tokens, which is 20% to 30% of a window that size.¹ A window of that size holds roughly 333 to 500 such invoices.¹ Windows have grown since 2024: as of October 2026 some models accept far more than 200,000 tokens, so read your model's limit in the vendor's context-window documentation (read October 2026).

Check three things. The window of your model, as of today, since sizes change by model and version. The room you need for instructions and the answer. And whether scanned pages convert to text at a size you expect. You now know whether stuffing is physically possible; the next step asks whether it answers the questions.

Step 2: Check what the questions need

Questions fall into three kinds. A lookup (the due date on one vendor's invoice) reads one document. A summary or comparison reads a few. An aggregate (the total across all March invoices) reads every document and adds numbers.

Aggregates break both options. Retrieval returns only k chunks, so a total over a hundred invoices silently covers k of them. Context stuffing includes everything, but Liu et al. (2023) report that models use information in the middle of a long prompt less reliably than at the edges. Separately, as judgment rather than a finding of that paper, a model adding many figures in its head can drop one.

Instead, extract fields such as vendor, date and amount into a table once, answer totals with a database query, and keep the model for wording. Anthropic (2024) notes that an embedding model can find content about error codes in general but miss one exact error-code string, which keyword search finds; for invoice numbers and vendor identifiers, combine both.

Step 3: Count the cost per question

Context stuffing resends the whole corpus on every question, so cost grows with corpus size times question volume. At 60,000 tokens a question and 1,000 questions, that is 60,000,000 tokens read.¹ A RAG prompt of five retrieved chunks at 500 tokens each plus 300 tokens of instructions is 2,800 tokens (illustrative), so 1,000 questions read 2,800,000 tokens, about 21 times fewer.¹ Embedding tokens are priced differently from model input tokens, so the 60,000 tokens embedded once are counted separately and are not in that ratio.¹

Prompt caching changes the picture. It bills a repeated prefix at a reduced rate for a fixed time; the prompt-caching documentation gives the current multiplier and duration. A cached prefix at an illustrative 10% rate counts as the equivalent of 6,000,000 full-price tokens over the 1,000 questions, about 2.1 times the RAG prompts, ignoring any cache-write surcharge.¹ That is the 2025 to 2026 argument for stuffing a small, stable corpus; see making RAG fast and cheap.

Retrieval wins on cost at high volume or large corpora, not at a hundred documents with a few questions.

Text description of the diagram

• Full context reads 60,000,000 tokens over 1,000 questions.
• RAG prompts read 2,800,000 tokens over 1,000 questions.
• A cached prefix at an illustrative 10% rate counts as the equivalent of 6,000,000 tokens.

Step 4: Choose the simplest design that passes

Pick the first design that survives the three checks. If the corpus fits, questions are lookups or summaries, and volume is modest, use context stuffing. If questions are aggregates, extract fields into a table and query it. If the corpus is larger than the window, or volume makes the cost above unreasonable, Anthropic notes that RAG is the typical solution: split documents into chunks, embed them, and add the most relevant chunks to the prompt. At 100,000 documents of about 500 tokens each, the corpus is 50,000,000 tokens: 250 windows of 200,000 tokens, or still 50 windows of an illustrative 1,000,000-token window, so stuffing is impossible at either size.¹

Retrieval also brings what Lewis et al. (2020) describe: a separate, searchable memory next to the model, which you can update by changing the index instead of the model. For real-world pages with tables and scans, see designing RAG over real documents.

The Staff-level answer

What makes it Staff-level: it decides from measured size and question type, names the aggregate failure, and states the condition under which the answer changes.

The rubric

Three variants of this question

When would you choose RAG over fine-tuning?

Choose RAG when the model needs facts that change or must be cited, and choose fine-tuning when it needs a consistent behavior, tone or format. The two combine: a fine-tune shapes how the model answers, and retrieval supplies what it answers from. Updating a retrieval index is cheaper than retraining a model every time a document changes.

Why not use cosine similarity with a 0.7 threshold instead of k-nearest neighbors?

Cosine similarity is the score inside the search, not an alternative to it. A similarity threshold is fragile because absolute scores are not comparable across embedding models: with illustrative scores, a 0.7 cutoff returns three chunks from one model and none from another, while top-k returns three from both.¹ Top-k, tuned with a recall test on real questions, is the safer default.

If you extract text from an invoice with OCR, do you see any issue?

Yes. Optical character recognition (OCR) turns a scanned image into text, and invoices stress it: tables flatten into a stream of words, columns merge, and a misread digit changes an amount silently. A strong answer validates extracted fields (line items sum to the total, dates parse) and keeps the page image for review; the page on RAG over real documents covers tables and charts.

Common mistakes

• "I would build a vector database first." (wrong) It skips sizing. (right) "I would measure the corpus in tokens; if it fits, I pass it whole."
• "Long context makes retrieval obsolete." (wrong) Cost grows with every resend and recall varies by position. (right) "Long context removes retrieval for small corpora, not for large or high-volume ones."
• "RAG will total the invoices." (wrong) Retrieval returns k chunks, not all. (right) "I extract amounts into a table and run a query."
• "RAG or fine-tuning, pick one." (wrong) They solve different problems. (right) "RAG supplies facts, fine-tuning shapes behavior, and I can use both."

Follow-up questions

Do 100 one-page documents even need MapReduce, or RAG at all?

No. A map-reduce summary calls the model once per document and then merges the results, which is built for corpora too big for one prompt. A hundred one-page documents fit in one prompt, so a direct call is simpler and has fewer failure points.

Does a long context window make retrieval obsolete?

No. It moves the threshold. Cost per question grows with corpus size, and Liu et al. (2023) report uneven use of information across a long prompt, so large corpora still need retrieval.

How do you walk through k-nearest-neighbor retrieval over chunks?

Embed the question with the same model used for the chunks, score it against every chunk with cosine similarity, sort, and return the top k. Choose k by testing recall on real questions, not by a fixed score cutoff. For 100 chunks, exact k-nearest-neighbor (KNN) search is trivial: 76,800 multiply-adds for 768-number vectors, against 768,000,000 for a million.¹ Approximate nearest-neighbor (ANN) indexes trade a little recall for speed and matter at that larger scale.

What changes when the corpus grows to 100,000 documents?

Stuffing stops being possible, and you need chunking, an index, keyword plus semantic (hybrid) search for exact identifiers, reranking and evaluation of retrieval quality; see evaluating a RAG system stage by stage. Questions that need several steps of searching lead to agentic RAG, and a production design is walked through in Design Perplexity.

Where does this decision live in a platform?

Routing between direct prompts and retrieval is often a gateway concern for a large language model (LLM) platform; see the LLM gateway interview questions. Spoken systems add latency limits, covered in how to design a voice agent.

Sources

1. Contextual Retrieval in AI systems — Anthropic, 19 Sep 2024
2. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401) — arXiv (Lewis et al.), 22 May 2020
3. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — arXiv (Liu et al.), 6 Jul 2023
4. Context windows (Anthropic documentation) — Anthropic, 26 Oct 2026
5. Prompt caching (Anthropic documentation) — Anthropic, 26 Oct 2026