RAG architecture diagram: every component and what it does

In this article

1. Before you start
2. Why interviewers ask this
3. The RAG architecture from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the retrieval-augmented generation learning path.

Why interviewers ask this

A language model only knows what was in its training data, so any product that answers from private or recent documents needs retrieval, and "draw the architecture" is the usual way to test whether a candidate has built one. A weak answer draws a box called "vector database" between the user and the model.

A strong answer separates the work done once, when documents are indexed, from the work done on every question, and says what can fail in each stage. A Staff answer adds the numbers: how many chunks, how big the index is, how many tokens the prompt gets, and where quality is lost. The question links to embeddings and vector search and, in system design rounds, to agentic RAG.

The RAG architecture from first principles

RAG

Retrieval-augmented generation (RAG) is a way of answering questions in which a system first retrieves passages relevant to the question from a document collection and then has a language model write the answer from those passages.

Chunk

A chunk is a piece of a source document, usually a few hundred tokens long, that is embedded and stored as one retrievable unit.

In plain words

Think of a reference librarian. Before visitors come, the librarian cuts books into cards and files them by topic (indexing). When a visitor asks a question, the librarian pulls the closest cards and hands them to a writer, who composes the reply (querying). The analogy breaks here: filing is by meaning, not title, so a card about "cancelling a plan" can match a question about "getting my money back" with no shared words.

In one sentence: RAG keeps knowledge in an index outside the model and brings the relevant part into the prompt at question time. Lewis et al. (2020) named the idea. Their system paired knowledge in a model's weights (parametric memory) with a searchable index of Wikipedia passages (non-parametric memory). A learned retriever connects the two.

Step 1: Split the documents into chunks

Indexing starts by cutting each document into chunks, because the model and the index both work on small units. A chunk of 512 tokens is a common size, and an overlap of 20% repeats the end of one chunk at the start of the next so a sentence cut at a border is not lost (both values are illustrative choices, not a standard).

Anthropic notes that a knowledge base under 200,000 tokens, about 500 pages, can simply go into the prompt without RAG (Anthropic, 2024). Cut into chunks of 512 tokens, that same 200,000 tokens becomes 391 chunks with no overlap or 488 with 20% overlap.¹ Input: documents. Output: a numbered list of chunks, each keeping a pointer to its source. The chunk boundary is the first place quality is decided, because a fact split across two chunks may never be retrieved whole.

Step 2: Embed every chunk

An embedding model turns each chunk into a vector, a fixed-length list of numbers, so that chunks with similar meaning get nearby vectors. OpenAI's embedding models return 1,536 numbers for text-embedding-3-small and 3,072 for text-embedding-3-large, and accept up to 8,192 input tokens each (OpenAI embeddings guide, as of October 2026).

Karpukhin et al. (2020) showed that this dense retrieval works on its own: a dual encoder, one encoder for questions and one for passages trained together so that matching pairs score high, beat a strong keyword baseline, Best Matching 25 (BM25), by 9% to 19% absolute in top-20 passage accuracy on their datasets. Input: chunk text. Output: one vector per chunk. Questions must later land in the same vector space, through the same model or its jointly trained question encoder, or the distances mean nothing.

Step 3: Store the vectors in an index

The vectors go into a vector index so that nearest-neighbour search is fast. Comparing a question to every vector would cost time proportional to the number of chunks, so indexes use approximate nearest-neighbour search, which can occasionally miss a true nearest vector in exchange for speed. A widely used method is the Hierarchical Navigable Small World (HNSW) graph of Malkov and Yashunin (2016), a multi-layer graph of links between near vectors that lets a search start coarse at the top layer and refine below, scaling logarithmically in their analysis.

Storage is easy to count: 1 million chunks of 1,536 numbers at 4 bytes each is 6.14 GB of raw vectors, and 10 million is 61.4 GB, before the index's own links.¹ Input: vectors and chunk ids. Output: an index that returns the nearest ids. Indexing is done: everything so far runs once, ahead of questions.

Text description of the diagram

• 1 million chunks of 1,536 numbers at 4 bytes each is 6.14 GB of raw vectors.
• 10 million chunks need 61.4 GB before the index's own links.

Step 4: Embed the question and retrieve the top k

The query pipeline embeds the question into the same vector space and asks the index for the top-k chunks. Top-k is the set of the k stored vectors nearest to the question vector. Nearness is usually measured by cosine similarity. Cosine similarity is a score that compares the direction of two vectors and ignores their length, scoring 1 for the same direction.

A toy example with 3-number vectors (illustrative) shows the idea: for a question vector of (0.9, 0.1, 0.2), the chunk "refund policy" scores 0.987, "cancel a plan" 0.956, "holiday dates" 0.434 and "office address" 0.256, so the top 2 are the refund and cancellation chunks.¹ Input: a question. Output: k chunk ids with scores. Keyword search can run beside this step; the combination is hybrid search.

Step 5: Rerank and build the prompt

The first retrieval is fast and rough, so many systems retrieve a wider set, rerank it with a reranker, and keep the best few. A reranker reads the question and a chunk together to judge relevance; see cross-encoder rerankers. Anthropic's test first enriched indexing: contextual embeddings add a short note on what each chunk is about before embedding it, and contextual keyword search indexes that enriched text. Its reranker then sorted the top 150 chunks into a top 20.

Failed top-20 retrievals were 5.7% for a baseline, 3.7% with contextual embeddings, 2.9% adding contextual keyword search and 1.9% adding reranking, reductions of 35%, 49% and 67% (their test set; yours will differ).¹ The kept chunks, plus the question and instructions, form the prompt. Five chunks of 512 tokens add 2,560 tokens; twenty add 10,240.¹ Input: k chunks. Output: one prompt. Order matters: Liu et al. (2023) found accuracy is typically highest when the relevant passage is at the start or end of the input and falls when it is in the middle.

Text description of the diagram

• Failed top-20 retrievals were 5.7% for the baseline and 1.9% with contextual embeddings, keyword search and reranking.
• The reductions were 35%, 49% and 67%.

Step 6: Generate the answer from the chunks

The language model writes the answer from the prompt, and current systems usually instruct it to answer only from the supplied chunks and to cite the chunk ids. That instruction is a design choice, not part of the original paper: the RAG of Lewis et al. (2020) trained a sequence-to-sequence generator together with its retriever, and the abstract reports more specific, diverse and factual text than a model using only its stored parameters, plus state-of-the-art results on three open-domain question-answering tasks. Input: the prompt. Output: an answer with source ids that a user can check. The model can still ignore or misread the chunks, so a complete design also evaluates retrieval and generation separately, as RAG evaluation explains.

1 · Chunk2 · Embed3 · Index4 · Retrieve5 · Rerank6 · GenerateReplay

Step-by-step description

1. Documents become chunks.
2. Each chunk becomes a vector.
3. Vectors go into an index.
4. The question, embedded the same way, finds the top-k chunks.
5. The best few build the prompt.
6. The model answers with citations.

• Indexing runs once and has three steps: chunk, embed, store.
• The query pipeline embeds the question into the same vector space, retrieves the top-k chunks, reranks them and generates the answer.

Worked example

We sized a small system with the numbers above.¹ The knowledge base is 200,000 tokens (illustrative, from Anthropic's 500-page figure), split into chunks of 512 tokens with 20% overlap, which gives a stride of 410 tokens and 488 chunks. With text-embedding-3-small each chunk becomes 1,536 numbers, so the raw vectors take 488 × 1,536 × 4 bytes, about 3 MB: a collection this small barely needs an index. At 1 million chunks the raw vectors take 6.14 GB, and at 10 million 61.4 GB; the larger model with 3,072 numbers doubles each figure.

The prompt rows show what retrieving 20 and keeping 5 buys: the model reads a quarter of the text, and the relevant chunk is less likely to sit in the middle of a long input, provided the reranker keeps it.

Trade-offs and edge cases

Chunk size trades precision for context: small chunks match a question tightly but may lose the surrounding facts, and large chunks keep context but add noise and tokens. The value of k trades recall, the chance that the right chunk is among those retrieved, for prompt cost, as the table shows. Reranking trades extra latency for accuracy: in Anthropic's reported test it took failed retrievals from 2.9% to 1.9%, but whether that is worth the wait depends on your latency budget, which is a product decision.

Three edge cases are worth saying aloud. First, documents change, so the index needs a re-indexing path for updated and deleted documents, or answers cite stale text. Second, who may see which document must be enforced at retrieval time, not after generation. Third, questions that need several hops across documents are where plain RAG struggles and agentic RAG or graph RAG are tried. Reported practice differs on how much the order of chunks matters for current long-context models, since the Liu et al. study covers the models of 2023, so test your own model.

How to answer it in an interview

Say it in this order: the two pipelines, the six steps, then where quality is lost.

Common mistakes

• "RAG is a vector database plus a model" (wrong). It leaves out chunking, reranking, prompt building and evaluation, where most quality is lost. (Right: "RAG is an indexing pipeline and a query pipeline; the index is one box in it.")
• "I would embed questions and documents with different models" (wrong). Distances between vectors from different models are meaningless. (Right: "the same embedding model, or a matched pair trained together, for both.")
• "Retrieve more chunks to be safe" (wrong). More chunks add tokens and can bury the answer mid-prompt. (Right: "retrieve wide, rerank, and keep a few.")
• "RAG removes hallucinations" (wrong). A hallucination is a confident statement the sources do not support, and the model can still misread or ignore the chunks. (Right: "RAG grounds the answer; I still check faithfulness to the chunks.")
• "Always use RAG" (wrong). A collection under about 200,000 tokens can go straight into the prompt. (Right: "I choose RAG when the corpus is too large for the prompt, or too costly to send in full on every question.")

Follow-up questions

What is the role of the retriever in a RAG architecture?

The retriever turns the question into a vector, finds the nearest chunks in the index and returns them with scores, so it decides what the model can see. If the retriever misses the passage, no prompt and no model can recover the fact.

How does reranking work in RAG?

A fast first retriever returns a wide candidate set, a reranker scores each question and chunk pair by reading them together, and the system keeps the highest-scoring few. In Anthropic's reported test, reranking 150 candidates into a top 20 cut failed top-20 retrievals from 2.9% to 1.9%, so measure the gain against the extra delay.

What are the different chunking strategies for RAG?

Fixed-size chunks with overlap are the simplest; other common choices split on sentences or paragraphs, follow the document's headings, or group sentences by meaning. Whichever you choose, measure retrieval failures, because the chunk boundary decides what can be retrieved whole.

When should you skip RAG and put everything in the prompt?

When the knowledge base is under about 200,000 tokens, the figure Anthropic gives. Beyond that, or when sending every token on every question costs too much, retrieval is needed.

Why does the order of retrieved chunks matter?

Language models in the Liu et al. study used passages at the start or end of the input best and the middle worst. Put the highest-ranked chunks first, or first and last.

What changes if the corpus grows from 1 million to 10 million chunks?

Raw vector storage grows tenfold, from 6.14 GB to 61.4 GB at 1,536 numbers per vector, so memory, build time and the choice of approximate index matter more; the large language model inference side of the system is unchanged.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv (Lewis et al.), NeurIPS 2020, v4 revised 2021-04-12, 22 May 2020
2. Dense Passage Retrieval for Open-Domain Question Answering — arXiv (Karpukhin et al.), 10 Apr 2020
3. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — arXiv (Malkov and Yashunin), 30 Mar 2016
4. Lost in the Middle: How Language Models Use Long Contexts — arXiv (Liu et al.), 6 Jul 2023
5. Introducing Contextual Retrieval — Anthropic, 19 Sep 2024
6. Vector embeddings guide (text-embedding-3-small and -large) — OpenAI (date = read date), 24 Oct 2026