Generative AI system design interview: the complete guide

In this article

1. The vocabulary of generative AI system design
2. What these rounds test
3. Start here: the learning path
4. The RAG pipeline end to end
5. Retrieval quality: hybrid search, reranking and context
6. Agentic RAG: when the model controls retrieval
7. Evaluating a RAG system
8. Capacity, latency and cost: a worked example
9. Access control and multi-tenant design
10. Product-shaped designs
11. Questions interviewers ask on generative AI system design
12. All 30 guides in GenAI system design
13. Follow-up questions
14. Sources

The vocabulary of generative AI system design

Generative AI system design

Generative AI system design is the practice of choosing the parts around a language model, such as the data it reads, how it retrieves, how answers are checked, and how cost, speed and access are controlled, so that a product built on it works reliably for many users at once.

Grounded answer

A grounded answer is a model answer that is built from retrieved passages and points back to them, so that a reader can check each claim against its source instead of trusting the model's memory.

Permission-aware retrieval

Permission-aware retrieval is a retrieval method that returns only the passages the asking user is allowed to see, by applying the user's access rights as a filter inside the search, before any passage reaches the model.

What these rounds test

A generative AI system design round tests whether you can turn a vague product request into a pipeline with named parts, numbers and failure modes. The interviewer rarely wants a favourite tool. They want to hear the requirements you clarified first, the order of the data flow, what you would measure, and what breaks at scale.

We group the answer into seven decisions. Say them in this order and you cover most of what a round probes: how documents are ingested and chunked, how they are indexed and retrieved, how results are reranked and grounded, whether the model controls retrieval, how you evaluate, how you control cost and latency, and how you keep each customer's data separate.

Text description of the diagram

• A generative AI system design answer walks seven decisions in order.
• The seven decisions are ingest and chunk, index and retrieve, rerank and ground, route and loop, evaluate, control cost and latency, and secure by tenant.

Start here: the learning path

Read the sections below in the order of the figure above: each covers one of the seven decisions, with the numbers and trade-offs an interviewer probes.

1. What is permission-aware RAG, and how do you build it?
2. How do you design ChatGPT / Claude.ai in an interview?
3. How do you design RAG over real documents with tables and charts?
4. How do you design a voice agent? Pipeline, latency and barge-in
5. GenAI system design interview walkthrough: design a Perplexity answer engine
6. What is agentic RAG? How it differs from standard RAG
7. RAG in a regulated bank: how to design it
8. Evaluating a RAG system stage by stage: what to measure at each step
9. How do you deploy LLMs for a regulated enterprise?
10. How do you make RAG fast and cheap?
11. How do you design Hippocratic-style clinical LLM agents?
12. How do you design an LLM document-processing pipeline?
13. How do you design an image-generation product end to end?
14. Do you even need RAG? How to decide in an interview
15. What is RAG and its usefulness?
16. Where RAG latency and cost go, stage by stage
17. How do you architect a multi-tenant LLM deployment with a tenant-aware gateway?
18. What are the different chunking strategies for RAG?
19. How do you evaluate a RAG system?
20. How would you make a project production-ready for a large user base?
21. Caching LLM calls: how prompt and semantic caching cut cost
22. How do you design an AI agent system? Loop, tools, memory, safety

The RAG pipeline end to end

A retrieval-augmented generation (RAG) pipeline has two halves: an indexing half that runs ahead of time and a query half that runs on every question. The original paper by Lewis et al. (revised April 2021) describes the pattern as combining a pre-trained generator, the parametric memory, with a pre-trained neural retriever over a dense vector index, the non-parametric memory (Lewis et al., 2020). In practice the indexing half reads documents, cuts them into chunks, turns each chunk into an embedding, and stores the vectors with the source text and metadata. The query half embeds the question, finds the nearest chunks, and gives them to the model with the question.

The first design question is the chunk. Cut by structure where you can, such as headings and paragraphs, keep a little overlap, and store the source and section with each chunk so that an answer can cite it. Embeddings and vector indexes are explained in embeddings and vector search.

Retrieval quality: hybrid search, reranking and context

Retrieval quality sets the ceiling of the whole system, because the model can only use what is retrieved. Three levers matter most. Combine meaning-based vector search with keyword search, which catches exact names and codes; the idea is covered in hybrid search. Rerank the top candidates with a stronger model, as in cross-encoder rerankers. And give each chunk its context.

Anthropic's contextual retrieval post (September 2024) prepends a short explanation to each chunk before it is embedded and indexed by keyword (BM25). It reports that this cut failed top-20 retrievals (cases where the relevant passage is not among the 20 results) by 49%, and by 67% when combined with reranking, on the data it tested (Anthropic, 2024). Treat that as a vendor result on its own data: it shows the direction, and you should measure on your own documents. The full walk-through is designing RAG over real documents.

Agentic RAG: when the model controls retrieval

Standard RAG runs one retrieve-then-generate pass. Agentic RAG lets the model decide whether to search, write the query, grade the results and retry, which helps on questions that need facts from several passages. The cost is more model calls on every question, so the design answer includes a router that sends easy questions around the loop and a cap on rounds.

The full walk-through, with a worked cost model, is in agentic RAG. For questions about whole-corpus themes rather than single facts, GraphRAG builds a knowledge graph over the documents, and for the general agent pattern see AI agents. Say in the interview which question types you would route where, and why.

Evaluating a RAG system

A RAG system is evaluated in parts, because a wrong answer can come from retrieval or from generation. Ragas, a framework for reference-free evaluation, scores whether retrieval finds focused context, whether the model stays faithful to it, and the quality of the answer, without requiring human-labelled answers for every case (Es et al., revised April 2025).

A strong answer names a small labelled question set from the real corpus, one metric per stage, and a rule for what blocks a release. Add production signals: traces of every step so a bad answer can be reconstructed, and a thumbs-down review queue. Related methods: Ragas, LLM evaluation and LLM observability. Evaluate each stage on its own: RAG evaluation.

Capacity, latency and cost: a worked example

A common prompt is to design a RAG system over ten million documents at a thousand queries per second (QPS) with a sub-second answer, which in a streamed answer conventionally means the time to the first token. Do the arithmetic aloud.

The inputs below are illustrative assumptions, stated so you can change them: ten chunks per document, 1,024-dimension embeddings, five retrieved chunks of 300 tokens per query, 200 further tokens for the question and instructions, an 800-millisecond budget for the time to first token (retrieval, reranking and reading the prompt; see LLM inference), 250 output tokens per answer (a token is a short piece of text, often part of a word), and a decode speed of 50 tokens per second for each answer being streamed.¹

Ten million documents at ten chunks each make 100 million vectors. At four bytes per number, the raw vectors take 409.6 gigabytes; at two bytes, 204.8; at one byte (int8), 102.4, before any index overhead, which is why compression is the first lever.¹ The answer does not finish at the first token. Writing 250 tokens at 50 a second takes 5 seconds, so each query spends 800 milliseconds plus 5 seconds, which is 5.8 seconds, in the system.¹

The number of requests in flight, called answers in flight, equals the arrival rate times the average time in the system (Little's law): 1,000 per second times 5.8 seconds is 5,800, of which about 5,000 are writing at once and about 800 are still waiting for their first token.¹ Each query sends 1,700 tokens to the model, so the model layer reads 1.7 million tokens per second and writes 250,000 per second.¹

Text description of the diagram

• Ten million documents at ten chunks each make 100 million vectors.
• At 1,024 dimensions, the raw vectors take 409.6 GB in float32, 204.8 GB in float16 and 102.4 GB in int8.

The levers follow from the numbers: stream the answer so the user sees text at the first token, ask for shorter answers (output tokens set the time in system), cache answers to repeated questions, retrieve fewer and shorter chunks, route easy questions to a smaller model, and keep the per-token serving cost in view, as in LLM inference and the key-value (KV) cache. One more rule from the long-context literature: on the models tested in 2023, performance was often highest when the relevant information was at the start or end of a long input and lower in the middle (Liu et al., 2023), so more chunks is not always better.

Access control and multi-tenant design

Access control is the question that separates a demo from a product. The rule is to filter inside the retrieval step: the user's rights go into the search as a filter, so a passage they may not see never reaches the prompt. Filtering the top results afterwards also keeps forbidden text out of the prompt, but it can leave the list short of results, so the answer loses recall. The Open Worldwide Application Security Project (OWASP) GenAI Security Project's entry on vector and embedding weaknesses (2025) recommends "permission-aware vector and embedding stores" and "strict logical and access partitioning of datasets in the vector database" (OWASP, 2025).

In an answer, name the two common layouts. One index with a tenant or group field on every chunk and a mandatory filter, which is cheaper and needs discipline. Or one index per tenant, which isolates harder and costs more as tenants multiply. Mention that access rights change, so the permissions on a chunk must be re-synced when a document's sharing changes.

Product-shaped designs

Interviewers often wrap the pipeline in a product: a chat assistant, a voice agent, an answer engine over the web, a document-processing pipeline, or an image-generation product. For the products that answer from documents the pipeline above applies and what changes is the constraint that dominates; an image-generation product is not a retrieval pipeline, so only the evaluation and cost-and-latency decisions carry over, safety filtering is added, and serving and graphics processing unit (GPU) cost per image dominate.

A voice agent is dominated by the time budget of each turn, an answer engine by freshness and citation, a document pipeline by extraction accuracy, and a regulated setting by what the system must refuse to say. Start every such answer by naming the dominant constraint, then reuse the decisions that apply. For the general method of a design round, see machine learning (ML) system design and system design.

Questions interviewers ask on generative AI system design

Follow-up questions

How to design a RAG system?

Clarify the users, the documents and the quality bar, then walk the pipeline: ingest and chunk, embed and index, retrieve and rerank, generate with citations, and evaluate. Close with the cost, latency and access-control numbers. The architecture is detailed in RAG architecture.

How to design a production-ready RAG system architecture, and what are the trade-offs?

The trade-offs are chunk size against precision, vector against keyword search, reranking quality against latency, and loop depth against cost. Say which you chose for the stated requirements and what you would measure to change your mind. See RAG architecture.

How would you design a production RAG system over 10 million documents at about 1,000 queries per second?

Compute the sizes first: under the assumptions of the worked example above, that is 100 million vectors, 5,800 answers in flight (800 milliseconds to first token plus 5 seconds to write 250 tokens, at 1,000 queries per second) and 1.7 million input tokens and 250,000 output tokens a second. Then choose compression, caching, streaming and a router to bring each down; serving cost is covered in LLM inference.

Do I need RAG, or can I put everything in the prompt?

If the knowledge base is small, you may not need RAG. Anthropic's post says a knowledge base under 200,000 tokens, about 500 pages, could be included in the prompt directly, as of September 2024 (Anthropic, 2024); beyond that, retrieval earns its cost. Also compare against fine-tuning, which trains the model on examples instead of handing it documents at question time.

How do you stop one customer's documents leaking to another?

Apply the user's rights as a filter inside the search so that forbidden passages never reach the prompt, partition the data by tenant, and test with a question that only the other tenant can answer.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401 v4) — arXiv (Lewis et al.), 12 Apr 2021
2. Introducing Contextual Retrieval — Anthropic, 19 Sep 2024
3. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172 v3) — arXiv (Liu et al.), 20 Nov 2023
4. Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv 2309.15217 v2) — arXiv (Es et al.), 28 Apr 2025
5. LLM08:2025 Vector and Embedding Weaknesses — OWASP GenAI Security Project, 28 Apr 2025