How do you make RAG fast and cheap?

In this article

1. Before you start
2. Why interviewers ask this
3. Making RAG fast and cheap from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the generative AI system design interview guide. It assumes a RAG system already works and asks how to speed it up and cut its bill.

Why interviewers ask this

A slow or costly RAG endpoint loses users and budget, so interviewers ask how you would fix one. Candidates report versions of the question: optimise the latency of a RAG pipeline, take an endpoint that returns in 1.5 seconds and get it under 100 milliseconds, and cache model responses to cut cost.

A weak answer lists tricks: "use a faster model, add a cache." A strong answer starts by measuring, names which stage each trick helps, and says what each one costs. A Staff-level answer separates time to first token from total time, separates caching the prompt from caching the answer, and says which money is spent once at indexing and which on every query. That separation is what the rest of this page builds, one step at a time.

Making RAG fast and cheap from first principles

Prompt caching

Prompt caching is a provider feature that stores the processed beginning of a prompt, so a later request that starts with the same text skips that work. It lowers both the wait for the first token and the price of the repeated input tokens, while the model still writes a fresh answer.

Answer cache

An answer cache is a store that keeps a finished reply against the question that produced it, so a repeated or near-identical question returns the saved reply without calling the model. It saves the most cost and time, and it risks serving an out-of-date answer.

In plain words

Think of a restaurant kitchen. Prepping vegetables before service is work done at indexing time. Keeping the standard sauce ready is the prompt cache. Reusing the dish you cooked for the last table is the answer cache. The analogy breaks in one place: a sauce stays good all evening, while a cache entry expires after a set time and a saved answer can go stale when the documents change.

In one precise sentence, making RAG fast means shrinking the stage times that add up before the first token, and making it cheap means paying for fewer input tokens and fewer model calls.

Step 1: Measure the time and tokens of every stage

A RAG request runs in stages. The system embeds the question, searches an index, optionally reranks the hits, builds the prompt, waits for the model's first token, then streams the rest. A reranker is a second, slower model that re-scores the search hits so the best few go into the prompt.

Input: one real request. Output: a time and a token count for each stage. Without these two lists you are guessing, and the fix for a slow search is not the fix for a slow model. The worked example below uses an illustrative budget to show how the stages add up.¹ You now know where the time goes, so the next steps remove repeated work. Chunking and parsing choices also set what the index holds; RAG over real documents with tables and charts covers them.

Step 2: Put static text first so the prompt cache hits

Prompt caching works on the beginning of the prompt: a request reuses the cache only if its start matches the stored text. In a RAG prompt the system instructions and tool definitions repeat on every call, while the retrieved chunks and the user question change. Put the repeated text first and the changing text last.

Anthropic's cache has a default time to live of five minutes that refreshes on each use, and writes cost 1.25 times the base input price for five minutes or 2 times for one hour. Cache reads cost 0.1 times the base price, or less on some newer models (0.05 times on Claude Opus 5.5 and Claude Sonnet 5.5), according to the same documentation.

OpenAI's caching is automatic for prompts longer than 1,024 tokens, and the 201 guide reports requests 7% faster at 1,024 tokens and 67% faster time to first token at 150,000 tokens or more. Anthropic lists a minimum cacheable length per model, so a short prefix may not cache at all. Mark the cache point after the static text; put it after the chunks and every request pays the write premium and never gets a read.¹

Step 3: Move work to indexing time and trim the query path

Work done once when documents are indexed costs far less than work repeated on every query. Contextual retrieval is a method that has a model write a short note about the source document and attaches it to each chunk before the chunk is embedded and indexed. The contextualisation happens once at ingestion, not during every query.

Anthropic reports that the method cuts failed retrievals by 49% when the notes are added to both the embeddings and the keyword index (a keyword ranking function called Best Matching 25 (BM25)), and by 67% with reranking added. With prompt caching the same post puts the one-off cost, as of September 2024, at 1.02 US dollars per million document tokens under its stated assumptions. For an illustrative corpus of 10,000,000 tokens that is 10.20 US dollars.¹

On the query path, approximate nearest-neighbour (ANN) search returns vectors that are close, though not always the closest, in exchange for speed. Send the reranker a shorter list, and check recall after each cut.

Step 4: Cache answers and serve more requests per GPU

Two levers remain. The first is the answer cache: a repeated question returns the saved reply with no model call at all. Give each entry a freshness limit, key it by anything that changes the answer (user, permissions, document version) and clear it when documents change. A semantic cache, which matches near-duplicate questions by embedding similarity, also fits; a loose threshold returns another question's answer. A shared LLM gateway lets every product reuse it.

The second lever applies when you host the model. The key-value (KV) cache is the store of attention keys and values, called key-value (KV) pairs, for tokens already read, so the model does not recompute them for each new token. A KV cache is called "KV cache" because it holds those pairs. Kwon et al. (2023) note that wasted key-value cache memory limits the batch size, and their PagedAttention design in the vLLM serving system reports 2-4× higher throughput at the same latency than FasterTransformer and Orca. More requests per graphics processing unit (GPU) lowers cost per request, but it does not speed up one request.

1 · Measure2 · Cache the prefix3 · Index-time work4 · Answers and batchingReplay

Step-by-step description

1. Time and count tokens for each stage.
2. Static text first, chunks last.
3. Pay once when documents are indexed.
4. Skip repeated calls, serve more per GPU.

• Measure each stage before changing anything.
• Put static prompt text first so the cache hits.
• Move work to indexing time.
• Cache answers where it is safe and batch requests.

Worked example

All figures in this section are illustrative inputs or results computed from them, and the stage times are not measurements of any real system.¹ Take a naive endpoint that returns in 1,500 milliseconds: 60 to embed the question, 120 to search, 350 to rerank, 700 for the model's first token and 270 to finish the answer. The wait before the first token is 1,230 milliseconds.

After tuning (a cached query vector, an approximate index, a shorter rerank list and a cached prefix), the stages become 11, 17, 24 and 45 milliseconds: a first token in 97 milliseconds, 92.1% below the naive 1,230. The answer still takes 270 milliseconds to stream, so the full reply takes 367 milliseconds. A reply that returns whole in under 100 milliseconds is possible only from an answer cache.¹

Text description of the diagram

• Illustrative time to first token is 1,230 milliseconds for the naive pipeline.
• Illustrative time to first token is 97 milliseconds once each stage is tuned.

Now the cache bill. Assume a 2,000-token static prefix and 1,600 tokens of retrieved chunks and question (illustrative), and assume cache reads cost 0.1 times the base price (the Anthropic documentation gives 0.1 for most models, lower on some newer ones: 0.05 on Claude Opus 5.5 and Claude Sonnet 5.5) and a model whose minimum cacheable prefix is below 2,000 tokens. Over 10 requests inside the five-minute window, the uncached cost is 36,000 base-price token units. With the cache point after the static prefix it is 20,300, or 43.6% less. With the cache point after the changing chunks it is 45,000, which is 25% more than no caching.¹

Text description of the diagram

• Two calls cost 1.35 times the base price with a five-minute cache against 2 uncached.
• Three calls cost 2.2 with a one-hour cache against 3 uncached.

Trade-offs and edge cases

Every lever buys speed or cost with something else.

Two levers act on the model call itself. Output length sets both generation time and output price, so cap it with a maximum-token limit and ask for concise answers. Model routing sends simple questions to a smaller, faster model and escalates only hard ones; the risk is a hard question misjudged as easy.

Three edge cases matter. First, a cache that expires between requests: on Anthropic you pay the write premium again and get no reads, so low-traffic endpoints can lose money. Second, reranking: it is part of the 67% failure reduction in the contextual retrieval results above, so removing it to save time has a quality price, and you must measure it.

Third, loops: an agent that retrieves several times repeats every stage, so latency multiplies; agentic RAG covers when that is worth it. Long-context models with prompt caching can replace retrieval for a small, stable corpus (see do you even need RAG); test answer quality and cost at your corpus size before dropping retrieval. The wider design context sits in the GenAI system design guide, and the full learning library covers neighbouring topics.

How to answer it in an interview

The spoken structure: measure, cache the prefix, move work to indexing, cache answers with a freshness limit, then name the trade-off.

Common mistakes

• "Prompt caching is the same as caching answers." (wrong) Prompt caching reuses the processed prompt start and still generates a fresh answer. (right) "Prompt caching cuts input cost and first-token time; an answer cache skips the model."
• "Caching always saves money." (wrong) On Anthropic a write costs 1.25 or 2 times the base input price, so a prefix that is never reused costs more; OpenAI charges no write fee. (right) "On Anthropic, caching pays when later calls reuse the prefix inside its time to live."
• "I'll put the retrieved chunks at the top of the prompt." (wrong) Changing text first leaves no shared beginning to cache. (right) "Static instructions first, chunks and question last."
• "Contextual retrieval slows every query." (wrong) The contextualising runs once at ingestion. (right) "It is an indexing cost; queries pay only for the retrieval steps they already run."
• "Bigger batches are always cheaper." (wrong) Wasted KV cache memory caps the batch size. (right) "Throughput rises until KV memory runs out, so I manage that memory first."

Follow-up questions

How to reduce RAG latency?

Time each stage, then cut the largest: cache the prompt prefix, shorten the reranker's list, use an approximate index, stream the reply, cap output length, route simple questions to a smaller model, and answer repeated questions from a cache. The steps above give the order.

A naive RAG endpoint returns in 1,500 milliseconds. How do you get it under 100 milliseconds?

State what "returns" means. Tuning every stage can put the first streamed token near 100 milliseconds in the illustrative budget, but a whole generated reply cannot finish that fast. Only an answer cache hit can.¹

How would you cache LLM responses to reduce cost?

Key the cache by the question and by anything that changes the answer, add a freshness limit, and clear entries when documents change. Consider caching the prompt prefix too, since it still gives a fresh answer.

But why would you cache questions that are not related to your main functionality?

Usually you would not cache them; you would route them away with a cheap check before the pipeline, since off-topic traffic costs tokens and returns nothing useful. Caching helps only if the same off-topic question repeats often. This is design judgement, not a sourced rule.

What are common RAG failure points and how do you debug them?

Debug in order: was the right chunk retrieved, was it ranked into the prompt, and did the model use it? Fix the earliest failing stage first, because a later fix cannot recover a passage that was never retrieved.

What if the cache time to live runs out between requests?

Each later request pays the write premium again and gets no reads. Send a request inside the window to refresh it, move to a longer time to live if traffic justifies it, or drop caching for that prefix.

Sources

1. Introducing Contextual Retrieval — Anthropic Engineering, 19 Sep 2024
2. Prompt caching (Claude Platform documentation) — Anthropic, 26 Oct 2026
3. Prompt Caching 101 (OpenAI Cookbook) — OpenAI, 26 Oct 2026
4. Prompt Caching 201 (OpenAI Cookbook) — OpenAI, 26 Oct 2026
5. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al., SOSP 2023), 12 Sep 2023
6. Enhancing RAG with contextual retrieval (Claude Cookbook) — Anthropic, 26 Oct 2026