Where RAG latency and cost go, stage by stage

In this article

1. Before you start
2. Why interviewers ask this
3. The RAG request from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the retrieval-augmented generation guide. Read it with a rough idea of these three things:

Why interviewers ask this

Candidates report this question in generative AI system design rounds in several forms: speed up a slow RAG endpoint, estimate a budget for a large document collection, or fit a design inside a latency and spending limit. The consequence of a wrong mental model is concrete: a candidate who tunes the vector index first spends the round on the part of the pipeline that is small.

A weak answer lists tricks (cache, shard, use a faster database) with no measurement. A strong answer splits one request into stages, says which stage owns the wait and which owns the bill, and names a lever for each. A Staff-level answer adds that latency and cost respond to different levers, so one change can help one and barely touch the other.

The RAG request from first principles

Vector search

Vector search is a lookup that compares a query embedding with stored embeddings and returns the entries closest in meaning, which in a typical RAG request is a short step compared with generating the reply.

In plain words

Think of a research assistant who first pulls a few pages from a filing cabinet, then reads them, then dictates a reply. Pulling the pages is quick. Dictating takes as long as the reply is long, and the reading and dictating are what the assistant bills for. The analogy breaks in one place: a model reads its input in one fast pass, but writes its output one token at a time.

In one precise sentence, a RAG request spends little time on retrieval, most of its time on generating output tokens, and most of its money on input and output tokens together.

Step 1: Retrieval adds a short front end

A RAG request runs six stages in order: embed the question, search the vector index, optionally rerank the results, assemble the prompt, prefill, and decode. Prefill is the pass in which the model reads every input token before it writes anything. Decode is the stage in which the model writes its output tokens one after another.

The first four stages are the retrieval front end. In the illustrative budget used on this page they take 30, 25, 150 and 5 milliseconds, 210 in total, about 4.5% of a request.¹ These stage times are assumptions, not measurements, so replace them with your own traces. You now know the stages; the next step shows which one owns the wait.

Step 2: Output tokens set the latency, and trimming the prompt barely does

Generating tokens is, in the OpenAI latency guide, almost always the highest-latency step, and cutting 50% of output tokens may cut about 50% of latency. The same guide says cutting 50% of the prompt may improve latency by only 1–5%. The illustrative budget agrees. With 3,000 input tokens (an assumed 200-token system prompt, a 100-token question and six 450-token chunks) read at an assumed 10,000 tokens a second, prefill takes 300 milliseconds.

With 250 output tokens written at an assumed 60 tokens a second, decode takes about 4,167 milliseconds, about 89% of the 4,677 total. Halving the output saves about 44.5% of the wait; dropping three of six retrieved chunks removes 1,350 tokens and saves 135 milliseconds, about 2.9% of the wait, yet it cuts about 33.7% of the per-query cost (Step 3).¹ That is the answer to the follow-up above.

The limit is context length. With a 100,000-token context, prefill alone is 10 seconds at the same assumed rate, so for long-context RAG trimming input does cut latency, mostly the wait before the first token.¹ The 1–5% figure is a heuristic for typical prompt sizes.

1 · Retrieval2 · Generation3 · BillReplay

Step-by-step description

1. Retrieval and prompt assembly are short.
2. Decode takes most of the wait.
3. Token counts times prices set the cost.

• Retrieval is a short front end to the request.
• Decode takes most of the wait.
• Token counts of input and output set the bill.

Text description of the diagram

• Illustrative decode time is about 4,167 milliseconds.
• Illustrative retrieval stages take 210 milliseconds in total.

Step 3: Tokens times price set the cost

The bill for a request is the number of tokens in each category multiplied by that category's price. The OpenAI cost guide lists fewer requests, fewer tokens and a smaller model as the levers. Retrieval puts the chunks into the input, so the input side is where RAG pays.

Prices change, so the example uses illustrative price units per million tokens: 1 for input, 4 for output and 0.1 for the query embedding. For a million requests that is 3,000 units of input, 1,000 of output and 10 for embedding a 100-token question each time, 4,010 in total.¹

This is the per-query bill. The other side is fixed: embedding the corpus once, re-embedding changed documents, and keeping the vector index in memory (an illustrative 204.8 gigabytes of raw vectors in the 10-million-document variant below)¹. At low query volume that fixed side can outweigh the token bill.

Both rows are computed from the illustrative inputs above.¹ The two levers are not rivals; they act on different columns.

The Staff-level answer

What makes it Staff-level: it separates the latency lever (output tokens) from the cost lever (input tokens), names the trade-off, and starts from measurement rather than a favorite tool.

The rubric

Three variants of this question

A naive RAG endpoint returns in 1.5 seconds. How do you get it under 100 milliseconds?

At the assumed 60 tokens a second, 100 milliseconds buys about 6 output tokens, so a full generated answer rarely fits.¹ Say that out loud, then pull the levers: a semantic or exact cache for repeated questions, returning retrieved passages without generation, a smaller model on faster serving, dropping the 150-millisecond reranker, or moving the target to time to first token with streaming. Measure your own stages first, because these figures are assumptions.

Estimate the budget for a RAG pipeline over 300,000 legal contracts.

Count tokens, then price them. With an assumed 1,500 tokens per contract and 300-token chunks, 300,000 contracts become 1,500,000 chunks; at an assumed 1,024 numbers of 4 bytes each, that is about 6.1 gigabytes of raw vectors, and embedding the collection once costs 45 illustrative units.¹ The recurring per-query bill dominates, as in Step 3.

How do you scale a RAG system to 10 million articles?

Memory and index choice change first. At the same assumptions, 10 million documents become 50 million chunks, about 204.8 gigabytes of raw 32-bit vectors or 51.2 gigabytes with 8-bit numbers, and brute-force search would need about 51.2 billion multiply-adds per query, so an approximate nearest-neighbor (ANN) index is needed.¹ Relationship-heavy collections may call for GraphRAG. See embeddings and vector search and chunking strategies.

Common mistakes

• "The vector database is why it is slow." (wrong) In the budget it is a small slice. (right) "I would trace each stage first; generation usually owns the wait."
• "I will halve the context to make it twice as fast." (wrong) Halving the prompt may give only 1–5%, by the heuristic in the OpenAI latency guide, and that holds for typical prompt sizes, not very long contexts. (right) "Trimming context cuts cost; shorter answers cut latency; with very long contexts trimming also shortens the wait for the first token."
• "Streaming makes the whole answer faster." (wrong) Microsoft documents that streaming does not change the time to get all tokens. (right) "Streaming lowers time to first token."
• "Fewer chunks is free." (wrong) Fewer chunks can lose the passage that holds the answer. (right) "I cut chunks only while retrieval quality holds on an evaluation set."

Follow-up questions

How would you optimize latency in a RAG pipeline?

Trace every stage, then act on the largest: shorten the output, use a smaller model if quality holds, stream the answer, make fewer requests, and drop reranking if it is not worth its stage time. Also use prompt caching for the repeated system prompt: providers that support it reuse an unchanged prompt prefix, which can lower input cost and time to first token, so check your provider's rules. The RAG interview questions page has related prompts.

How do you keep a 10-million-document system with per-user access control fresh, fast and cheap?

A strong answer attaches permissions as metadata to every chunk and filters at query time, re-embeds only changed documents so freshness does not mean a full rebuild, and watches the per-query token bill. Prefer pre-filtering, which restricts the search to chunks the user may see, over post-filtering the top results, which can return too few results or none. Check your vector store's filtering behavior, and see securing a RAG system for access control in depth.

How do you fit a two-second latency budget and a fixed monthly spend?

Write the budget per stage, then cap output tokens and retrieved chunks, because those set both limits. Fixed spend means requests times tokens per request times price must stay under the cap, so a smaller model or a cache is a design decision, not a tweak.

Does streaming make a RAG answer faster?

It makes the answer start sooner, not finish sooner. Microsoft Learn says it does not change the time to get all the tokens but reduces the time for first response.

Does a reranker belong in a low-latency RAG pipeline?

It is a trade: in the illustrative budget it is the largest retrieval stage at 150 milliseconds.¹ Keep it if evaluation shows it lifts answer quality, and drop it first when the latency target is tight.

Sources

1. Latency optimization (OpenAI API documentation) — OpenAI (accessed 2026-10-26), 26 Oct 2026
2. Cost optimization (OpenAI API documentation) — OpenAI (accessed 2026-10-26), 26 Oct 2026
3. Performance and latency (Azure OpenAI documentation) — Microsoft Learn (accessed 2026-10-26), 26 Oct 2026