What is KV caching in the context of transformer inference?

In this article

1. Before you start
2. Why interviewers ask this
3. KV caching from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits in the large language model (LLM) inference optimization guide and builds on the transformer architecture.

Why interviewers ask this

The question separates candidates who know that generation is slow and memory-hungry from those who can say why. A weak answer is "it caches earlier results to go faster." A strong answer names what is cached (keys and values, not queries), per layer, and says the saving is real work avoided, with mathematically identical outputs, up to floating-point rounding.

The consequence is what the interviewer probes next: a cache is memory, and memory limits how many requests a GPU serves together. A Staff-level answer gives the size formula, says which knobs shrink it, and connects it to batching and memory layout, such as paged attention. The question appears in AI and machine learning system design rounds, and the same sub-topics come up in serving questions such as serving a 70B model at a latency target.

KV caching from first principles

KV caching

KV caching is a technique in transformer inference that keeps the key and value vectors of every token already processed, in every layer, and reuses them when generating each later token, so earlier tokens are never projected twice.

In plain words

Think of a student writing an essay one sentence at a time. Before each new sentence, the student could reread the whole essay and retake all notes. Or the student could keep the notes and add one line. KV caching is keeping the notes.

The analogy breaks in two places. The notes take up desk space, and the desk is the GPU memory, which is limited. Also, unlike notes, the cache holds exact numbers, one set per layer, so nothing in it is a summary that could lose detail.

In one precise sentence: the cache holds one key vector and one value vector per token per layer, and every new token only adds its own.

Step 1: Each token is projected into a query, a key and a value, and the keys and values are stored

Attention starts by multiplying each token's vector by three learned matrices. That gives a query (what this token is looking for), a key (what this token offers) and a value (what this token passes on). Input: the token vectors. Output: three vectors per token, per layer.

The first pass over the prompt is called prefill: the model processes all prompt tokens together and stores their keys and values. The Hugging Face documentation explains why this matters: autoregressive models predict one token at a time, and "A KV cache stores these calculations so they can be reused without recomputing them." Prefill still computes every prompt key and value once. The cache does not shorten that first pass.

You now know what is stored: keys and values, never queries.

Step 2: Each decode step computes one new query and reads the stored keys and values

To produce the next token, the model projects only the newest token. Its query is compared against every stored key, the scores go through softmax, and the result weights the stored values. Input: one query plus the cache. Output: the next token's attention result, and the new key and value are appended to the cache.

Without a cache, each step would re-project every earlier token. Over a 2,048-token sequence that is 2,098,176 projections without a cache against 2,048 with one.¹ Projection work per step is now constant, but attention still reads all stored keys and values; the cache removes the quadratic re-projection, not the linear read. The price is memory: one key and one value per token per layer, which is 819,200 bytes per token in the illustrative 13-billion-parameter shape.¹

The result is mathematically the same, up to floating-point rounding: the cache stores the very values recomputation would produce, so nothing is approximated. A small seeded script with one head and six tokens confirms the cached and recomputed paths match.¹

Step 3: The cache trades compute for memory

The cache size per token is two (one key, one value) times the number of layers, times the key-value width, times the bytes per number. For a 13-billion-parameter model with 40 layers, a width of 5,120 and 2-byte 16-bit floating point (FP16) numbers (the Open Pre-trained Transformer (OPT) 13B model, the example in Kwon et al. (2023), §3, treated here as illustrative), that is 819,200 bytes, or 800 kibibytes, per token.¹

Kwon et al. note that "The KV Cache size grows quickly with the number of requests." They also describe serving throughput as memory-bound. That is why batching and memory layout matter, covered in continuous batching and the KV cache size page.

Text description of the diagram

• The cache is 0.4194 GB at 512 tokens and 3.3554 GB at 4,096 tokens.
• The cache holds a fixed number of bytes per token.

The chart shows the same shape at four lengths: 0.4194 GB at 512 tokens, 0.8389 at 1,024, 1.6777 at 2,048 and 3.3554 at 4,096.¹

1 · Project and store2 · Attention reads3 · Memory growsReplay

Step-by-step description

1. The new token's key and value go into the cache.
2. The query is compared with every stored key.
3. The next token is produced and the cache grows, trading compute for memory.

• Each token is projected into a query, a key and a value.
• The key and value are stored in the cache.
• Attention reads the stored keys and values to output the next token.

The Staff-level answer

What makes it Staff-level: it states the formula, the unchanged output, the memory limit on batch size, and the levers that follow.

The rubric

Three variants of this question

Does the KV cache help during training?

No. Training processes all positions of a sequence in one parallel pass, so there is no one-token-at-a-time loop to reuse. KV caching is an inference technique for generating tokens one after another.

Why not cache the queries too?

A query is used once, by the token that produced it, so storing it saves nothing. Only keys and values of past tokens are read again by later tokens.

What if the sequence gets twice as long?

The cache doubles, since it holds a fixed number of bytes per token. For the illustrative shape, 2,048 tokens take 1.6777 GB and 4,096 take 3.3554 GB.¹

Common mistakes

• "The cache stores the queries too." (wrong) Queries are used once. (right) "It stores keys and values of past tokens; the new token's query is computed fresh."
• "KV caching makes the model approximate." (wrong) It reuses exact values. (right) "The outputs are the same; only repeated work is skipped."
• "It is 800 kibibytes per token everywhere." (wrong) That is one illustrative shape. (right) "The size scales with layers, key-value width and bytes per number."
• "It speeds up prefill." (wrong) Prefill computes every prompt key and value once. (right) "It speeds up the decode steps that follow."

Follow-up questions

How much memory does a KV cache use?

Two times layers times key-value width times bytes per number, per token, multiplied by tokens and by concurrent requests. For the illustrative 13B-class shape it is 819,200 bytes per token.¹

How do you shrink the KV cache?

Share keys and values across query heads with grouped-query attention (GQA), which keeps one key-value set for a group of query heads. With 8 key-value heads of width 128 on the same 40 layers (illustrative), the per-token cost falls to 163,840 bytes, five times smaller, and 41 full-length requests fit instead of 8.¹ Quantizing the cache to fewer bits helps in the same way; see LLM quantization.

Does the KV cache make the first token faster?

No. Prefill must still project every prompt token once, so the first token waits on that pass. The cache pays off on every later token.

Is reading the cache a bandwidth problem as well as a capacity problem?

Each decode step reads the stored keys and values from GPU memory, so both capacity and traffic matter; the high bandwidth memory (HBM) page covers the hardware side. Reusing a cached prompt prefix across calls is a related idea at the service level: caching LLM calls.

Sources

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al., SOSP 2023), 12 Sep 2023
2. KV cache strategies, Transformers documentation — Hugging Face, 26 Oct 2026