What is a KV cache in an LLM, and how big does it get?

In this article

1. Before you start
2. Why interviewers ask this
3. KV cache from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference optimization learning path.

Why interviewers ask this

The KV cache is the reason a long chat costs more memory than a short one, and the reason serving one more user is a memory question before it is a compute question. Interviewers use it to separate candidates who know that a language model generates text one token at a time from candidates who can say what that costs.

A weak answer says "it caches attention to make things faster." A strong answer names what is cached (keys and values, not queries), says why that is safe, gives the formula for its size, and says what you do when it does not fit. A good answer ends with a sizing calculation, which the worked example below walks through.

KV cache from first principles

KV cache

A KV cache is a per-layer store of the attention keys and values of every token already processed that lets a language model skip recomputing them at each generation step.

Prefill

Prefill is the first pass of generation, in which the model reads the whole prompt in parallel and fills the KV cache for every prompt token.

Decode

Decode is the loop that follows prefill, in which the model produces one new token per step and reads the whole cache each time.

KV head

A KV head is one of the separate key-and-value channels stored per layer; in modern models there are often far fewer KV heads than query heads.

In plain words

Think of a person writing a long reply while keeping a notebook of everything they have already read. Each time they write a new word, they glance back through the notebook; they do not re-read the original pages from the start. The KV cache is the notebook. The analogy breaks in one place: the notebook is not a summary. It stores two full vectors, a key and a value, for every earlier word at every layer, so it gets big.

In one precise sentence: during generation the model computes a key and a value for each new token once, stores them, and reuses them for every later token. The idea follows from the attention design in Vaswani et al. (2017), where each token's output is a weighted mix of the value vectors of all tokens up to and including itself.

Step 1: A token becomes a query, a key and a value

Every transformer layer turns the newest token into three vectors: a query, a key and a value. The query asks "what am I looking for?", each key says "this is what I contain", and each value is the information handed back when a key matches.

Each of the three is made by multiplying the token's vector by its own learned weight matrix, and every layer has its own matrices, so one token gives one query, key and value per layer. Inputs are the token's vector from the layer below; outputs are those three vectors. Nothing in this step looks at earlier tokens. You now know what the cache will store: the keys and values, per token, per layer.

Step 2: The cache holds the past

For every token already processed, the model stored its key and value in the cache. Prefill filled the cache for the whole prompt in one parallel pass; decode adds one row per step after that. Each layer keeps its own cache, because the keys and values at one layer are computed from that layer's input and are not the ones the next layer needs; that is why the size formula below multiplies by the number of layers.

Without the cache, generating the next token would mean pushing the entire prefix through every layer again. For a 1,000-token reply that is 500,500 token positions processed instead of 1,000, about 500 times more work.² The cache replaces that repeated work with memory.

Step 3: Write the new k and v, then the query reads the whole cache

First the new token's key and value are written into the cache as its newest row. Then the new token's query is compared with every cached key, including the key it just wrote, because a token attends to all positions up to and including its own. The scores become weights, and the weights mix the cached values into the layer's output, which feeds the choice of the next token.

Only the newest query is needed, because a token's output depends on its own query and on the keys and values of positions up to it. No later token ever needs an earlier token's query. This is why the structure is a KV cache and not a query-key-value (QKV) cache. The cost of this step is memory traffic: each layer reads its own slice of the cache, so every step reads the whole cache once.

Step 4: The row stays, and KV heads set the size

The row written in Step 3 stays for every later step, so the cache grows by one row per layer per token. Its size follows one formula:

Here $L$ is the number of layers, $n_{\text{kv}}$ the number of KV heads, $d$ the head size and $b$ the bytes per value; the leading 2 is one key plus one value. Candidates often put the number of attention heads where $n_{\text{kv}}$ belongs, and that is the error. Llama 3 70B has 64 attention heads but only 8 KV heads: its configuration file (as of October 2026) lists 80 layers, 64 attention heads, 8 KV heads, a model width of 8,192 and 16-bit (bf16) values.

The head size is 8,192 divided by 64, which is 128. Sharing each KV head across eight query heads is called grouped-query attention (GQA); Ainslie et al. (2023) report quality close to full multi-head attention (MHA) at speed comparable to multi-query attention (MQA), which shares one KV head across all query heads (Shazeer, 2019).

1 · Token to q, k, v2 · The cache holds the past3 · Write k, v, then read4 · The row staysReplay

Step-by-step description

1. The newest token becomes a query, a key and a value vector.
2. Keys and values of earlier tokens are stored, not recomputed.
3. The new k and v join the cache; the query scores every cached key, its own included.
4. The row is kept for every later step, one row per layer per token.

• A transformer layer turns each token into a query, a key and a value vector.
• The KV cache stores the keys and values of every token already processed, so only the newest token's vectors are computed at each step.
• Each step writes one new key and value per layer into the cache, then reads the whole cache.

Worked example

The cache size of Llama 3 70B follows in four lines of arithmetic. Bytes per token = 2 × 80 layers × 8 KV heads × 128 head size × 2 bytes = 327,680 bytes, which is 320 KiB.¹ One sequence of 8,192 tokens, the model's maximum position count in its configuration file, holds 327,680 × 8,192 bytes = 2.5 GiB.

Thirty-two such sequences in one batch hold 80 GiB (85.9 gigabytes, GB), more than one 80 GB H100 SXM holds (NVIDIA H100 page, as of October 2026 lists 80 GB of memory and 3.35 terabytes per second (TB/s) of bandwidth for the SXM card). Units: 1 KiB is 1,024 bytes, 1 GiB is 1,073,741,824 bytes (1.0737 GB) and 1 GB is 1,000,000,000 bytes; the cache figures below are in KiB and GiB and GPU memory is in GB.

The last row is an idealisation: real servers also keep activations and fragmentation overhead, so the true number is lower.²

As the context doubles from 1,024 to 8,192 tokens, one sequence's cache goes 0.3125, 0.625, 1.25 and 2.5 GiB: doubling the context doubles the cache.¹

1 · 1,024 tokens2 · 2,048 and 4,0963 · 8,192 tokensReplay

Step-by-step description

1. 0.3125 GiB for one sequence.
2. Doubling the context doubles the cache: 0.625 GiB, then 1.25 GiB.
3. 2.5 GiB for one sequence, before any batching.

• For Llama 3 70B in 16-bit values the KV cache costs 327,680 bytes (320 KiB) per token.
• One 8,192-token sequence holds 2.5 GiB of KV cache; doubling the context length doubles it.

The cache also decides speed. In an idealised memory-bound estimate, each decode step reads all the weights once plus every sequence's cache. On four H100 SXM GPUs with a combined 13.4 TB/s, one sequence at 8,192 tokens takes about 10.6 milliseconds per step, and a batch of 32 takes about 16.9 milliseconds per step while producing 1,898 tokens per second in total.³ That is the case for batching, and it is why cache size, not arithmetic, limits how many users one machine can serve.

Trade-offs and edge cases

The KV cache trades memory for compute, and every refinement of it is a way to spend less memory. Three levers matter.

First, share keys and values across heads. Full multi-head attention with 64 KV heads would need 2,560 KiB per token for this 70B model, eight times the 320 KiB that GQA needs; MQA with one KV head needs 40 KiB.² The saving costs a little quality, which GQA was designed to keep small.

Text description of the diagram

• Full multi-head attention with 64 KV heads would need 2,560 KiB of cache per token for a 70B model.
• Eight KV heads (grouped-query attention) need 320 KiB per token; one KV head (multi-query attention) needs 40 KiB.

Second, manage memory in pages. Existing serving systems waste cache memory through fragmentation and duplication, and Kwon et al. (2023) report that vLLM, which stores the cache in fixed-size blocks, improves throughput by 2 to 4 times at the same latency.

Third, shrink the stored values. Keeping cache entries in 8-bit instead of 16-bit halves the memory per token, at some quality risk that must be measured on your task; see the notes on 16-bit number formats.

Edge cases candidates miss: the cache holds one copy per sequence, so a shared prompt prefix can be stored once only if the server supports prefix sharing; and long context hurts twice, because the cache grows and every step reads more of it. For the surrounding picture of serving, LLM inference optimization is the hub, and FlashAttention covers the attention computation that reads the cache.

How to answer it in an interview

The spoken structure: say what is cached and why, give the formula, give one number, then name the remedy.

Common mistakes

• "The model caches queries too." (wrong) Queries are used only for the current token, so caching them saves nothing. (right) "Only keys and values are reused, because later tokens attend back to them."
• "The cache size is layers times attention heads." (wrong) Models using GQA store far fewer KV heads than attention heads. (right) "I use the number of KV heads from the config."
• "The cache is used in training too." (wrong) Training processes a whole sequence in parallel, so there is nothing to reuse between steps. (right) "It is an inference-time optimization for the decode loop."
• "The cache makes the model faster, full stop." (wrong) It trades compute for memory and, at long context, a bigger read per step. (right) "It removes repeated compute but makes decoding memory-bound."

Follow-up questions

Is the KV cache used during training?

Not in the standard training pass. Training runs the whole sequence through the model in one parallel pass with a causal mask, so every key and value is computed once anyway and there is no step-by-step loop to save work in. The cache belongs to inference, and any training procedure that generates text one token at a time, such as sampling model outputs, uses it the same way inference does.

How do you calculate the KV cache size?

Multiply 2 (key and value) by layers, KV heads, head size and bytes per value, then by tokens and by batch size. For Llama 3 70B in 16-bit that is 327,680 bytes per token.¹

What is the difference between the KV cache and the context window?

The context window is the maximum number of positions the model supports, for example 8,192 in Llama 3 70B's configuration file. The KV cache is the memory currently holding keys and values for the positions in use; it grows toward the window and is what makes a full window expensive.

Why cache keys and values but not queries?

A token's output depends on its own query and on the keys and values of every position up to and including itself, so only keys and values are read again by later tokens. An earlier token's query is never needed by a later token.

What do you do when the KV cache does not fit?

Reduce KV heads with GQA or MQA, store the cache in pages, quantize cache entries, shorten context, or lower the batch size. Each trades memory against quality or throughput. The machine learning operations (MLOps) hub and the deep learning hub cover the neighbouring production and training topics.

Sources

1. Attention Is All You Need — arXiv (Vaswani et al.), 12 Jun 2017
2. Fast Transformer Decoding: One Write-Head is All You Need — arXiv (Shazeer), 6 Nov 2019
3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — arXiv (Ainslie et al.), 22 May 2023
4. Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv (Kwon et al.), 12 Sep 2023
5. Meta-Llama-3-70B config.json (mirror of the model configuration) — Hugging Face (date = read date), 24 Oct 2026
6. NVIDIA H100 Tensor Core GPU — NVIDIA (date = read date), 24 Oct 2026