What is a context window in an LLM, and what does it cost?

In this article

1. Before you start
2. Why interviewers ask this
3. Context window in an LLM from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one part of the transformer architecture path, and it assumes the basics from how neural networks work.

Why interviewers ask this

A candidate who only says "a bigger window is better" misses where the bill comes from. Interviewers use the context window to test whether you can connect a product limit to the mechanism under it: why long inputs cost so much, why quality can fall inside a window the model technically accepts, and how modern models stretch beyond the length they were trained on.

Candidates report being asked what the context window is and what it costs, what breaks first when length rises, and how recent open models reach very long windows. One reported interviewer wanted rotary positional embeddings and their scaling methods rather than "better graphics processing units (GPUs)".

A weak answer names a number of tokens. A strong answer separates the limit, the cost and the quality, and says which grows with the square of the length and which grows in a straight line. A Staff-level answer adds what you would do in a real system: retrieve less, cache more, and measure recall at the lengths you actually use.

Context window in an LLM from first principles

Context extension

Context extension is a family of methods that lets a model work at a longer length than the one it was trained on, such as rescaling its position encoding or training further on long documents.

Lost in the middle

Lost in the middle is the measured pattern in which a language model finds relevant information most reliably at the start or end of a long input and less reliably in the middle, even when the whole input fits inside its window.

In plain words

Think of a desk. The context window is the desk's surface. Only what lies on it can be read right now. Your own written answer also needs space on it. Anything in the filing cabinet is out of reach until someone puts it on the desk. The analogy breaks in one place. A real desk does not get harder to read as it fills up, but a model's attention does, as the next steps show.

In one precise sentence, the window is the longest sequence of tokens, input plus output, for which the model's attention and position handling were built and tested.

Step 1: Tokens fill the window, and the answer needs room too

Text is first cut into tokens. Every token of the prompt, of any pasted document, of earlier chat turns and of the answer being written counts against one shared budget. A long answer therefore leaves less room for input.

The window is not the model's memory. The model starts each request with nothing but its trained weights and what you send, so a chat that "remembers" earlier turns is simply sending them again every time. The limit is an engineering choice about how long a sequence the model was trained on and can serve. Vendors publish it per model and change it often; read the documentation of the model you use on the day you need it. You now know what fills the window; the next step explains why filling it is expensive.

Step 2: Attention compares every token with every other token

Self-attention scores each token against every other token in the window. For a window of n tokens that is n times n scores, one grid per attention head in each layer. The cost grows with the square of the length, so doubling the length quadruples the number of scores.

For an illustrative 8,192-token input the grid holds 67,108,864 scores; at 131,072 tokens it holds 17,179,869,184; at 1,000,000 tokens, 1,000,000,000,000.¹ Stored naively in 16-bit numbers, one grid for one head in one layer would take 0.125 GiB at 8,192 tokens and about 1,862.65 GiB at 1,000,000.¹ Implementations such as FlashAttention (Dao et al., 2022) avoid writing the full grid to memory, which fixes the memory problem but not the number of comparisons. Input: n tokens. Output: n squared scores.

Going from 8,192 to 16,384 tokens multiplies these scores by 4 but the cache of the next step by only 2.¹ The scores hit first in training and in the first pass over a long prompt; the cache hits first when a server holds many long requests together.

Step 3: The KV cache grows in a straight line with length

When a model writes an answer one token at a time, it keeps the keys and values of every earlier token so it does not recompute them. This store is called the key-value (KV) cache. A KV cache is a per-layer store of the keys and values of every earlier token. Each new token adds the same fixed amount, so the cache grows linearly with length.

The size per token is 2 × layers × KV heads × head dimension × bytes per number. For an illustrative model shape of 80 layers, 8 KV heads, a head dimension of 128 and 2 bytes per number, that is 327,680 bytes per token.¹ One sequence then needs 2.5 GiB at 8,192 tokens and about 305.18 GiB at 1,000,000 tokens.¹ The cache is proportional to length: 8 times the tokens need 8 times the memory, 5 GiB at 16,384 tokens against 40 GiB at 131,072.¹ Serving many users at once multiplies this again. Input: n tokens. Output: a cache n times larger than for one token.

Step 4: Positions must stretch beyond the trained length

A transformer reads order from a position signal. Many open models, such as Llama 2, use rotary positional embedding (RoPE, Su et al., 2021), which encodes position by rotating the query and key vectors by an angle that depends on the token's position; the rotary positional embedding page shows the rotation in detail. Angles beyond the trained length are unfamiliar to the model.

Context extension methods rescale those angles. Position interpolation divides every position by a scale factor so new, longer positions map into the range the model already knows. Neural tangent kernel (NTK) aware scaling, a community proposal from 2023 that the YaRN paper describes, changes the rotation base instead. YaRN (Yet another RoPE extensioN method; Peng et al., 2023) refines the rescaling across frequencies. These methods are often followed by a short run of training on long text. As an illustrative example, a model trained on 4,096 tokens and stretched to 16,384 uses a scale factor of 4, so position 12,000 is treated as position 3,000 and position 1 as 0.25.¹

1 · Tokens fill it2 · Pairs multiply3 · Cache grows4 · Positions stretchReplay

Step-by-step description

1. Prompt and answer share one token budget.
2. Every token is compared with every other.
3. Each token adds the same cache amount.
4. Rescaled positions reach past the trained length.

• Attention compares every token with every other token.
• The KV cache grows linearly with length.
• Position scaling extends the trained length.

Worked example

Take the illustrative shape above and compare lengths. Each row counts the pairs that attention scores and the key-value cache for one sequence.¹

The ratios are exact: from 8,192 to 16,384 tokens the number of scores rises by a factor of 4.0 and the cache by a factor of 2.0.¹ That is why "what breaks first" depends on the setup. Comparison count and activation memory bite in training and in the first pass over a long prompt; cache memory bites when many long requests are served together.

Text description of the diagram

• The KV cache for one sequence is 2.5 GiB at 8,192 tokens.
• The KV cache for one sequence is 305.18 GiB at 1,000,000 tokens.

The arithmetic uses a model shape chosen for illustration; real models differ. Grouped-query attention (grouped-query attention (GQA), Ainslie et al., 2023) lets several query heads share one key-value head, which is why the illustrative shape has 8 KV heads. With 64 query heads sharing 8 key-value heads, the cache is 8 times smaller than with 64 key-value heads.¹ For where this sits among attention variants, see multi-head attention.

Trade-offs and edge cases

A window that fits is not a window that works. Liu et al. (2023) tested multi-document question answering and key-value retrieval and found that performance is often highest when the relevant information is at the beginning or end of the input and degrades significantly when it sits in the middle, even for models built for long inputs. So a long window raises what you may send, not how reliably the model uses it.

Second, extension is not free. Rescaling positions lets the model run at a longer length, but short-text behaviour and long-range recall must be checked, not assumed. Third, a longer input costs more per request, because the first pass over the prompt and the cache both scale with the tokens you send, not with the size of the window; a longer trained length also raises training cost per sequence. Vendor pricing pages state the per-token price for each model.

Text description of the diagram

• Retrieval changes which text enters the window.
• These methods are often followed by a short run of training on long text.

Fourth, retrieval-augmented generation (RAG) sidesteps the limit by searching a store and putting only the relevant passages in the window. It trades a long window for a retriever that can miss.

How to answer it in an interview

The spoken structure: define the limit, give the two costs with their growth rates, name the quality caveat, then the extension methods and the alternative.

Common mistakes

• "Context length is only about how much I can paste in." (wrong) Position handling and recall limit it too. (right) "The window is the limit; recall and cost depend on how much of it I fill."
• "Bigger windows are better, full stop." (wrong) Recall in the middle can fall, and cost rises. (right) "A bigger window allows more input; I measure recall at the length I use and compare it with retrieval."
• "Attention memory is the whole problem." (wrong) Avoiding the full grid helps memory but not the number of comparisons or the cache. (right) "Efficient kernels fix the grid's memory; the comparisons and the KV cache remain."
• "Long context comes from better GPUs." (wrong) Hardware helps, but the model also needs position handling that works at that length. (right) "Rescaled rotary positions plus long-text training, then serving optimisations."

Follow-up questions

Which LLM has the largest context window?

The answer changes every few months. As of April 2025, Meta's Llama 4 announcement states a 10 million token window for Llama 4 Scout, and OpenAI's generative pre-trained transformer (GPT)-4.1 announcement states support for up to 1 million tokens, so the two differ by a factor of ten, and newer releases may have moved both figures. Meta credits attention layers without positional embeddings and inference-time attention temperature scaling for Scout's length. Read each vendor's documentation on the day you need it, compare input and output limits separately, and test recall at your own lengths, because the stated window is a limit, not a quality measure.

How to increase the context window of an LLM?

For a model you train or host, rescale the rotary position encoding (interpolation, NTK-aware scaling or YaRN) and continue training on long documents. For a hosted model you cannot change the window, so trim or summarise the input, or retrieve only the relevant passages.

What is the effect of changing an LLM's context window size?

A larger window lets more text in. The longer inputs it allows raise the compute for the first pass, grow the KV cache linearly and can dilute recall in the middle. A smaller window forces more trimming or retrieval, while a longer trained length raises training cost per sequence.

How do LLMs handle long context?

LLMs handle long context by combining position encodings that tolerate longer sequences, efficient attention kernels that avoid storing the full score grid, a KV cache that skips recomputation, and training or fine-tuning on long text. Many systems add retrieval so that the window holds only what matters.

What breaks first when you push context length way up?

In serving it is usually memory: the KV cache per sequence grows linearly, so fewer requests fit on one accelerator. In training and the first pass over a long prompt, the square growth in comparisons hits first. After that, recall in the middle of the input degrades.

Is the context window the same as the model's memory?

No. The window is the text visible in one request; it is rebuilt from scratch each time, and anything outside it is gone unless your application resends or retrieves it. More questions of this kind are in the LLM interview questions.

Sources

1. Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172, v3) — arXiv (Liu et al.; accepted at TACL), 6 Jul 2023
2. RoFormer: Enhanced Transformer with Rotary Position Embedding (arXiv 2104.09864) — arXiv (Su et al.), 20 Apr 2021
3. Extending Context Window of Large Language Models via Positional Interpolation (arXiv 2306.15595) — arXiv (Chen et al.), 27 Jun 2023
4. YaRN: Efficient Context Window Extension of Large Language Models (arXiv 2309.00071) — arXiv (Peng et al.), 31 Aug 2023
5. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv 2305.13245) — arXiv (Ainslie et al.), 22 May 2023
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv 2205.14135) — arXiv (Dao et al.), 27 May 2022
7. Llama 2: Open Foundation and Chat Models (arXiv 2307.09288) — arXiv (Touvron et al.), 18 Jul 2023
8. The Llama 4 herd: the beginning of a new era of natively multimodal AI innovation — Meta AI, 5 Apr 2025
9. Introducing GPT-4.1 in the API — OpenAI, 14 Apr 2025