Transformers & Attention

What breaks first when you push context length way up?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 11 min read

What breaks first when you push context length way up? In a typical serving setup graphics processing unit (GPU) memory breaks first: the key-value (KV) cache grows with every token and every request, and quality and position handling follow. The follow-up trap is that FlashAttention, which removes the stored attention score matrix, leaves that cache and the quadratic compute untouched.

In this article

Before you start
Why interviewers ask this
Context length from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page sits inside the transformer architecture guide, and the neural-network basics are in how neural networks work.

Why interviewers ask this

A candidate who says "attention is quadratic" and stops has named one limit out of four. The question is reported by candidates in large language model (LLM) and system-design rounds, and it separates people who know the formula from people who have run a model at length.

A weak answer picks a single cause, usually the quadratic compute, and treats longer context as a pure speed problem. A strong answer separates memory, compute, position handling and quality, and says which one binds first for the situation described. A Staff-level answer also says that the order depends on the setup: training and serving differ, batch size matters, and the answer changes once a fast attention kernel or a smaller cache is in use. The honest framing is "typically", because no single published ordering covers every model and hardware choice.

Context length from first principles

Score matrix: A score matrix is the table of attention scores that one attention head computes in one layer, with one row and one column per token, so n tokens give n times n entries before the softmax turns each row into weights.
Effective context length: The effective context length of a model is the longest input over which it still does a given task at acceptable quality, which can be shorter than the longest input it accepts.

In plain words

Picture a student reading a long report. For every page, the student compares it with all earlier pages. That is the quadratic part. The student also keeps one note card per page so that earlier pages need not be reread. That is the cache. The analogy breaks in one place: a model does this comparison in every layer and in every head, in parallel.

In one precise sentence, longer context raises the memory of attention scores and of cached keys and values, raises the compute per generated token, and pushes the model into position ranges and use patterns where it was trained little or not at all.

Step 1: Naive attention stores an n-by-n score matrix, and FlashAttention removes only that

Naive attention stores a score matrix of n squared entries per head and layer. Here n is the number of tokens. The self-attention cost per layer grows with n squared, as in the complexity table of Vaswani et al. (2017).

Take a Llama-2-7B-style configuration (illustrative): 32 layers, 32 heads, head size 128, 16-bit numbers. At 4,096 tokens one such matrix is 32 MiB. At 131,072 tokens it is 32 GiB, and at 1,048,576 tokens it is 2,048 GiB.¹ That is the first thing that fails if attention is written naively.

FlashAttention removes the stored score matrix but not the quadratic compute. Dao et al. (2022) present an exact attention method that avoids writing the full matrix to GPU memory, so this wall moves away. The arithmetic does not shrink. The FlashAttention page explains the method, and the page on making attention cheaper lists the alternatives.

Step 2: The KV cache grows linearly and sets how many requests fit

During generation the model does not recompute the keys and values of earlier tokens. A KV cache is a per-layer memory of each processed token's attention keys and values, kept so every new step can reuse them instead of redoing that work. It grows linearly with context length and with batch size.

Its size is 2 × layers × KV heads × head size × tokens × bytes per number × batch. For the illustrative configuration this is 524,288 bytes per token: 2 GiB for one sequence of 4,096 tokens, 64 GiB (about 69 GB) at 131,072 tokens and 512 GiB at 1,048,576 tokens.¹

Take an 80 GB GPU holding about 14 GB of 16-bit weights for a model of about 7 billion parameters (illustrative round figures), which leaves about 66 GB. That fits 30 sequences of 4,096 tokens, 3 of 32,768 tokens and none of 131,072 tokens.¹ The cache stays resident for the whole request, so a longer context first cuts the batch, and so the throughput per GPU.

The cache grows linearly; at 131,072 tokens one sequence's 64 GiB (about 69 GB) exceeds the roughly 66 GB left after the weights.CC BY 4.0 — source: What breaks first when you push context length way up?

Text description of the diagram

The KV cache of one sequence is 2 GiB at 4,096 tokens and 512 GiB at 1,048,576 tokens.
The cache is 64 GiB at 131,072 tokens.

The KV cache of one sequence is 2 GiB at 4,096 tokens and 512 GiB at 1,048,576 tokens.¹

Step 3: The cost of every generated token climbs

Each new token must read the whole cache, plus the weights. At an illustrative memory bandwidth of 2 terabytes per second, reading a 64 GiB cache takes about 34 milliseconds per generated token, against about 1 millisecond for a 2 GiB cache.¹ The 14 GB of weights add a fixed read of about 7 milliseconds, so the part that grows is the cache read, and latency per token rises in a straight line with context, even when everything fits.

Attention compute has two forms. Per generated token, attention reads and multiplies against the whole cache, about 4 × n × d floating-point operations (FLOPs) per layer, with d the model width, 4,096 here (illustrative), so it grows linearly with n.

Over a whole input (prefill or training), the scores plus the weighted sum cost about 4 × n² × d FLOPs per layer: about 275 billion at 4,096 tokens and about 281 trillion at 131,072 tokens, 1,024 times more for 32 times the length.¹ As a share of per-token FLOPs, from 4nd attention against about 24d² for the projections and the feed-forward layer (illustrative), attention is about 14% at 4,096 tokens, 57% at 32,768, 84% at 131,072 and 98% at 1,048,576.¹

For the illustrative 7B-style model, one sequence's cache outgrows the memory left after the weights on an 80 GB GPU between 32,768 and 131,072 tokens.
Context length	KV cache, one sequence	Score matrix, one head, fp16	Cache read per token	Sequences in the 66 GB left
4,096	2 GiB	32 MiB	1.07 milliseconds	30
32,768	16 GiB	2 GiB	8.59 milliseconds	3
131,072	64 GiB	32 GiB	34.36 milliseconds	0
1,048,576	512 GiB	2,048 GiB	274.88 milliseconds	0

Step 4: Positions and quality fail without a crash

A model that accepts a long input has not necessarily learned to use it. Models handle position with schemes such as rotary positional embedding (RoPE), and positions far beyond the trained range are unfamiliar; extension methods rescale or interpolate positions to cover them, as in Chen et al. (2023). The RoPE page shows why.

Even inside the trained range, use is uneven. Liu et al. (2023) note that relatively little is known about how well models use longer context. On multi-document question answering and key-value retrieval, they find that accuracy is often highest when the relevant passage is at the start or end of the input and drops significantly when it sits in the middle.

This failure is silent: the model still answers, and nothing errors. So the effective context length can be shorter than the advertised window, and only a test on your own task shows it.

Context length meets four different limits, and the KV cache is the one that fills the GPU first in a typical serving setup.CC BY 4.0 — source: What breaks first when you push context length way up?

Step-by-step description

Naive attention stores n squared scores.
The cache grows linearly and cuts the batch.
Each token reads the whole cache.
Long input may not be used well.

Naive attention stores a score matrix of n squared entries per head and layer.
The KV cache grows linearly with context length and with batch size.
FlashAttention removes the stored score matrix but not the quadratic compute.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.