In this article
Before you start
Why interviewers ask this
A candidate who says "just use a bigger context window" has missed the point of building with large language models (LLMs). Context rot decides how you design agents, retrieval systems and long chats, so the question shows up as "what happens when the prompt gets very long", "why does my agent get worse after many turns", and "how would you keep an agent on track over hours". It usually comes as a follow-up to an agent or retrieval design question.
A weak answer says the model "forgets". A strong answer names the measured effect, explains why attention spreads thin, and lists what you do about it.
Context rot from first principles
In plain words
- Context rot
- Context rot is the measured decline in a language model's accuracy as the number of input tokens grows, even on tasks that stay equally easy. It is a property of how models use long inputs, not a bug in one product.
- Attention budget
- An attention budget is the fixed amount of weight a model can spread across the tokens it reads. Every extra token takes a share, so relevant tokens get less.
- Distractor
- A distractor is a passage in the input that looks related to the question but does not answer it. Distractors compete with the right passage for attention.
Think of a teacher who has exactly one hundred points to hand out when she marks an essay. Give her one paragraph and she can reward the one right sentence with most of the points. Hand her a hundred pages and the same hundred points are spread over far more sentences, so the right sentence gets a thin share. That thin share is what the model sees.
The analogy breaks here: a model has many budgets, one per attention head in each layer. It can also learn to score the right sentence much higher.
In one precise sentence, context rot is accuracy falling as input length rises while difficulty stays fixed.
Step 1: Accuracy falls as the input grows
Accuracy on a fixed task drops as the input gets longer, and this has been measured by several independent groups. Liu et al. (2023, v3) found that accuracy changes when the relevant information moves, with models doing best at the start or end of the input and worst in the middle, even for models built for long inputs.
The decline starts at or well inside advertised limits. The long-context benchmark paper by Hsieh et al. (2024, v3) tested 17 long-context models and reports that all claim a context of 32,000 tokens or more, but only about half keep satisfactory performance at 32,000. The NoLiMa paper (2025, v3) tested 13 models that claim at least 128,000 tokens and reports that at 32,000 tokens 11 of them fall below 50% of their short-context baselines; the generative pre-trained transformer (GPT) model GPT-4o from OpenAI drops from 99.3% to 69.7%.
Chroma's report (July 2025) tested 18 models and concludes that models do not use context uniformly: performance gets less reliable as input grows, even on simple tasks.
Step 2: Attention is a budget that distractors dilute
Attention is a budget because the weights a token assigns to the tokens it reads always sum to 1, so adding tokens shrinks every share. The transformer of Vaswani et al. (2017, v7) turns each token's scores into weights with softmax. Softmax is a function that turns scores into positive weights that add to 1. Anthropic (2025) says models have an "attention budget".
Take one relevant token whose score beats each distractor's by 5 units (illustrative). Its weight is for n distractors. With 100 distractors it receives 0.5974 of the weight; with 1,000 it receives 0.1292; with 100,000 it receives 0.00151. The score advantage did not change. Only the crowd did.
Dilution is one cause, not the whole story. Anthropic adds that models see fewer long sequences than short ones in training. Du et al. (2025, v1) report performance fell 13.9% to 85% as input grew across five models. It still fell on most tasks with the irrelevant tokens replaced by whitespace, and with them masked out, which leaves nothing to dilute.
Step 3: Position and look-alike distractors make it worse
Where the relevant text sits and how much the rest resembles it both change accuracy. Besides the position effect from Liu et al., Chroma varied how similar the question was to the answer passage: lower-similarity pairs lost accuracy faster as input grew (Chroma, July 2025).
A single distractor lowered performance against a baseline with none, and four lowered it further, with uneven effects between distractors. Chroma also found that models did better on shuffled haystacks than on coherent text, which shows that structure matters as well as length. On a conversational-memory benchmark of 306 prompts (Chroma, July 2025), models scored much higher on focused inputs of about 300 tokens than on full inputs of about 113,000 tokens.
The NoLiMa paper adds that benchmarks where the question shares words with the answer flatter models, because matching words is easy at any length. When the match must be inferred, accuracy falls sooner.
Step 4: Context engineering keeps the input small and relevant
The fix is to make the model read fewer, better tokens, which is what context engineering means in practice. Anthropic (2025) lists techniques for long-running agents: compaction, which summarises a near-full conversation and continues from the summary; structured note-taking to storage outside the window; sub-agents that return condensed summaries; and just-in-time retrieval of data by reference.
Match the technique to what the task must remember, because each one throws something away. Exact values favour notes, a long history favours compaction, and independent subtasks favour sub-agents with their own windows. Retrieval pulls in only the passages a question needs; see embeddings and vector search and AI agents.
Step-by-step description
- Accuracy drops as input grows.
- Weights sum to one, so more tokens mean thinner shares.
- Where text sits and what resembles it matter.
- Read fewer, better tokens.
- Accuracy on a fixed task drops as the input gets longer.
- Attention is a budget because the weights a token assigns to the tokens it reads always sum to 1.
- Where the relevant text sits and how much the rest resembles it both change accuracy.
- The fix is to make the model read fewer, better tokens, which is what context engineering means in practice.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
| Distractors | Weight on the relevant token | Weight on all distractors | Pairs of tokens (n × n) |
|---|---|---|---|
| 100 | 0.5974 | 0.4026 | 10,000 |
| 1,000 | 0.1292 | 0.8708 | 1,000,000 |
| 10,000 | 0.0146 | 0.9854 | 100,000,000 |
| 100,000 | 0.0015 | 0.9985 | 10,000,000,000 |
To hold half of the weight against 1,000 distractors the advantage must reach about 6.91 units, against 4.61 for 100 and 11.51 for 100,000, because the required advantage is the natural log of the distractor count1. A reader can redo this: , so with 1,000 distractors the weight is 148.4 / (148.4 + 1,000) = 0.1292. Each tenfold rise cuts the weight 4.6-fold, then 8.8-fold, then 9.9-fold1: the fall nears tenfold only once the distractors far outnumber 148.4.
Text description of the diagram
- With 100 distractors it receives 0.5974 of the weight; with 1,000 it receives 0.1292; with 100,000 it receives 0.0015.
- The score advantage did not change.
Anthropic (2025) points to the n² pairwise relationships among n tokens; the pair-count column counts them, a 100-fold rise for each tenfold rise in length1. FlashAttention computes the same scores without storing the full matrix, which saves memory and time but not arithmetic, and a key-value (KV) cache, the cache of keys and values, saves recomputing past keys and values for each new token. Neither changes the softmax, so neither removes the dilution. Real models learn sharper scores than this toy, so read the figures as an illustration, not a forecast. The memory side is covered in LLM inference.
Trade-offs and edge cases
No mitigation is free, and context rot is not the same in every model or task. The right choice depends on what the task needs to keep.
Text description of the diagram
- Exact values favour notes, a long history favours compaction, and independent subtasks favour sub-agents with their own windows.
- Match the technique to what the task must remember, because each one throws something away.
- Shorter input. Cheapest and most reliable, but you may drop a detail the task needs.
- Compaction. Keeps a long history in few tokens, but a summary can lose an exact value that mattered later.
- Retrieval. Sends only relevant passages, but a miss in retrieval means the model never sees the answer.
- Sub-agents. Give each subtask a clean window. Anthropic's multi-agent write-up reports the cost: about fifteen times the tokens of chat.
- A bigger window. Raises the limit, not the quality at the limit; the Hsieh and Modarressi benchmarks both found drops well below advertised sizes.
Reported results differ by model and task, and Chroma notes that its models were not all used in every experiment. Say "measured on these benchmarks" instead of claiming a universal law. Tool descriptions and results also fill the window, so see the Model Context Protocol page for why the number of tools matters.
How to answer it in an interview
What is context rot, and how do you deal with it in an agent?
Context rot is accuracy falling as the input grows longer, even when the task stays equally easy. Independent benchmarks show it well inside advertised limits. One cause is that attention weights sum to one, so every extra token and look-alike distractor takes a share from the relevant text. It is not the only cause: position matters, and accuracy still drops when the padding is masked out. I handle it by keeping the context small: compact old turns, keep exact facts in notes outside the window, retrieve only what the next step needs, and give subtasks their own agents. I would also measure it by running my own task at several input lengths.
The spoken structure is: define it with the symptom, give the attention-budget reason and its limit, then list the mitigations and say how you would measure.
Common mistakes
- "The model forgets the middle of the prompt" (wrong). Accuracy is worse for middle positions in some tests, but it also falls with length and with distractors (right: say accuracy degrades, then name position as one factor).
- "A bigger context window solves it" (wrong). A larger limit lets you send more tokens, and the benchmarks above show accuracy falling well below the limit (right: size the window to the task, then keep it lean).
- "Put everything in the prompt to be safe" (wrong). Extra passages are distractors that take attention (right: include only what the next step needs).
- "It only matters for huge inputs" (wrong). The Hsieh and Modarressi papers report drops at 32,000 tokens (right: test at the length your product actually uses).
- "Summaries are lossless" (wrong). Compaction drops detail by design (right: store exact values outside the window).
Follow-up questions
How do you apply context engineering?
Start with the task: list what the next step must know, put only that into the window, and move everything else to notes or retrieval. Then measure accuracy at the lengths you actually run.
Context engineering vs memory engineering?
Context engineering decides what goes into the window for this call; memory engineering decides what is stored outside it and when it is recalled. The first manages a single request, the second the state between requests.
Context engineering vs harness engineering?
Context engineering chooses what goes into the window for one call; harness engineering builds the program around the model that runs its loop, calls tools and carries work from one window to the next. Anthropic (November 2025) calls its agent software development kit (SDK) a general-purpose agent harness with context management such as compaction, and found compaction alone was not enough for work that spans sessions.
Does a bigger context window fix context rot?
No. It raises the maximum input, but Hsieh et al. found only about half of the models that claim 32,000 tokens or more hold up at 32,000, and Modarressi et al. found 11 of 13 models that claim 128,000 fall below half their baseline at 32,000.
How would you measure context rot in your own application?
Run the same task with the relevant text fixed and the surrounding input padded to several lengths and positions, then plot accuracy against length. Chroma's method of varying needle position and distractors is a model for this; LLM observability covers logging the length and result of each call in production.
What if the distractors are removed but the input stays long?
Accuracy still falls. Du et al. (2025, v1) report drops of 13.9% to 85% as input grew across five models, and accuracy still fell on most tasks with the irrelevant tokens replaced by whitespace and, for both open models tested, with them masked out, so cut the length as well as the distractors. The same paper reports that asking the model to recite the evidence first helps.
- Context rot is accuracy falling as input length grows while the task stays equally hard.
- Attention weights sum to one, so every extra token and distractor dilutes the relevant text, but length hurts even without dilution.
- Position, look-alike distractors and inferred matches all make it worse.
- A bigger window raises the limit, not the quality at the limit.
- Keep the context small: compact, take notes outside the window, retrieve, and split work across sub-agents.
Sources
- Context Rot: How Increasing Input Tokens Impacts LLM Performance (Hong, Troynikov, Huber) — Chroma technical report, 14 Jul 2025
- Lost in the Middle: How Language Models Use Long Contexts, arXiv 2307.03172 v3 — arXiv (Liu et al.), 20 Nov 2023
- RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv 2404.06654 v3 — arXiv (Hsieh et al.), 6 Aug 2024
- NoLiMa: Long-Context Evaluation Beyond Literal Matching, arXiv 2502.05167 v3 — arXiv (Modarressi et al.), 9 Jul 2025
- Context Length Alone Hurts LLM Performance Despite Perfect Retrieval, arXiv 2510.05381 v1 — arXiv (Du et al.), Findings of EMNLP 2025, 6 Oct 2025
- Attention Is All You Need, arXiv 1706.03762 v7 — arXiv (Vaswani et al.), 2 Aug 2023
- How we built our multi-agent research system — Anthropic, 13 Jun 2025
- Effective context engineering for AI agents — Anthropic, 29 Sep 2025
- Effective harnesses for long-running agents — Anthropic, 26 Nov 2025
