In this article
Before you start
This page is part of the AI agent architecture learning path.
Why interviewers ask this
Agent questions in design rounds quickly turn to memory, because every agent eventually runs out of room. A strong answer shows that you know where the limit bites and which tool fixes which problem.
A weak answer says "use a model with a bigger context window" or "add a vector database" and stops. A strong answer separates three problems: the model uses a long context unevenly, a long run overflows the window, and several agents multiply cost. It then matches a practice to each problem and names what the practice costs. A Staff-level answer adds a rule for when not to use several agents at all. The same ideas appear in the system-design question how do you design an AI agent system, where memory is one of the parts you must draw.
Long-context memory from first principles
- Context window
- A context window is the maximum amount of text, counted in tokens, that a language model can read and write in one call, covering the prompt, the history, tool results and the reply together.
- Compaction
- Compaction is the practice of summarizing a conversation that is nearing the context window limit and starting a fresh window that holds the summary in place of the full history.
In plain words
Think of a desk and a filing cabinet. The desk is the context window. You can only work on what is spread in front of you. A crowded desk makes things hard to find. The cabinet is memory outside the window. It holds a lot, but you must fetch a page. The analogy breaks in one place. A model does not look away from a crowded desk. It reads everything on it, and it reads some parts worse than others.
In one precise sentence: keep each window small and relevant, store the rest outside it, and split big jobs across agents that each have their own window.
Step 1: Treat the context window as a budget, not a bin
A bigger window does not mean the model uses all of it equally well. Liu et al. (2023) tested multi-document question answering and key-value retrieval and report that "performance is often highest when relevant information occurs at the beginning or end of the input context", with a significant drop for facts in the middle, even in models built for long context.
Anthropic's context post describes the same effect as context rot: as the number of tokens in the window grows, the model's accuracy at recalling information from it falls. Its advice is to treat context as a finite resource with diminishing returns. The context rot page explains the effect in more detail.
Input: everything you could put in the window. Output: only what the next step needs, with the key instruction at the start or end. You now know the rule behind every later step: fewer, better tokens beat more tokens.
Step 2: Price the run before you split the work
Price a multi-agent run before splitting the work: it adds capacity and adds cost. In Anthropic's data (2025), agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more. These are tokens billed across a whole run, because every turn re-reads the growing context and several agents add up. They say nothing about how full one window gets. The same post says these systems fit valuable tasks with heavy parallelism, information larger than one context window, and many tools.
What drives the split is how much material the task must read. Take an illustrative research task that must read 600,000 tokens of sources. One 200,000-token window cannot hold that. Split across five subagents, each reads about 120,000 tokens and returns a condensed summary of 1,000–2,000 tokens. The lead agent then holds about 5,000–10,000 tokens of findings, not 600,000.2 A task that fits one window rarely justifies the extra spend.
Input: the volume of material to read. Output: a go or no-go on several agents. Willison's notes summarize the benefit as managing the context limit.
Step 3: Compact the history before the window fills
Compaction keeps a long single-agent run alive past the window limit. When the history nears the limit, the agent writes a summary of the decisions, open problems and key facts, and a new window starts from that summary. Anthropic's context post names compaction as one of three techniques for long runs, together with structured note-taking and multi-agent architectures.
The cost is lost detail. A summary can drop a fact that matters later, so a good summary is written for the next step: it states the goal, what is done, what is decided and what is still open, and it leaves out raw tool output that can be fetched again. As advice, compact before the window is nearly full: the context-rot effect in Step 1 means a crowded window is read less well.
Input: a long history. Output: a short summary plus the latest few turns.
Step 4: Save plans and notes outside the window
Structured notes put durable facts in storage outside the context window, and the agent reads them back when needed. In Anthropic's research system, the lead agent saves its plan to memory because the context is truncated once it passes 200,000 tokens, and the plan must survive (Anthropic, 2025).
Good notes are short, labelled and written to be re-read: the plan, the decisions so far, and what each subagent found. Short-term memory is what sits in the window now. Long-term memory is a store the agent writes to and reads from across steps or sessions, such as files, a key-value store or a vector index, with write, read and expiry rules. Retrieval from a vector index suits large, searchable knowledge; its risk is stale or irrelevant hits that fill the window with noise. Anthropic also describes a just-in-time pattern: keep lightweight references such as file paths or links and load data only when a step needs it. It notes this is slower than reading pre-computed data.
Input: facts worth keeping. Output: a small note the agent can reload.
Step 5: Give each subagent its own clean window
Subagents isolate context. Anthropic's context post describes specialised subagents handling focused tasks with clean windows while the main agent coordinates a high-level plan. A worker may explore at length in its own window, and the lead reads a condensed result, not the whole trail. Windows are private, so shared state lives in the notes store. A subagent returns either a condensed summary of 1,000–2,000 tokens or, for large output, saves it to external storage and passes back a lightweight reference, so the lead is not filled with copied text.
Coordination is where it goes wrong. The multi-agent post lists early errors such as spawning 50 subagents for a simple query and agents distracting each other with excessive updates, and says prompt engineering was the main lever for fixing them. Give each subagent an explicit objective, output format and boundary. The same post warns that domains where all agents must share one context, or have many dependencies, are a poor fit. See also why agents fail on long tasks and context folding, a related way to shrink a long trajectory.
Step-by-step description
- Treat the window as a budget.
- Price the run before splitting.
- Summarize history before it fills.
- Save plans outside the window.
- Give each worker a clean window.
- Treat the context window as a budget, because models use long inputs unevenly.
- Price a multi-agent run before splitting the work.
- Compaction, notes outside the window and subagents with clean windows keep each context small.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The arithmetic below uses the sourced multipliers and illustrative assumptions: a research task that must read 600,000 tokens, five equal subagents, summaries of 1,000–2,000 tokens, a 10,000-token chat answer as the cost base, and a budget of 1,000,000 tokens.2 For that cost base, an agent bills about 40,000 tokens and a multi-agent run about 150,000, so one multi-agent run bills as many tokens as fifteen chat answers.
| Quantity | Value (illustrative inputs) |
|---|---|
| Material the task must read | 600,000 tokens |
| One context window | 200,000 tokens |
| Fits one window? | No, it is three times the window |
| Reading per subagent with five subagents | 120,000 tokens |
| Summary returned by each subagent | 1,000–2,000 tokens |
| Findings held by the lead agent | 5,000–10,000 tokens |
| Chat answer (cost base) | 10,000 tokens |
| Single agent at 4×, billed | 40,000 tokens |
| Multi-agent run at 15×, billed | 150,000 tokens |
| Chat answers per 1,000,000 tokens | 100 |
| Single-agent runs per 1,000,000 tokens | 25 |
| Multi-agent runs per 1,000,000 tokens | 6 |
The 4× and 15× rows are billed totals across a run, not window sizes. The reading task does not fit one window, but each subagent reads a fifth of it and the lead sees only the summaries. The price is that the same budget buys 6 multi-agent runs instead of 100 chat answers. In the cited research system, token use by itself explained 80% of the variance in performance on the BrowseComp evaluation (Anthropic, 2025), so some of the gain is simply more tokens spent.
Text description of the diagram
- For an illustrative 10,000-token chat answer, a single agent uses about 40,000 tokens.
- A multi-agent run uses about 150,000 tokens, fifteen times the chat answer.
Trade-offs and edge cases
Every memory strategy trades something for a smaller window, and the choice depends on the task. Compaction is cheap but lossy, notes survive truncation but need good write and read rules, subagents add capacity but multiply tokens, and just-in-time retrieval avoids stale indexes but is slower.
Text description of the diagram
- Compaction is cheap but a summary can drop detail.
- Subagents add capacity but multiply tokens.
- Just-in-time retrieval is slower than pre-computed data.
Edge cases to name in an interview. Errors compound in long runs: the multi-agent post says agents are stateful, a failure cannot simply restart from the beginning, and its team built systems that resume from where the agent stopped. A task with many dependencies between steps often does better in one agent.
Anthropic's context post suggests a hybrid, loading some context up front and retrieving the rest at run time, for less dynamic content, such as stable reference material. Framework choices change the plumbing, not the principles; see LangGraph, CrewAI and AutoGen compared. Measure the result with agent evaluation, because an unmeasured memory scheme is a guess. Model context sizes and prices change often, so check a vendor's current documentation before quoting one.
How to answer it in an interview
How would you handle memory for a long-running multi-agent LLM system?
I treat context as a budget, because models use long inputs unevenly and accuracy drops for facts in the middle. I price the design first: multi-agent runs use about fifteen times a chat's tokens, so I only split work that exceeds one window or is truly parallel. Then I compact old history, save the plan and findings to notes outside the window, and give each subagent a clean window with a clear objective. I measure with an evaluation set, not by feel.
The spoken structure: name the limit, price the design, then give compaction, notes and subagents with one cost each.
Common mistakes
- "A bigger window means the model uses all of it." (wrong) Accuracy is often worst for facts in the middle. (right) "I keep the window small and put key instructions at the start or end."
- "Multi-agent is always better." (wrong) It costs about fifteen times a chat's tokens, and token use explains much of the gain. (right) "I use several agents only when the task is valuable, parallel or larger than one window."
- "Memory means a longer prompt." (wrong) A longer prompt is the same window, just fuller. (right) "Memory is a store outside the window plus rules for reading and writing it."
- "Subagents share everything." (wrong) Each has its own window, so it knows only what the lead tells it. (right) "I give each subagent its objective, format and boundary."
Follow-up questions
How does memory work in an LLM agent?
The model itself keeps nothing between calls, so an agent builds memory around it. Short-term memory is the context window, which holds the working state of the current task, and long-term memory is a store outside the window, such as notes, files, a key-value store or a vector index. The agent reads and writes that store under rules for what to save, when to retrieve and when to expire it, and the risk is stale or irrelevant hits. In a multi-agent system, windows stay private and the shared state lives in that store.
Does a bigger context window mean the model uses all of it well?
No. Liu et al. (2023) find that accuracy often drops for relevant information in the middle of a long input, even for models built for long context.
When should you use several agents instead of one?
Use several when the task is valuable, heavily parallel, larger than one window or tool-heavy, as Anthropic's post describes. For a small or tightly dependent task, one agent is cheaper and simpler.
How do you stop an agent's context from filling up on a long task?
A strong answer compacts the history early, moves plans and findings into notes outside the window, and hands focused sub-tasks to subagents with clean windows. Each fix has a cost, so apply the lightest one that works.
What is the difference between compaction and external memory?
Compaction rewrites the history into a shorter summary inside the window. External memory stores facts outside the window and loads them on demand. Use both: compaction for the running conversation, notes for durable facts.
- Context is a limited budget; models use long inputs unevenly, so keep windows small.
- Multi-agent runs use about 15× a chat's tokens, so use them only for valuable, parallel or oversized tasks.
- Compact early, save plans and findings outside the window, and give subagents clean windows.
- Every strategy trades detail, speed or tokens for a smaller window; name the cost.
- Measure with an evaluation set, and check vendor documents for current limits.
Sources
- Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172) — Stanford, UC Berkeley, Samaya AI (Liu et al.; TACL 2024), 6 Jul 2023
- How we built our multi-agent research system — Anthropic Engineering, 13 Jun 2025
- Effective context engineering for AI agents — Anthropic Engineering, 29 Sep 2025
- Anthropic's multi-agent research system: notes — Simon Willison's Weblog, 14 Jun 2025
