In this article
Before you start
For the wider picture, start with the AI agent architecture overview.
Why interviewers ask this
Interviewers use "architect an agent system" to see whether a candidate can bound an unpredictable component. One common wording names five parts: the agent loop, tool interfaces, memory, orchestration and safety. Reported variants ask for a customer-service agent, a code-review agent, a claims-processing backend or a multi-agent research system that returns a cited report.
A weak answer draws a box called "agent" with arrows to "tools" and "database". A strong answer says where the model decides and where code decides, names what is stored between steps, and says what stops a run that goes wrong. A Staff-level answer adds numbers: how fast cost grows per step, how reliability compounds, and how many runs the platform serves at once. The rest of this page builds those numbers from scratch.
AI agent system design from first principles
- AI agent system
- An AI agent system is a program that gives a language model a goal and a set of tools, lets the model choose its own next action in a loop until the goal is met or a limit is hit, and surrounds that loop with memory, orchestration and safety controls.
- Tool interface
- A tool interface is the contract that tells the model what one action does: a name, a plain description and a typed list of inputs, plus the code that checks those inputs and runs the action.
In plain words
Think of a new employee with a to-do list, a phone and a notebook. The employee decides the next call, makes it, reads the answer, writes a note and decides again. A manager sets spending limits, checks big decisions and keeps a log. The analogy breaks in one place: the employee understands consequences, while the model only continues text, so the manager's limits must live in code, not in a polite request.
In one precise sentence, an agent system is a model-driven loop of choose, act and observe, bounded by code. Anthropic's engineering team separates two designs. Workflows (Anthropic, 2024-12-19) run the model along paths your code fixes in advance. Agents let the model direct its own steps and tool use. As of December 2024, the same article advises finding the simplest solution and adding complexity only when needed. Agentic systems trade latency and cost for task performance.
Step 1: The agent loop chooses, acts and observes
The loop is the core. Each turn the model receives the goal, the tool list and the history, and outputs either a tool call or a final answer. Code runs the call and appends the result. In short, the agent loop is the model choosing an action, code running it and the result returning to the model. The ReAct paper (Yao et al., 2022, v3 2023) describes interleaving reasoning traces with actions: the reasoning helps the model track and update a plan and handle exceptions, while actions let it gather information from outside sources.
Input: goal plus history. Output: one action or one answer. The loop must end, so it needs an exit: a final-answer action, a step limit or a budget. Anthropic's article stresses that at each step the agent should get ground truth from the environment, such as a tool result, to judge its own progress. You now know the core; the next step is what the model is allowed to call.
Step 2: Tool interfaces decide what the model can do
A tool interface is where most agent bugs live. The model sees only a name, a description and the input fields, so vague descriptions produce wrong calls. A good interface has one clear job, typed inputs (for example a date, not free text), validation in code before anything runs, and error messages that tell the model what to fix.
Separate read tools from write tools. Reading a record is safe to retry; sending an email or issuing a refund is not, so write tools take an idempotency key, an identifier that makes a repeated call do nothing the second time. The Model Context Protocol (MCP) is a standard way to list and call tools across programs; it standardises the connection, not the loop or the safety rules, which stay your job. Output of this step: a short, safe menu of actions.
Step 3: Orchestration runs the loop, checkpoints it and caps it
An orchestrator is the program that starts runs, calls the model, executes tools, enforces limits and stores state; the model never runs itself. Its jobs are a queue of pending runs, a worker pool that executes steps, retries with backoff, a step and token budget per run, a timeout per tool call, and a checkpoint after every step.
Checkpoints decide what a failure at step nine costs. If step nine fails, a run with checkpoints reloads the state after step eight and retries only step nine. Without them, the run starts over, repeating model calls and, worse, repeating tool side effects. The worked example below puts numbers on that cost. Many teams use a durable workflow engine or a graph framework for this layer; the LangGraph, CrewAI and AutoGen comparison covers three of them.
Step 4: Memory and context keep the run inside its window
Memory in an agent system has three layers. Working memory is the context window, the text the model sees now. Run state is structured data held by the orchestrator, such as the plan and the results so far. Long-term memory is a store the agent reads and writes across runs, such as user preferences or notes.
The window fills because every step adds text and every call re-reads all of it. Three fixes are common: compaction, where an older history is replaced by a short summary; notes, where the agent writes facts to an external store and reads them back when needed; and sub-agents, where a worker does a messy subtask and returns only a brief result. A longer window delays the problem and adds noise; see context rot and context folding for long-horizon agents.
Step 5: Safety limits what a wrong decision can cost
Safety assumes the model will sometimes choose wrongly or be fooled by text in a web page or document, called prompt injection. So the design limits damage in code: least-privilege credentials per tool, an allow-list of tools per task, input validation, output filtering, spending caps, an audit log of every call, and a human approval step before irreversible actions.
Human handoff is part of this layer. The run pauses at a checkpoint, a person sees the plan and the pending action, and approves, edits or rejects it; Anthropic's article notes that agents can pause for human feedback at checkpoints or blockers. Design the handoff trigger explicitly: low confidence, a high-value action or a repeated tool failure. See human in the loop AI agents.
Step-by-step description
- The model chooses, code acts, the result returns.
- Typed, validated actions the model may call.
- Limits, retries and a checkpoint per step.
- Run state, notes and summaries.
- Permissions, approvals and an audit log.
- The agent loop is the model choosing an action, code running it and the result returning to the model.
- Tool interfaces, orchestration with checkpoints, memory and safety are layers around that loop.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Take one illustrative run: a support agent with 3,000 tokens of instructions and tool definitions, 12 steps, and 500 tokens added to the context per step. Every call re-reads the whole context, so the input tokens processed across the run are 12 × 3,000 plus 500 × (0 + 1 + … + 11), which is 69,000.1
Stretch the run to 40 steps and the total is 510,000, which is 7.39 times the uncached input tokens for 3.33 times the steps, because the tokens processed grow with the square of the step count.1 Prompt caching bills re-reads of an unchanged prefix at a small fraction of the base input price (one-twentieth for Claude Opus 5.5 in Anthropic's prompt-caching pricing table, as of October 2026), but it does not lift the window limit.
Compaction helps modestly here. If, after step six, the history is replaced by an 800-token summary (illustrative), the total falls from 69,000 to 55,800, a saving of about 19%.1
Reliability compounds too. If each step succeeds with probability 0.95 (illustrative) and failures are detected, a 12-step run completes cleanly with probability 0.54.1 Restarting from scratch on every failure costs about 17.01 steps on average; retrying only the failed step from a checkpoint costs about 12.63.1 Last, scale: 200 concurrent runs, each step taking 4 seconds of model time and 2 seconds of tool time, produce 33.33 model calls per second and finish 10,000 runs an hour.1
| Quantity (illustrative inputs) | Value |
|---|---|
| Input tokens processed, 12 steps, no compaction | 69,000 |
| Input tokens processed, 40 steps, no compaction | 510,000 |
| Input tokens, 12 steps, compaction after step six | 55,800 |
| Run success chance at 0.99 per step | 0.89 |
| Run success chance at 0.95 per step | 0.54 |
| Run success chance at 0.90 per step | 0.28 |
| Expected steps, restart from scratch at 0.95 | 17.01 |
| Expected steps, retry from checkpoint at 0.95 | 12.63 |
| Model calls per second, 200 concurrent runs | 33.33 |
Text description of the diagram
- A 12-step run at 0.95 per-step reliability completes cleanly with probability 0.54.
- At 0.90 per-step reliability the same run completes with probability 0.28.
Trade-offs and edge cases
The first trade-off is autonomy against predictability. If the steps are known in advance, a fixed workflow is cheaper, faster and easier to test than an agent; Anthropic's advice is to start with the simplest solution. A multi-agent design, where an orchestrating model hands subtasks to worker models, adds more model calls and more places to fail (Anthropic, 2025-06-13), so it earns its cost only when work splits into independent parts, as in a research task that gathers sources in parallel and a final step that attaches citations.
Text description of the diagram
- A workflow follows paths your code fixes in advance, while an agent chooses its own steps.
- A multi-agent system adds model calls, so start there only when work splits into independent parts.
Edge cases to name. A browser agent reads pages that can contain hostile instructions, so treat page text as data and never as commands. An insurance-claims backend needs an audit trail and human sign-off on denials. A run that changes real systems needs idempotent writes. Evaluation is hard when the underlying application programming interfaces (APIs) change: replay recorded tool results for regression tests and run a smaller live suite; see evaluating agents. Reported practice differs on how much to trust model self-checks, so verify with code where you can.
How to answer it in an interview
Walk me through how you would architect an agent platform end to end, from the loop to the safety limits.
I'd start with the simplest thing that works. The core is a loop where the model picks a typed tool, code validates and runs it, and the result goes back as context. Around it, an orchestrator queues runs, caps steps and tokens, retries, and checkpoints after every step so a failure at step nine resumes there. Memory is run state plus summaries or notes so the context window doesn't fill. Safety means least-privilege tools, human approval for irreversible actions, and an audit log. I'd evaluate by replaying recorded trajectories.
The spoken structure: loop first, then the orchestrator's limits and checkpoints, then memory, then safety, then how you evaluate.
Common mistakes
- "I'll use a multi-agent system." (wrong) It adds cost and failure points before you know a single agent fails. (right) "I'd start with one loop or a fixed workflow and split only when subtasks are independent."
- "The model decides when to stop." (wrong) A model can loop forever. (right) "Code enforces a step limit, a token budget and a timeout."
- "MCP handles the agent." (wrong) MCP connects tools; it does not run the loop or enforce safety. (right) "MCP standardises tool access, and my orchestrator still owns limits and approvals."
- "A bigger context window solves memory." (wrong) Uncached cost grows with every re-read, caching cuts the price but not the window limit, and old text adds noise. (right) "I compact, take notes and use sub-agents."
- "The prompt tells the agent not to do harmful things." (wrong) Prompts are not controls. (right) "Permissions and approvals live in code."
Follow-up questions
How would you manage the context window for long-running tasks?
Keep the working context small: summarise old history, write durable facts to an external notes store and read them back on demand, and delegate noisy subtasks to sub-agents that return short results. In the worked example, one compaction cut input tokens by about 19%.
How would you handle this through MCP?
Expose each capability as a described, typed tool, list them to the model and call them through the protocol, but keep an allow-list, argument validation and approval for risky calls in your own orchestrator. Treat tool descriptions from third-party servers as untrusted input. See what MCP is.
How do you design human handoff for an AI agent?
Define triggers (low confidence, high-value action, repeated failure), pause at a checkpoint, show the person the plan and pending action, and resume with their decision recorded in the audit log. The escalation case in a support system works the same way.
How do you evaluate an agent when the underlying APIs keep changing?
Record full trajectories with tool results, replay them against new versions to catch regressions, and keep a small live suite for drift. Score the final outcome and the path, such as wrong tool or wasted steps.
How would you solve a tool execution planning scenario?
Two designs exist. In the interleaved style of the ReAct paper, the model reasons, calls one tool, reads the result and decides again, so it adapts to surprises but spends a model call on every step. In plan-then-execute, the model writes the whole plan first, code checks each planned call against the allow-list and its argument types, execution runs the steps in order, and the plan is revised when an observation shows a failure. Plan-then-execute needs fewer model calls and leaves an inspectable plan to approve; it adapts less well when the world changes mid-run.
How would you design a multi-agent research system that produces a cited report?
Use several agents only when work splits into independent parts that can run separately and a single context would overflow; otherwise one agent or a fixed workflow is cheaper and easier to debug. For a cited report, an orchestrating agent plans sub-questions, worker agents search and read sources and return short findings with source links, and a final step assembles the report and checks each claim against its source. Budget tokens per worker, because Anthropic's account of its research system (2025-06-13) describes multi-agent runs as using more tokens than single-agent ones.
More practice questions are in the agentic AI interview questions.
- An agent system is a model loop of choose, act and observe, bounded by code.
- Tool interfaces are typed, validated and split into safe reads and guarded writes.
- The orchestrator caps steps and tokens and checkpoints every step.
- Uncached context tokens grow with the square of steps, so cache, compact, take notes or delegate.
- Safety lives in permissions, approvals and audit logs, never in the prompt alone.
Sources
- Building effective agents — Anthropic Engineering (accessed 2026-10-26), 19 Dec 2024
- ReAct: Synergizing Reasoning and Acting in Language Models (arXiv 2210.03629 v3) — arXiv (Yao et al.; accessed 2026-10-26), 10 Mar 2023
- How we built our multi-agent research system — Anthropic Engineering (accessed 2026-10-26), 13 Jun 2025
- Prompt caching — Anthropic documentation (undated page, pricing table read 2026-10-26), 26 Oct 2026
