In this article
Before you start
This page is one step of the AI agent architecture learning path, and it builds on prompt engineering.
Why interviewers ask this
An answer that lists components without saying who controls the loop leaves every follow-up question open. Candidates report this prompt as an architecture question: describe how you would architect an AI agent system, including the agent loop, tool interfaces, memory design, orchestration and safety.
A weak answer names a framework and a vector database. A strong answer says what runs in a cycle, what each tool call returns, where state lives between steps, why some control flow is fixed code, and which limits the model cannot override. A Staff-level answer adds the cost: context grows every turn, and small per-step errors compound over a long run. This page builds that answer from zero. The design-an-agent-system walkthrough applies it as a full interview design.
Agent loop, tools, memory and orchestration from first principles
- Agent loop
- An agent loop is a cycle in which a language model reads its context, chooses either a final answer or a tool call, receives the tool's result as an observation, and repeats until a stop condition is met, so each step is chosen using what the previous step returned.
- Orchestration
- Orchestration is the code or runtime that decides which model calls, tools and worker models run, in what order, with what shared state, and who controls that flow: paths written in advance or the model itself.
In plain words
Picture a new employee with a task, a phone, a notebook and a manager. The employee decides what to do next, uses the phone to get facts, and writes notes. The manager sets a deadline and approval rules. The loop is the employee's cycle. The tools are the phone. Memory is the notebook, and orchestration is how the work is assigned.
The analogy breaks in one place. A person remembers on their own. A model call only knows what the application puts into its context. In one precise sentence, an agent system wraps a model in code that feeds it observations, runs its tool requests and enforces limits.
Step 1: The loop runs until a limit outside the model stops it
The agent loop is the model choosing an action, a tool running it, and the observation going back to the model. Anthropic describes agents as typically just LLMs using tools based on environmental feedback in a loop. Yao et al. (ICLR 2023) named the pattern ReAct: the run alternates thoughts and actions, so it becomes a series of thought-action-observation steps. Each step needs ground truth from the environment, such as a tool result.
The loop must end. A loop limit stops the run from outside the model. Anthropic notes it is common to include stopping conditions such as a maximum number of iterations. The model can finish by itself with a final answer, but your code cannot count on that. A limit written in the prompt is a request. A limit written in code is a rule. You now know the core cycle; next comes what a tool is.
Step 2: Tools are functions the model can request but never runs
A tool is a function the application exposes to the model, described by a name, a description and an input schema. In the Model Context Protocol (MCP), a tool definition carries a name, an optional title, a description, an inputSchema in JSON Schema and an optional outputSchema (MCP specification, 2025-06-18). Clients discover tools with a tools/list request. A call carries the exact name and the arguments that match the schema (architecture overview, 2026-07-28).
An MCP host is the AI application that coordinates one or several MCP clients. In the stateful 2025-06-18 model, each client keeps a connection to an MCP server; the 2026-07-28 architecture overview describes a stateless request and response core. The model never runs the function. Your code validates the arguments, runs it and returns the result as the next observation. Anthropic treats tool design as the agent-computer interface (ACI) and says it deserves as much prompt-engineering attention as the overall prompts.
Step 3: Memory is state the application saves and puts back into context
Agent memory is state that the application stores and re-inserts into the model's context; in typical use the model call itself keeps nothing between calls. Short-term memory is the message history and working state of one conversation thread, and in LangGraph it is part of the agent's state. A checkpointer is a component that saves that state to a database so the thread can be resumed at any time (LangGraph persistence).
Long-term memory is information about a user or an application kept across sessions and shared between threads. LangGraph keeps it in a store as JSON documents organized by namespace and key. Memory carries state between steps and sessions, and the design questions are what to write, when to read it back and when to drop it. A context that only grows eventually hurts quality; see context rot and context folding for long-horizon agents.
Step 4: Orchestration decides who controls the flow
Orchestration code decides who controls the flow, and the first choice is workflow or agent. A workflow is a system where LLMs and tools are orchestrated through predefined code paths. In an agent, the LLM dynamically directs its own process and tool use, as Anthropic defines the difference.
Anthropic names five workflow patterns: prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer. In orchestrator-workers, a central LLM breaks a task into parts, delegates them to worker LLMs and synthesizes the results. The word has a second sense: a framework runtime such as LangGraph's state graph with persistence, compared in LangGraph vs CrewAI vs AutoGen. MCP is not an orchestrator. It connects applications to tools and data, and the orchestration logic stays in the host application.
Step 5: Guardrails and approvals sit outside the model
Guardrails are permission, approval and input-handling rules enforced by code that the model cannot rewrite, placed between the model's tool request and the tool's execution. Prompt injection is text that changes a model's behavior against its instructions, as the Open Worldwide Application Security Project (OWASP) describes in its entry on prompt injection. OWASP calls it indirect when the model accepts input from an external source such as a website or file, which is how a tool result can attack an agent. OWASP's entry on excessive agency lists three root causes: excessive functionality, excessive permissions and excessive autonomy.
The Cloud Security Alliance reports that in the 2026 list excessive agency rises to No. 3, and advises identity-based, least-privilege access outside the model. The MCP specification says clients must treat tool annotations as untrusted unless the server is trusted, and advises confirmation prompts so a human stays in the loop. See human in the loop for AI agents.
Step-by-step description
- A model picks actions; a limit stops the run.
- A tool runs the action and returns an observation.
- State carries between steps and sessions.
- Code decides who controls the flow.
- Approvals and permissions sit outside the model.
- The model picks an action, a tool runs it, and the observation goes back to the model.
- A loop limit stops the run from outside the model.
- Memory carries state between steps and sessions.
- Orchestration code decides who controls the flow.
- Guardrails and approvals are enforced outside the model.
The five steps add up as follows. The model picks an action, a tool runs it, and the observation goes back to the model. A loop limit stops the run from outside the model. Memory carries state between steps and sessions. Orchestration code decides who controls the flow. Guardrails and approvals are enforced outside the model.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
Here is an illustrative refund-support agent. The trace is hand-built in the ReAct style and is not measured data.
Goal: refund the customer if the order is eligible.
Thought: I need the order record first.
Action: lookup_order(order_id from the message)
Observation: delivered, within the return window, paid by card.
Thought: eligible; refunds above the approval limit need a human.
Action: issue_refund(order_id, amount)
Observation: held for human approval.
Final answer: the refund is requested and awaiting approval.Now the cost. Assume, as an illustration, that the system prompt and goal take 1,000 tokens, and every turn appends 150 tokens of model output and 400 tokens of tool result, so each call is 550 tokens larger than the last.1 Call 10 then reads 5,950 tokens, and ten calls process 34,750 input tokens in total, against 10,000 if the context never grew. Twenty calls process 124,500, against 20,000.1
Errors compound too. If every step succeeds with probability 0.95 (illustrative), ten steps all succeed with probability 0.5987 and twenty steps with 0.3585.1 To keep a ten-step run at 0.9, each step must succeed with probability 0.9895, and a twenty-step run needs 0.9947.1 This assumes independent steps and no recovery; retries, validation and checkpoints raise the real figure, which is why errors that go unnoticed are the dangerous ones.
| Quantity | Value (illustrative) |
|---|---|
| Context at call 1 | 1,000 tokens |
| Context at call 10 | 5,950 tokens |
| Input tokens over 10 calls | 34,750 |
| Same 10 calls with a fixed context | 10,000 |
| Input tokens over 20 calls | 124,500 |
| Same 20 calls with a fixed context | 20,000 |
| All of 10 steps right at 0.95 per step | 0.5987 |
| All of 20 steps right at 0.95 per step | 0.3585 |
| Per-step success needed for 0.9 over 10 steps | 0.9895 |
| Per-step success needed for 0.9 over 20 steps | 0.9947 |
Text description of the diagram
- In this illustration, at 0.95 per step, 10 steps succeed about 0.5987 of the time and 20 steps about 0.3585.
- To keep a ten-step run at 0.9, each step must succeed with probability 0.9895.
In this illustration, at 0.95 per step, 10 steps succeed about 0.5987 of the time and 20 steps about 0.3585. Anthropic states the same qualitative point: the autonomous nature of agents means higher costs and the potential for compounding errors. Measuring this on real runs is the subject of evaluating agents.
Trade-offs and edge cases
An agent buys flexibility with cost and latency. Anthropic says agentic systems often trade latency and cost for better task performance, so you should consider when that makes sense. Its advice is to find the simplest solution first, which might mean not building an agentic system at all. A single model call with retrieval is often enough. The token totals above ignore prompt caching and context compaction, which reduce the cost of re-reading a growing context and are the usual follow-up in a design round.
Text description of the diagram
- A workflow follows predefined code paths.
- An agent directs its own process and tool use.
A workflow follows predefined code paths, and an agent directs its own process and tool use. Use a workflow for fixed, known tasks. Use an agent where the number and order of steps cannot be known in advance.
Edge cases follow. Frameworks add abstraction layers that can obscure prompts and make debugging harder, so Anthropic suggests starting with the model application programming interfaces (APIs) directly. The tooling landscape moves quickly, so check the date of any framework advice (as of October 2026). The MCP architecture overview of 2026-07-28 describes a newer revision with a stateless protocol core, a server/discover request and a protocol version in each request's _meta field, so check which revision your client targets. Finally, tool results are untrusted input: a loop that obeys instructions found in a web page has handed control to whoever wrote the page.
How to answer it in an interview
Walk me through the parts of an agent and who controls each.
I'd build a loop. The model gets the goal and history, then either answers or requests a tool call. My code validates the arguments, runs the tool and appends the observation, then repeats. Tools get clear names, descriptions and JSON schemas with least-privilege permissions. Short-term memory is saved thread state; long-term memory is a store keyed by user. I'd start with a fixed workflow and add model-directed steps only where the task needs them. Safety lives outside the model: an iteration cap, approvals for risky actions, and tool results treated as untrusted input.
The spoken structure: describe the loop, then tools, then memory, then who controls the flow, and finish with the limits the model cannot override.
Common mistakes
- "An agent is any chatbot that uses an LLM." (wrong) A workflow follows predefined paths, and an agent directs its own steps. (right) "An agent is a model that chooses its own actions in a loop; a fixed pipeline is a workflow."
- "The model remembers the earlier steps." (wrong) The application stores state and puts it back into the context. (right) "Memory is saved state that I re-inject, per thread or per user."
- "The system prompt tells the agent to stop after a few steps." (wrong) A prompt is a request, not a limit. (right) "The iteration cap and the approval rules are enforced in code."
- "MCP is our orchestrator." (wrong) MCP connects applications to tools and data. (right) "MCP exposes tools; the host application owns the loop and the flow."
- "We trust the tool descriptions from any server." (wrong) The specification says annotations are untrusted unless the server is trusted. (right) "Descriptions from untrusted servers are treated as input, and risky calls need approval."
Follow-up questions
What is the difference between a workflow and an agent?
In a workflow, predefined code paths orchestrate the model and the tools. In an agent, the model directs its own process and tool use. Choose the workflow when the steps are known and the agent when they are not.
How does an agent decide when to stop?
The model can return a final answer instead of a tool call, but your code enforces the stop with a limit such as a maximum number of iterations. Human checkpoints can pause the run as well.
What is the difference between short-term and long-term memory in an agent?
Short-term memory is thread-scoped state, saved by a checkpointer so the conversation can resume. Long-term memory lives in a store across sessions, as JSON documents organized by namespace and key.
Is MCP an agent framework?
No. The Model Context Protocol is an open protocol that connects applications to tools and data. The host application still owns the loop and the orchestration, as the MCP overview explains.
Is ReAct the same as chain-of-thought?
No. Chain-of-thought is reasoning only, while ReAct interleaves reasoning with actions that touch the environment. Yao et al. (ICLR 2023) report absolute success-rate gains of 34% and 10% over imitation and reinforcement-learning baselines on two interactive benchmarks, using one or two in-context examples.
How do you defend an agent against prompt injection?
Use layers: treat tool results as untrusted, give each tool the least privilege it needs, enforce permissions outside the model and require human approval for sensitive actions. Questions like this recur in agentic AI interview questions.
- The agent loop is model, tool call, observation, repeated until a limit stops it.
- Tools are described by a name, a description and an input schema; the application runs them.
- Memory is application state: thread-scoped short-term state and a cross-session store.
- Orchestration decides whether code or the model controls the flow; start with the simplest option.
- Limits, permissions and approvals belong outside the model, and long runs compound errors and cost.
Sources
- Building effective agents — Anthropic Engineering (modified 2026-10-09; accessed 2026-10-26), 19 Dec 2024
- ReAct: Synergizing Reasoning and Acting in Language Models (arXiv 2210.03629) — arXiv (Yao et al.; accessed 2026-10-26), 6 Oct 2022
- Tools, Model Context Protocol specification 2025-06-18 — Model Context Protocol (accessed 2026-10-26), 18 Jun 2025
- Architecture overview, Model Context Protocol 2026-07-28 — Model Context Protocol (accessed 2026-10-26), 28 Jul 2026
- Memory overview, LangChain documentation — LangChain (undated page; accessed 2026-10-26), 26 Oct 2026
- Persistence, LangGraph documentation — LangChain (undated page; accessed 2026-10-26), 26 Oct 2026
- LLM01 Prompt Injection, OWASP GenAI Security Project — OWASP GenAI Security Project (Top 10 for LLM Applications 2025, released 2024-11-18; accessed 2026-10-26), 18 Nov 2024
- LLM06 Excessive Agency, OWASP GenAI Security Project — OWASP GenAI Security Project (Top 10 for LLM Applications 2025, released 2024-11-18; accessed 2026-10-26), 18 Nov 2024
- The OWASP Top 10 for LLM Applications 2026: what changed and why it matters — Cloud Security Alliance (accessed 2026-10-26), 21 Oct 2026
