AI agent architecture: how AI agents work, and how to answer questions on them

In this article

1. Start here: the learning path
2. How do AI agents work?
3. What is the difference between agents and workflows?
4. What are the components of an AI agent?
5. What are the main agent architecture patterns?
6. Why do agents fail on long tasks?
7. How do agents use context engineering?
8. What guardrails and human oversight do agents need?
9. AI agents questions by branch
10. Questions interviewers ask on AI agents
11. All 37 guides in AI agents
12. Follow-up questions
13. Sources

Start here: the learning path

AI agent

An AI agent is a system that uses a language model to control the execution of a task by choosing its own next action, calling tools, observing the results and deciding when the task is complete, within limits set by its instructions.

Workflow

A workflow is a system in which language models and tools are orchestrated through paths written in advance in code, so the sequence of steps is fixed by the developer and not chosen by the model.

Guardrail

A guardrail is a check placed around an agent, such as an input filter, a rule or a human approval, that limits what the agent may do or say, and that is layered with other checks because no single check is sufficient.

The path runs from the loop and its vocabulary, through tools, valid tool arguments and human oversight, to debugging a running agent and practising interview answers. Each step adds one idea to the loop.

Text description of the diagram

• The hub has six branches: tools and protocols, oversight, observability, memory and context, reliability and evaluation, and multi-agent systems with design problems.
• Observability means recording every model call, tool call and result so a misbehaving run can be replayed.

1. What is the Model Context Protocol (MCP)?
2. Human in the loop AI agents
3. How do you design an AI agent system? Loop, tools, memory, safety
4. What is context rot in LLMs?
5. Why do agents fail on long tasks?
6. Evaluating agents: trajectories, tool accuracy and reliability
7. How do you design a customer-support agent?
8. Claude Code vs Cursor: how the two coding agents differ
9. How do you design a computer-use / browser agent?
10. How would you make a PDF-based workflow reliable and fast?
11. How do you design an agent platform for production?
12. How do you design a deep-research agent?
13. What are multi-agent systems, and when should you use one?
14. How do you design an agent evaluation harness?
15. What is LLM observability? Tracing, cost and quality signals
16. Are LLMs stateful or stateless?
17. What different strategies could you use for long-term memory?
18. AI agent security: how prompt injection attacks work and how to stop them
19. When would you use n8n in an AI system, and when would you avoid it?
20. How would you handle tool calling failures and partial results?
21. Have you built any end-to-end agentic systems?
22. When have you decided not to build an agent? Four checks that decide it
23. How do you evaluate agent trajectories and tool-use accuracy?
24. How do you get reliable tool use across multi-step tasks?
25. What is RLHF, and how does it train a language model?
26. How Do You Handle State Management in Long-Running Agentic Workflows?
27. Should you build agent orchestration yourself or use a framework like LangGraph?
28. How do you test a non-deterministic agent?
29. How do you evaluate an AI agent, beyond just checking the final answer?

How do AI agents work?

An AI agent works by running a loop in which the model's output becomes an action and the action's result becomes the model's next input. The model reads the goal and history, outputs a tool call or a final answer, and the system executes any tool call and feeds the result back. The loop ends when the model produces a final answer or a stop limit is reached.

The reasoning-and-acting idea behind the loop comes from the ReAct paper, which has a model interleave reasoning steps with actions that query a Wikipedia interface, to reduce the hallucination and error propagation of chain-of-thought prompting. In the paper's Table 1 (PaLM-540B) ReAct beats chain-of-thought on fact verification (Fever, 60.9 against 56.3 accuracy) but slightly trails it on HotpotQA question answering (27.4 against 29.4 exact match); the best prompting results, 35.1 on HotpotQA and 64.6 on Fever, come from combining ReAct with chain-of-thought and self-consistency (Yao et al., 2022, arXiv v3 of March 2023).

OpenAI describes the same structure as a model, tools and instructions, and notes that an agent recognises when a workflow is complete and can halt and hand control back to the user on failure (OpenAI, April 2025). A production loop needs a step limit and a spend limit, because an unbounded loop is an unbounded bill.

1 · Goal and context2 · Model decides3 · Tool runs4 · Result returnsReplay

Step-by-step description

1. The system assembles the goal, instructions, tools and history.
2. The model outputs a tool call or a final answer.
3. The system executes the tool call with its arguments.
4. The result is added to the history; repeat or stop at the limit.

• The model's output becomes an action and the action's result becomes the model's next input.
• The loop ends when the model gives a final answer or a stop limit on steps or spend is reached.

What is the difference between agents and workflows?

A workflow follows paths written in advance, and an agent chooses its own path. Anthropic defines workflows as systems where models and tools are orchestrated through predefined code paths, and agents as systems where models dynamically direct their own processes and tool use (Anthropic, December 2024). The same post names five workflow patterns: prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer.

The practical rule is to start with the simplest design that works. Anthropic recommends agents for open-ended problems where the number of steps cannot be known in advance, and workflows for well-defined tasks that need predictability, and notes that agents cost more and can compound errors (Anthropic, December 2024). If you can draw the flowchart, build the flowchart. A strong interview answer begins by arguing for the least agentic design that solves the problem.

What are the components of an AI agent?

An AI agent has three core components in OpenAI's framing, a model, tools and instructions, and Lilian Weng's earlier survey describes planning, memory and tool use. The two lists describe the same system from different angles. The model plans and decides, the tools act, and the instructions bound behavior; memory is how the loop carries state between turns (OpenAI, April 2025; Weng, June 2023).

Weng splits memory into short-term, which is the model's context window, and long-term, usually an external vector store searched by nearest-neighbor methods. She lists three limits that still frame interview questions: finite context length, difficulty with long-term planning and recovery from unexpected errors, and the reliability of the natural-language interface (Weng, June 2023). The context window is the amount of text a model can read in one call.

What are the main agent architecture patterns?

The main patterns are a single agent in a loop, a manager that delegates to specialist agents, and decentralised agents that hand work to each other. OpenAI describes the manager pattern as a central model that orchestrates specialised agents through tool calls, and the decentralised pattern as agents handing execution to one another with one-way handoffs (OpenAI, April 2025). OpenAI also advises starting with a single agent and moving to multiple agents only when needed.

Anthropic's multi-agent research system uses an orchestrator-worker pattern: a lead agent plans, saves its plan to memory and spawns subagents that search in parallel, and a separate agent attributes claims to sources. Anthropic reports that this system beat a single Claude Opus 4 agent by 90.2% on its internal research evaluation (the multi-agent system used Claude Opus 4 as lead with Claude Sonnet 4 subagents), and that agents use about four times the tokens of chat while multi-agent systems use about fifteen times (Anthropic, June 2025). More agents buy capability and cost tokens; say both in an interview.

Why do agents fail on long tasks?

Agents fail on long tasks because small per-step failure rates compound across steps. If each step succeeds with chance $p$ and steps are independent, a run of $n$ steps has no failed step with chance $p^n$. At $p=0.95$ a ten-step run is clean with chance 0.5987 and a twenty-step run with chance 0.3585; at $p=0.99$ twenty steps give 0.8179, and at $p=0.9$ they give 0.1216.¹ These values are illustrative; real steps are not independent.

Measured reliability shows the same pattern. The tau-bench paper evaluates agents that talk to a simulated user with domain tools and policies, and reports that even strong function-calling agents succeed on under half of tasks, with pass^8 under 25% in the retail domain (June 2024, gpt-4o; Yao et al., 2024). Pass^k is the chance that an agent succeeds on all of k independent attempts of the same task, so it measures consistency and not luck. For a per-attempt success chance $c$ on one task, pass^k is $c^k$: at $c=0.8$ it is 0.8 for one attempt, 0.4096 for four and 0.1678 for eight.¹

Text description of the diagram

• At 0.95 per-step success, a ten-step run is clean with chance 0.5987 and a twenty-step run with chance 0.3585.
• At 0.99 per step, twenty steps give 0.8179; at 0.9 per step, twenty steps give 0.1216.

How do agents use context engineering?

Agents use context engineering to keep only the most useful tokens in the model's context window as a long task grows. Anthropic describes context engineering as curating the full set of tokens available at inference, including tools, external data and message history, and notes that as the number of tokens grows the model's ability to recall information from the context decreases (Anthropic, September 2025).

The post describes just-in-time retrieval and three techniques for long-horizon tasks: compaction, structured note-taking and sub-agent architectures. Compaction summarises a conversation near the limit and continues from the summary. Structured note-taking has the agent write notes to storage outside the window and read them back later. Sub-agent architectures give a specialised agent a clean window and take back a condensed summary. Just-in-time retrieval, covered in a separate section of the post, keeps lightweight references such as file paths and loads data only when needed.

What guardrails and human oversight do agents need?

Agents need layered guardrails and, for risky actions, a human approval step. OpenAI describes guardrails as a layered defence: a single guardrail is unlikely to be sufficient, so rules-based checks, model-based checks and a moderation service are combined, alongside authentication and access controls (OpenAI, April 2025).

The Model Context Protocol (MCP) specification, the open protocol that standardises how model applications connect to tools and data, states that tools represent arbitrary code execution, that hosts must obtain explicit user consent before invoking any tool, and that descriptions of tool behavior should be treated as untrusted unless they come from a trusted server (MCP specification, 2025-06-18). For interview answers: name the action's blast radius, put approval in front of irreversible actions, and log every call. Human in the loop for AI agents goes deeper.

AI agents questions by branch

Each branch below is a short answer to one theme of the path.

Tools and protocols

Tools are how an agent acts, and tool design matters as much as the prompt. A tool should have a clear name, a typed argument schema and an error message the model can act on. The open protocol for exposing tools to many applications is covered in tool use, function calling and MCP, and valid arguments are the subject of reliable JavaScript Object Notation (JSON) from a large language model (LLM).

Oversight and safety

Oversight means deciding which actions an agent may take alone. Reversible, low-risk actions can run automatically; irreversible or costly actions pause for approval. Human in the loop for AI agents covers approval design.

Debugging and observability

Observability means recording every model call, tool call and result so a misbehaving run can be replayed. Without a trace, an agent that failed on step seven cannot be debugged. LLM observability covers what to log and how to read it.

Memory and context

Memory is how an agent carries information between turns and sessions. Short-term memory is the context window; long-term memory is stored outside it and retrieved when relevant.

Reliability and evaluation

Reliability is measured over many attempts, not one run. Evaluate both the final answer and the path taken, and to report consistency with pass^k.

Multi-agent systems and design problems

Multi-agent designs split work among specialised agents and raise cost and coordination problems. For the surrounding design skills, see system design and the machine learning (ML) system design hub.

Questions interviewers ask on AI agents

The agentic AI interview questions page collects answered questions on agents, and the AI engineer interview questions page covers the wider AI-engineering set. For prompt-related rounds, see prompt engineering interview questions and the prompting hub; for how rounds are built, see interview loops.

Follow-up questions

How do AI agents work?

An agent runs a loop in which a model chooses an action, a tool runs it, and the result returns as input until the model gives a final answer or a stop limit is reached.

What are the five types of agent in AI?

The textbook classes are simple reflex, model-based reflex, goal-based, utility-based and learning agents, from the Russell and Norvig taxonomy (Russell and Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020, section 2.4; also summarised on Wikipedia); language-model agents are usually discussed as workflows versus agents instead.

Should I build an agent or a workflow?

Build a workflow when you can write the steps down, and an agent only when the number of steps cannot be known in advance (Anthropic, December 2024).

Why do agents fail on long tasks?

Per-step failures compound, so a 0.95 per-step success chance leaves a twenty-step run clean only 0.3585 of the time in the illustrative calculation above.

How do I measure whether an agent is reliable?

Run each task several times and report pass^k (the agentic AI interview questions page shows how to answer this in a round), the chance of succeeding on all k attempts, as the tau-bench paper does (Yao et al., 2024).

Sources

1. Building effective agents — Anthropic, 19 Dec 2024
2. A practical guide to building agents — OpenAI, 7 Apr 2025
3. ReAct: Synergizing Reasoning and Acting in Language Models (arXiv 2210.03629 v3) — arXiv (Yao et al.), 10 Mar 2023
4. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv 2406.12045) — arXiv (Yao et al.), 17 Jun 2024
5. How we built our multi-agent research system — Anthropic, 13 Jun 2025
6. Effective context engineering for AI agents — Anthropic, 29 Sep 2025
7. Model Context Protocol specification 2025-06-18 — Model Context Protocol project, 18 Jun 2025
8. Artificial Intelligence: A Modern Approach, 4th edition, section 2.4 (agent types) — Pearson (Russell and Norvig), 28 Apr 2020
9. Intelligent agent (pointer only) — Wikipedia, 24 Oct 2026
10. LLM Powered Autonomous Agents — Lilian Weng, 23 Jun 2023