How do you design an agent platform for production?

In this article

1. Before you start
2. Why interviewers ask this
3. Designing an agent platform from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

It helps to know how an AI agent works and the basics of prompting.

Why interviewers ask this

A prototype agent is a loop in one process, and a production agent is a service that must survive crashes, deploys, slow tools and people who take an hour to approve something. The question is reported by candidates in design rounds at AI labs and large tech companies, in wordings such as a workflow platform where agents run for minutes, an enterprise agent that sends email and asks for approval, or an orchestration platform for many concurrent sessions.

A weak answer draws a model, a few tools and a database. A strong answer separates the run from the process that executes it, says where state is saved, puts permission checks outside the model, and explains what a pause costs. A Staff-level answer also says when not to build an agent platform at all, and how a resumed step avoids doing a harmful thing twice.

Designing an agent platform from first principles

Agent platform

An agent platform is a service that accepts a goal, runs a language-model agent against approved tools as a managed run, and handles the parts a bare loop cannot: queuing, saved state, permissions, human approval, limits and observation of each run.

Durable execution

Durable execution is a way of running long programs by saving progress at key points, so a run can stop and resume where it left off, even after a crash, a deploy or a long wait.

Tool gateway

A tool gateway is a service between the agent and the outside world that receives every tool call, checks the caller's identity and permissions, applies limits, and only then runs the action or refuses it.

In plain words

Think of a video game with save points. If the console loses power, you reload the save and carry on. You do not replay the whole game. A durable agent run works the same way. The platform saves the run after each step.

The analogy breaks in one place. A save file cannot undo what the world already saw. If the agent sent an email before the crash, reloading does not unsend it.

In one precise sentence, an agent platform treats each run as saved state plus a queue of steps, and runs the model and the tools as steps that a worker picks up.

Workflow or agent

Anthropic separates two designs. A workflow orchestrates models and tools through predefined code paths. An agent lets the model direct its own process and tool use. The essay says workflows give "predictability and consistency for well-defined tasks" and recommends the simplest solution that works, which may mean no agentic system at all. A platform serves both, and the choice of design comes first. Workflows still need saved state and the gateway, because a fixed code path can crash mid-run and its steps still call tools with side effects.

Step 1: A run API and queue accept the goal

The run application programming interface (API) turns a user goal into a stored run with an identifier, an owner and limits, then puts the first step on a queue. It returns the identifier at once and does not hold the connection open while the agent works. The caller polls or subscribes for progress.

Input: a goal, a tenant and a set of approved tools. Output: a run identifier and a queued step. Limits belong here: a maximum number of steps, a maximum spend and a deadline per run. These limits stop runaway runs and answer the interview question about agents that repeat actions. A worker only needs the run identifier to find the state it must continue.

You now know the front door: it creates a run record and a queued step, and it never runs the agent in the request.

Step 2: Durable state lets a run resume, so side effects must be idempotent

A worker takes a step, calls the model, and writes the result to a state store before the next step starts. There are two common mechanisms.

A checkpoint is a snapshot of the run's state at one step. LangGraph saves one, keyed by a thread identifier, in a persistent store. Its checkpointer documentation says replay re-executes steps from a prior checkpoint. LangGraph's documentation on resuming after an interrupt describes the same behaviour: the interrupted node runs again from its start instead of continuing from the next line.

Temporal keeps an Event History, an ordered log of everything a Workflow did. After a crash, a new worker replays the workflow code to rebuild its state. Model calls and tool calls run as Activities, whose results are recorded and reused on replay (Temporal Workflow docs).

The trap: the interrupted step runs again, so any side effect in it must be idempotent. An idempotent step is one where doing it twice has the same effect as doing it once. Do not assume a side effect happens exactly once; design for repeats.

Step 3: A tool gateway checks every action

The tool gateway receives each tool call and decides whether to run it. It checks who the run acts for, which tools that identity may use, and what limits apply. The model proposes. The gateway disposes.

The OWASP excessive-agency entry says to implement authorization in downstream systems, not to rely on an LLM (large language model) to decide whether an action is allowed. It names three root causes: excessive functionality, excessive permissions and excessive autonomy. Fixes follow from them: give each agent only the tools it needs, only the functions of those tools it needs (a mailbox summary needs read, not send or delete), and credentials scoped to the user.

Triggers include hallucination, prompt injection and, in multi-agent systems, a compromised peer agent. A tool protocol such as the Model Context Protocol (MCP) standardises how tools are described, but it does not replace this check. You now know where permissions live: in the gateway and the downstream systems, never in the prompt.

Step 4: Approval is a saved pause, not a waiting thread

For sending email, deleting a record or moving money, the platform pauses the run and asks a person. The LangGraph interrupt documentation says the graph state is saved through the persistence layer, and the run waits until you resume it. An interrupt needs a checkpointer, a thread identifier and a payload the person can read.

Because the pause is saved state, no worker, thread or connection is held while someone thinks. A restart does not lose it. The person can approve, edit or reject, and the run resumes with that answer. Remember the Step 2 trap: resume re-runs the node, so put the interrupt before the side effect, or keep the earlier work idempotent. The human-in-the-loop page covers approval design in depth.

1 · Run API2 · Durable state3 · Tool gateway4 · ApprovalReplay

Step-by-step description

1. The goal becomes a stored run and a queued step.
2. State is saved after every step.
3. Every tool call is checked.
4. Risky actions pause as saved state.

• A run API turns a goal into a stored run and a queued step.
• A worker saves state after every step so a crash resumes the run.
• A tool gateway checks every tool call.
• Human approval is a saved pause that holds no worker.

Worked example

Size the platform for 1,000 agent sessions in flight. Every input below is an illustrative assumption: 20 model or tool steps per run, 3 seconds per step, 20% of runs waiting on a person for 300 seconds, 4,000 tokens per step, and 10 concurrent steps per worker.¹

A run spends 60 seconds in steps. On average it also spends 60 seconds waiting on people (20% of runs times 300 seconds), so the mean run lasts 120 seconds. Little's law, $L=\lambda W$, says the number of runs in flight equals the arrival rate times the mean duration, so 1,000 = 8.33 × 120 and 1,000 in flight needs about 8.33 new runs a second.¹ That arrival rate gives 500 runs actively stepping and 500 parked, waiting for approval. The parked runs need no worker because their state is saved.¹

Now the value of saved state. Assume each step fails with probability 0.02 (illustrative). A 20-step run then finishes with no failure 66.76% of the time.¹ If a failure restarts the run from step 1, a run executes about 24.89 steps on average. If it resumes from the last checkpoint, it executes about 20.41.¹ The gap grows with run length, which is why runs that last minutes need durable state.

Text description of the diagram

• Restarting from step 1 executes about 24.89 steps per run on average.
• Resuming from a checkpoint executes about 20.41 steps per run on average.

Trade-offs and edge cases

The first trade is checkpoints against event replay. Both save progress, but they save different things.

Text description of the diagram

• Checkpoint-based runtimes save graph state at each step.
• Event-replay runtimes keep an ordered log and replay deterministic workflow code to rebuild state.

Checkpoint-based runtimes save graph state at each step. Event-replay runtimes keep an ordered log and rebuild state by replaying deterministic workflow code, so that code must not read the clock or random numbers directly. Temporal's documentation says the clock differs on replay, so Workflow code must use software development kit (SDK) APIs that supply recorded values. The LangGraph, CrewAI and AutoGen comparison covers the frameworks.

Other edge cases a strong answer names:

• Simplest design first. Anthropic says agentic systems trade latency and cost for task performance, so a fixed workflow beats an agent when the task is well defined.
• Isolation for code execution. For a cloud IDE (integrated development environment) or any code-running tool, give each run its own sandbox and per-tenant limits. The sources read for this page do not compare isolation technologies, so state the boundary you choose and its cost, and do not claim more.
• Runaway runs. Budgets on steps, spend and time, plus detection of the same call repeated, end loops. See why agents fail on long tasks.
• Duplicated side effects. Use idempotency keys on writes, because a resumed step can run again. Activities can be retried, so they too need idempotency keys.

Multi-agent runs, tracing and provider limits

A multi-agent run is one parent run that starts child runs for sub-tasks. Give every child a run identifier linked to its parent, so the fan-out is visible, and let children inherit the parent's budget and identity, so a sub-agent never holds more authority or spend than the run that spawned it.

Each run also needs a trace stored under its run identifier: model and tool steps, tokens, cost and gateway decisions. Evaluation reuses the same trajectory, and an engineer debugs a bad run from it. Finally, the 666,667 tokens per second in the worked example must fit the model provider's rate limits.¹ So the queue applies per-tenant token budgets and backpressure, and a throttled step waits in the queue, not in a worker.

How to answer it in an interview

The spoken structure: say the run is durable, then where state is saved, then the gateway, then the approval pause, then the limits and the repeat trap. Draw the four boxes of the diagram as you speak.

Common mistakes

• "An agent platform is an LLM in a while loop." (wrong) A bare loop loses its place when the process dies. (right) "Each run is saved state plus queued steps, so a worker can resume it."
• "Resume continues from the next line." (wrong) LangGraph's documentation says resuming re-runs the interrupted node. (right) "A resumed step can run twice, so side effects are idempotent or placed after the pause."
• "Approval blocks a thread until the person answers." (wrong) That ties up a worker for an unbounded wait. (right) "The run is saved and parked; nothing is held."
• "The prompt tells the model which actions are allowed." (wrong) A prompt is not an access control. (right) "The tool gateway and the downstream system enforce least privilege."
• "Always build a multi-agent system." (wrong) Anthropic recommends the simplest solution that works. (right) "I start with a fixed workflow and add autonomy only where it pays for its latency and cost."

Follow-up questions

How do you design a multi-agent orchestration platform for a thousand or more concurrent agent sessions?

Size it from arrival rate and run duration, then separate runs actively stepping from runs parked on people, because only the first need workers. In the illustrative worked example above, 1,000 sessions needed about 50 workers, not 1,000 threads. Add per-tenant queues and limits so one tenant cannot starve the rest.

How would you build an evaluation platform for a company running fifty agent workflows?

Record every run as a trajectory of model and tool steps, replay the same tasks against each workflow version, and score the final result and the steps. A strong answer keeps one shared harness and a per-workflow set of tasks. See evaluating agents.

How do you stop an agent that repeats the same action?

Set a maximum number of steps and a deadline per run in the run API, and have the gateway flag the same tool call with the same arguments appearing again. When a limit trips, end the run with a clear status or pause it for a person.

How do you design agent identity, credentials and least-privilege authorization?

Give each run an identity tied to the user it acts for, and let the downstream system limit what that identity can do, as OWASP advises. Issue credentials scoped to the needed tools and functions, and keep them out of the prompt.

How would you design an enterprise agent that searches documents, updates tickets, sends emails and asks for approval?

Map each action to a least-privilege tool behind the gateway: search is read-only and limited to the documents the user may see, and ticket updates are scoped to the needed project and fields with idempotency keys on writes. Email send sits behind an interrupt: the run pauses and saves state, a person approves, edits or rejects, and only then does the send function run.

How would you design a secure cloud IDE for isolated code execution?

Run each session in its own sandbox with per-tenant resource limits and scoped credentials, and route any outside access through the same kind of gateway as agent tools. Name the isolation boundary you chose and what it costs. More design practice is in the agentic interview questions and the agent system design guide.

Sources

1. Building effective agents — Anthropic Engineering, 26 Oct 2026
2. Checkpointers (LangGraph) — LangChain documentation, 26 Oct 2026
3. Interrupts (LangGraph) — LangChain documentation, 26 Oct 2026
4. Understanding Temporal — Temporal documentation, 26 Oct 2026
5. Temporal Workflow — Temporal documentation, 26 Oct 2026
6. LLM06:2025 Excessive Agency — OWASP Gen AI Security Project, 26 Oct 2026