Human in the loop AI agents

In this article

1. Before you start
2. Why interviewers ask this
3. Human-in-the-loop for agents from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

The question usually arrives as a scenario: your agent deleted a production database, or your agent sends email for users, so how do you stop it doing harm? Behind it sits a simple test of whether you treat an agent as a model or as a system with credentials. A weak answer is "add a prompt that says be careful" or "have a human approve everything". A strong answer separates actions by reversibility and blast radius, shrinks permissions first, gates only the actions that can hurt, and then asks the follow-up that most candidates cannot answer: how do you know the human check is real? This page builds that answer in five steps and closes with the numbers.

Human-in-the-loop for agents from first principles

Blast radius

Blast radius is the amount of damage a single action can do if it is wrong, measured by how many records, users or systems it touches and whether it can be undone.

Automation bias

Automation bias is the tendency of a person to accept an automated recommendation without checking it, which turns an approval step into a rubber stamp.

In plain words

Treat an agent like a new employee on day one. You give them a badge for the rooms they need. You let them tidy files freely. You ask them to check with you before wiring money. You review their work more closely at first. The analogy breaks in one place. A new employee has judgement and fear of consequences. An agent can be talked into a harmful action by text buried in a web page or email, an attack called prompt injection, so the controls must hold even when the model is fooled.

In one sentence: oversight for an agent is code outside the model that limits what it can do, routes risky actions to a person, and records every decision.

Step 1: Classify every action by reversibility and blast radius

List every tool the agent can call and sort each action into a tier. Read-only actions, such as search or read a record, cannot change anything. Reversible writes, such as creating a draft, can be undone. Irreversible or external actions, such as sending an email or issuing a refund, cannot be recalled. High-blast-radius actions, such as dropping a production database or changing permissions, can hurt many users at once, so the agent is never allowed them. Deleting a table in a non-production database stays in the irreversible tier and waits for one approval.

The classification is the input to every later step, and it is a property of the action, not of the model's confidence. The OWASP GenAI Security Project, publisher of the Top 10 for large language model (LLM) Applications, names the failure it prevents: excessive autonomy, where an application "fails to independently verify and approve high-impact actions".¹ The output of this step is a table: action, tier, policy.

1 · Classify the action2 · Limit permissions3 · Route by risk4 · Pause and resume5 · Log and auditReplay

Step-by-step description

1. Sort each tool action by reversibility and blast radius.
2. Remove capability first; every removed permission is an approval saved.
3. A gateway runs, logs, holds or denies each call by its tier.
4. Save state, show the raw action, bind approval to its arguments.
5. Record each decision and test reviewers with seeded canaries.

• OWASP names three root causes of excessive agency: excessive functionality, permissions and autonomy.
• A gateway outside the model routes each call by its risk tier.
• A held action saves state, shows the raw action, binds approval to the exact arguments and refuses on timeout.
• Seeded canaries measure how many bad proposals reviewers catch.

Step 2: Give the agent only the permissions it needs

Before adding any approval, remove capability. OWASP's Excessive Agency entry lists three root causes: excessive functionality, excessive permissions and excessive autonomy. Its examples include a database identity with update, insert and delete rights when only select is needed, and a per-user tool that connects with a generic high-privilege account.

The fixes are concrete. Expose narrow tools, such as a dedicated file-writing tool instead of an open shell. Run actions in the user's own context with minimal OAuth scopes. An OAuth scope is a narrow permission grant an application requests on a user's behalf. Enforce authorization in the downstream system, not in the prompt, because the model cannot be the access-control layer. Every permission removed here is an approval you never have to ask for, which matters in the worked example.

Step 3: Route each action by risk

Put the policy in a gateway that sits between the agent and its tools, written in ordinary code the model cannot edit. For each call the gateway looks up the tier and applies the rule: run read-only actions, run reversible writes with a log and an undo window, hold irreversible actions for one approval, and deny high-blast-radius actions outright, with no approval path for the agent. If a person ever has to do one, a separate two-person break-glass procedure outside the agent handles it.

Anthropic's guide to building agents describes agents that can "pause for human feedback at checkpoints or when encountering blockers", notes that stopping conditions such as a maximum number of iterations are common, and recommends extensive testing in sandboxed environments with guardrails. A step cap and a sandbox for code execution belong in the same gateway. The output is a decision per call: allow, allow with undo, hold, deny.

Text description of the diagram

• The gateway runs read-only actions, runs reversible writes with a log and an undo window, holds irreversible actions for one approval, and denies high-blast-radius actions outright.
• A held action saves state, shows the raw action, binds approval to the exact arguments and refuses on timeout.

Step 4: Pause, ask, and resume with the state saved

A held action must stop the agent without losing its place. Save the agent's state, the pending action and its exact arguments, then notify the approver. The approval screen should show what will happen in plain terms and the evidence the agent used, and it should show the raw action, such as the actual recipient and body, not only the agent's summary of it, because a summary can be wrong or manipulated.

Bind the approval to the exact parameters, the structured arguments of the call: if the agent changes the recipient after approval, the approval no longer applies. Set a timeout whose default is to refuse, never to proceed. On approval the agent resumes from saved state and performs only the approved action. This is the pause-ask-resume loop interviewers mean by a real checkpoint.

Step 5: Log every decision and audit the reviewer

Record each action, its tier, who approved it and when. Logs do not prevent harm, as OWASP says of logging and rate limiting, but they make review and rollback possible. The harder part is the reviewer. Parasuraman and Manzey's review of the research reports that automation bias occurs in both novice and expert participants and is not prevented by training or instructions, and that complacency is not overcome by simple practice.

So measure the human. Seed a small share of known-bad proposals, canaries, into the approval queue and track how many reviewers catch. The EU AI Act's Article 14 makes a related requirement for high-risk systems: overseers must be enabled to stay aware of the tendency to over-rely on outputs, and to override or stop the system.

Worked example

Take an illustrative agent that attempts 2,000 actions a day, with an assumed mix of 70% read-only, 20% reversible writes, 8% irreversible or external and 2% high blast radius, the tier the agent may never use. The 40 are attempts the gateway refuses, and each refusal should raise an alert, because a well-scoped agent should rarely attempt one.²

The 160 irreversible or external actions need one approval each, and the 40 high-blast-radius attempts are denied by the gate and need none, so 160 approvals a day make 4 hours of review at an assumed 90 seconds each.² If that load is too heavy, the fix is not more reviewers: make more actions reversible, which puts them in a lower tier, or remove the capability altogether.

Now test whether the review is real. Of the 160 gated actions, assume 2.5% are bad, so 4 a day. If reviewers catch 30% of bad actions, 2.8 slip through; at 60%, 1.6; at 90%, 0.4.²

Text description of the diagram

• Of the 160 gated actions a day, 4 are bad; if reviewers catch 30% of bad actions, 2.8 slip through.
• At a 60% catch rate 1.6 bad actions a day get through, and at 90% it is 0.4.

A canary rate of 5% gives 8 canaries a day. If 6 of 8 are caught, the observed catch rate is 75%, but 8 trials say little. A 95% confidence interval is the range of true catch rates that fits what you saw; the Wilson method, which stays accurate at small counts, gives 41% to 93%, too wide to act on.²

Here $\hat{p}=6/8$ is the observed rate, $n=8$ the canaries and $z=1.96$ the 95% multiplier. To pin the rate within 10 points either way you need this many canaries:²

$E=0.1$ is the margin, and $p=0.5$ is the worst case because it makes $p(1-p)$ largest. At 8 a day you pool about two weeks of canaries.²

Trade-offs and edge cases

Safety against friction. Every approval adds delay and cost. Gate by tier, not by agent, and keep approvals rare enough to be read.

Approval fatigue. Reviewers who see mostly correct proposals start accepting them. Rotate reviewers, keep the queue small and use canaries.

The reviewer's view can be poisoned. If the approval screen shows only the agent's summary, an attacker who controls the agent's input controls the summary. Show the raw action.

Per action or per plan. This is a design trade-off: approving a whole plan is cheaper, approving each risky step is safer when plans can change mid-run.

A model is not a human. A second model can screen inputs, as Anthropic notes, and that helps, but it is a guardrail, not the human check the question asks for.

How to answer it in an interview

The spoken structure is: classify, restrict, route through a gateway, pause and resume, then audit the human.

Common mistakes

"I would add a prompt telling the agent to be careful." (wrong) A prompt is a request, and a manipulated model can ignore it. (right) "I enforce the policy in a gateway and in downstream permissions."

"A human approves everything." (wrong) Volume produces rubber-stamping and delay. (right) "I gate by risk tier and shrink permissions to cut the load."

"The approval screen shows the agent's summary." (wrong) A wrong or manipulated summary defeats the check. (right) "It shows the raw action and binds approval to its arguments."

"Our reviewers agree with the agent 98% of the time, so it works." (wrong) High agreement may mean the agent is right or the reviewers are not looking. (right) "I measure the catch rate with seeded canaries."

Follow-up questions

Your AI agent deleted a production database. How do you prevent irreversible actions?

Remove delete and drop rights from the agent's database identity so a production drop is denied by policy, with no approval button for the agent; a person who must do it uses a two-person break-glass path outside the agent. Deleting a non-production table takes one approval, and soft delete and backups make deletion reversible. A sandbox keeps experiments away from production data. See the agents hub for the wider design.

How do you design an agent's identity, credentials and least-privilege authorization?

Run each action as the requesting user with narrow OAuth scopes, never a shared high-privilege account, and enforce authorization in the downstream service, since the model cannot be trusted to decide what is allowed.

How do you let an AI agent execute code safely?

Run the code in a sandbox with no access to production credentials, limited network, a time and memory cap and a throwaway filesystem. Avoid giving the agent an open-ended shell where a narrow tool would do.

Your reviewers agree with the AI 98% of the time. Is your human-in-the-loop real?

Not necessarily. Agreement does not distinguish a correct agent from inattentive reviewers, so seed canaries and measure the catch rate, as in the worked example above. Pair this with observability for large language model (LLM) apps to see what each approver saw.

What changes if the number of approvals doubles?

Review hours double, from 4 to 8 in the example, so reduce the gated share by making actions reversible or tightening permissions instead of hiring reviewers. Related practice: agentic AI interview questions.

Sources

1. LLM06:2025 Excessive Agency (OWASP Top 10 for LLM Applications) — OWASP GenAI Security Project, 5 May 2025
2. Building effective agents — Anthropic, 19 Dec 2024
3. Article 14: Human oversight (EU Artificial Intelligence Act) — Future of Life Institute, artificialintelligenceact.eu, 12 Jul 2024
4. Complacency and bias in human use of automation: an attentional integration — Human Factors (Parasuraman, Manzey), 1 Jun 2010