How do you design a computer-use / browser agent?

In this article

1. Before you start
2. Why interviewers ask this
3. Computer-use agent design from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the AI agent architecture learning path.

Why interviewers ask this

A computer-use agent shows where agent design meets the real world: a screen is messy, pages change, and one wrong click can spend money. Candidates report being asked to design one end to end, for example an agent that books travel or fills forms, and to say why browser agents are unreliable. Treat that as reported by candidates, not as a statement about any one company.

The question separates people who can draw the loop from people who can say what breaks. A weak answer is "the model looks at the screen and clicks." A strong answer names the harness that executes actions, the check after each action, a sandbox, a limit on context, and which actions need a human. A Staff-level answer also says how you would measure success, so the design is judged on task completion and not on a demo.

Computer-use agent design from first principles

Computer-use agent

A computer-use agent is a program that lets a language model operate a graphical user interface (GUI) by repeatedly viewing a screenshot, choosing one mouse or keyboard action and observing the result, until the task is done or a limit is reached.

Agent harness

An agent harness is the ordinary program around the model that sends it the screen, executes each action it proposes in a controlled environment, returns the outcome, and enforces limits such as step caps and confirmations.

GUI grounding

GUI grounding is the skill of mapping an instruction such as "click Search" to the exact place on the screen, usually a pixel coordinate, where that element sits.

In plain words

Picture a remote helper who can only see your screen through photos you send. They say "click the blue button at the top right." You click, then send a new photo. The helper never touches your computer. That is the whole design.

The analogy breaks in one place. A human helper notices a click did nothing. A model often does not, unless you make it look.

In one precise sentence, the model proposes actions as structured tool calls, and your harness is the only thing that acts on the machine. The OpenAI guide describes the same split: you provide the environment and execute the model's requests.

Step 1: The loop turns screenshots into actions

The loop is the core of the design. Your harness gives the model a task and a toolset, the model answers with one or more tool calls, the harness runs them in order and returns results, and this repeats until the model stops asking for tools. Anthropic's documentation calls this repetition the agent loop, and its toolset has 17 member tools such as screenshot, left_click, type and zoom.

Input: the task and the latest screenshot. Output: a proposed action. The model proposes one action from a screenshot, and only the harness acts. Two details matter. The model reads coordinates off the screenshot it was given, so if you shrink screenshots you must scale the coordinates back before clicking. And the loop needs a cap on iterations, to stop runaway cost.

You now have a working agent that can still fail silently, which the next step fixes.

Step 2: Verify each action before taking the next one

Verification is a check that looks at the result of each action and returns pass, retry or stop to the model before the next action is chosen. The docs warn that the model sometimes acts as if an action succeeded without checking, and suggest prompting it to take a screenshot after each step, state in one sentence what it shows, and retry if the step failed.

Input: the screenshot after an action. Output: pass, retry or stop. Back this up in code too. If the page did not change after a click, the harness can flag it. For a form, it can read the field back before pressing submit. For a purchase, it can require the confirmation page text.

Retries are safe only for idempotent actions, which give the same result if repeated, such as scrolling or typing into an empty field. For a non-idempotent action such as submit or pay, first re-observe the end state, for example look for a confirmation page or an order number, and retry only if the action provably did not happen. A result that was merely unobserved may already have charged or booked.

The docs also say a batch of actions must stop at the first failure, with the rest answered as not executed, so the model can replan. Verification is what turns a loop into a reliable one; the worked example shows how much.

Step 3: Run every action inside a sandbox

A sandbox is an isolated machine the agent cannot use to reach your real data, and the harness is the only bridge into it. The docs recommend a dedicated virtual machine (VM) or container with minimal privileges. A sandbox runs every action the harness executes.

Input: an action from the model. Output: the same action performed in the isolated desktop, plus a screenshot. Limit internet access to an allowlist of domains, and keep account logins and sensitive files out of the environment. The model never connects to the machine itself; your application receives the request, translates it, captures the result and returns it.

If the work stays inside webpages, a page-level browser tool can be the closer fit, because it needs no full desktop (the trade-offs below explain the choice). The model context protocol is another way to give an agent structured tools, which avoids pixels when an interface exists.

Step 4: Keep the context small on long tasks

Every screenshot you return stays in the conversation, so a long task fills the model's context window and quality drops, an effect covered in context rot. The fix is trimmed history, which keeps the context small: decide what the model must still see.

Input: the full history. Output: a trimmed history. A sound design keeps the task, a short running summary of what is done, the last few screenshots and any values the agent must remember, and drops older images. Store large facts outside the window, in a file or a note the agent can reread. Context folding studies compressing long trajectories this way.

The same idea answers the question of an enormous environment state, say ten million tokens (illustrative). The agent never loads it all. It navigates, reads one region at a time, and keeps notes. See also why agents fail on long tasks.

Step 5: Gate risky actions and distrust page text

Prompt injection is an attack in which text on a page or in an image gives the model instructions that conflict with yours. The docs say the model can follow commands found in content, and that webpage or image instructions can override yours.

Input: any content the agent reads. Output: no action taken only because that content said so. Treat everything on screen as data. Ask a human to confirm decisions with real-world consequences, such as accepting terms or completing a financial transaction, and make that check before each action runs, because a batch can finish a multistep action in one turn. A human confirms risky actions before they run. This is the same pattern as human in the loop: the agent proposes, a person approves the irreversible step.

1 · Loop2 · Verify3 · Sandbox4 · Context5 · ConfirmReplay

Step-by-step description

1. A screenshot goes to the model, which proposes one action.
2. A check returns pass, retry or stop to the model.
3. The harness runs actions in an isolated machine.
4. Trim old screenshots and keep a summary.
5. A human approves risky actions before they run.

• The model proposes one action from a screenshot.
• A check looks at the result of each action and returns pass, retry or stop to the model.
• A sandbox runs every action.
• Trimmed history keeps the context small.
• A human confirms risky actions before they run.

Worked example

Why does verification matter so much? Because errors compound. If each step succeeds with probability p and steps are independent, a task of n steps succeeds with probability p to the power n. The per-step rates below are illustrative, not measurements of any product.¹

At 95% per step (illustrative), a 10-step task succeeds 0.5987 of the time and a 20-step task 0.3585 of the time; the chance drops below one half at 14 steps. At 99% per step it takes 69 steps to fall below one half. Now add a check and one retry per step, and assume a retry fails independently and that the check catches every failure (both illustrative): 95% per step becomes 0.9975. With a verified retry, a 20-step task at 95% per step succeeds 0.9512 of the time.¹

Real failures are not independent, which is why you also need a hard stop. The numbers explain the design: shorten the path, verify each step and retry.

Text description of the diagram

• Illustrative task success at 95% per step is 0.3585 at 20 steps.
• At 99% per step a 50-step task succeeds 0.605 of the time.
• With a verified retry, a 20-step task at 95% per step succeeds 0.9512 of the time.

The OSWorld benchmark defined the task in 2024 (369 computer tasks, version 2 of May 2024): humans completed over 72.36% of them, and the authors found models struggled with GUI grounding and operational knowledge. Those are findings from 2024; they show the shape of the task, not today's scores.

Trade-offs and edge cases

The first choice is what the agent observes: pixels or the page's structure.

Screenshots or the page's structure?

A browser agent can read a screenshot or the structure of the page. The document object model (DOM) is the browser's tree of every element on a page, such as buttons, links and fields. The accessibility tree is a simplified version of it that gives each element a role and a name, as screen readers use. With structure, the harness lists the elements with references or labels, and the model answers "click element 12" instead of a pixel coordinate.

That is cheaper than sending images and more precise, because no coordinate is guessed. But structure misses a canvas and any non-web application, which have no elements to list. Screenshots reach any application on screen, but the model must do GUI grounding to find the right coordinates. Page-level tools read the page's elements through the DOM or accessibility tree and need no full desktop, but only cover webpages. A common hybrid reads structure where the page offers it and falls back to a screenshot for the rest.

Text description of the diagram

• Screenshots reach any application on screen.
• Page-level tools read the page's elements through the DOM or accessibility tree and need no full desktop.

Other cases to name

• Hard widgets. Dropdowns and scrollbars can be tricky with mouse movement; keyboard shortcuts are often steadier.
• Small text. Use zoom to read a region at full resolution.
• Waiting. Pages load at uneven speeds, so wait and verify before acting.
• Logins. Giving credentials raises prompt-injection risk, so prefer a session you prepared and avoid sensitive accounts.
• Cost. Each step is a model call with an image, so cap iterations and prefer batches that end with a screenshot.

To judge the whole system, measure task completion over many runs, not one demo; agent evaluation covers trajectories and reliability.

How to answer it in an interview

The spoken structure: draw the loop, name verification and why errors compound, then the sandbox, the context limit and the human gate.

Common mistakes

• "The model clicks the button." (wrong) The model only proposes the click. (right) "My harness executes every action in a sandbox."
• "If the click was sent, it worked." (wrong) Actions fail silently. (right) "I verify the screen after each action and retry or stop."
• "I keep every screenshot so the model has full history." (wrong) Context fills and quality drops. (right) "I keep a summary and the last few screenshots."
• "I just retry the failed submit." (wrong) The submit may already have gone through and a retry can charge or book twice. (right) "I check whether it already went through before retrying."
• "I trust what the page says." (wrong) Page text can carry injected instructions. (right) "Page content is data, and risky steps need human approval."

Follow-up questions

How do you build a computer-use agent, and what makes browser agents unreliable?

Browser agents are unreliable because errors compound over many steps, the model must ground instructions to screen coordinates, pages load unpredictably, and page content can hijack instructions.

How do you handle a huge environment state the agent must navigate?

Never load it whole. Let the agent read one region at a time, keep notes outside the context window, and trim old observations; see context folding.

How do you let an agent execute code safely?

Run it in a container or virtual machine with minimal privileges, no sensitive data and an allowlisted network. The harness, not the model, decides what is allowed to run.

How do you test a browser agent?

Run a fixed set of tasks many times, score completion with a check on the end state, and read the trajectories of failures. Agent evaluation shows how.

What if the task is 50 steps instead of 10?

Plain success at 95% per step falls from 0.5987 at 10 steps to 0.0769 at 50 (illustrative), while a verified retry per step keeps it at 0.8824.¹ Long tasks need checkpoints, hard stops and a shorter path.

Where does this fit in a general agent design?

It is one tool-using agent with an unusually risky tool. The general agent design page covers loop, memory and safety.

Sources

1. Computer use tool - Claude Platform Docs — Anthropic (undated page, read 2026-10-26), 26 Oct 2026
2. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv 2404.07972 v2) — arXiv (Xie et al.), 30 May 2024
3. Computer use | OpenAI API — OpenAI (undated page, read 2026-10-26), 26 Oct 2026