How do you design an agent evaluation harness?

In this article

1. Before you start
2. Why interviewers ask this
3. Agent evaluation harness from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page assumes you know what an AI agent is and can write a decent prompt.

Why interviewers ask this

A team that cannot measure an agent cannot safely change it, so evaluation design is where agent design rounds separate people who have shipped from people who have read about it. Candidates report questions about an agent that looks fine on one-turn tests but breaks in a longer conversation, and about an eval harness whose ground truth drifts when the underlying application programming interfaces (APIs) change, in interviews at AI labs and agent companies; no public source ties either question to a named company.

A weak answer lists metrics: accuracy, latency, cost. A strong answer says what is run, what is recorded, who grades what, and how the result gates a release. A Staff-level answer adds three points: agents are stochastic so every task needs repeated attempts, the environment's end state is the ground truth rather than the final message, and the suite splits into tests that measure new ability and tests that protect old behavior.

Agent evaluation harness from first principles

Agent evaluation harness

An agent evaluation harness is a system that runs an agent on a fixed set of tasks, records what it does, scores each attempt with graders and aggregates the scores into numbers a team can compare between versions, so that changes to the agent are judged by evidence rather than by a few hand-picked chats.

Trajectory

A trajectory is the ordered record of an agent's attempt: every model message, tool call, tool result and state change from the first step to the last. It is also called a transcript.

In plain words

Think of a driving test. The examiner does not ask you to describe your driving. They set a route, watch you drive it, and check where the car ended up and whether you broke a rule. An agent evaluation works the same way: tasks are routes, the sandbox is the road, and graders are the examiner's checklist. The analogy breaks in one place. A human driver is fairly consistent, while an agent can pass today and fail the same route tomorrow, so the test must be repeated.

In one precise sentence, an agent harness measures how often an agent reaches a correct, allowed end state on a set of tasks, at what cost, and with what behavior along the way.

Step 1: Define tasks and the success state

The Anthropic engineering guide on agent evals (published 2026-01-09) explains that agents differ from single-turn chat because they use tools, change external state and can fail across several turns even when the final answer sounds correct. That is why a chat-style test set is not enough, and why the four steps below exist. Companion reading on the metrics side is evaluating agents: trajectories, tool accuracy and reliability.

A task is one problem the agent must solve, written with its starting environment, its goal, a success state and a list of forbidden actions. A trial is one attempt at a task, and a grader is the scoring logic. A task file can be short; this one is an illustration:

task: refund-order
start_state: orders db with one paid order, id A-1042
goal: "Refund order A-1042 for the damaged item"
success_state: refunds table has A-1042 with amount 25.00
forbidden: [refund any other order, email the customer twice]

Input: real failures and business-critical workflows. Output: a task file per problem. Start small, with tasks drawn from real failures and the success state and forbidden actions written down explicitly. Write the success state as something checkable, such as "the refund row exists with the right amount" and not "the agent was helpful".

Split the suite in two. Capability tasks measure ability the agent does not reliably have yet. Regression tasks protect behavior that already works, and the guide says they should sit near a 100% pass rate, so a drop signals a real break.

Step 2: Repeat each task in a sandbox and record the trajectory

Each task is repeated in a sandbox and its transcript is recorded. A sandbox is an isolated copy of the environment, with test accounts and stubbed or recorded tool responses, so a bad action cannot touch real data and every attempt starts from the same state.

Repeat trials because agent behavior is stochastic. The Anthropic guide separates pass@k from pass^k as answers to different reliability questions, and Yao et al. (2024, arXiv 2406.12045) introduced pass^k to measure agent reliability. Here pass@k is the chance that at least one of k attempts succeeds, and pass^k is the chance that all k attempts succeed. With p the per-trial success rate:

Both formulas assume independent trials at one fixed p. From n recorded trials with c successes, pass@k is not estimated by plugging in c/n; the unbiased estimator of Chen et al. (2021, arXiv 2107.03374) is one minus the share of the k-subsets of the n trials that contain no success. Suppose, as an illustration, an agent solves a task 80% of the time. Its pass^3 is only 0.512, so a requirement of three clean runs in a row fails about half the time.¹

The outcome is the state the environment is left in when the attempt ends, and the trajectory is how it got there. Save both for every trial.

Give the runner controls, as common engineering practice: a step, time and token budget per trial, a reset of the environment between trials, and the model version, seed and temperature stamped on every trial record so a change between runs can be traced. For conversational tasks, the guide notes that evals often need a second language model to simulate the user, so the harness can drive a multi-turn conversation.

Step 3: Grade the outcome, the trajectory and the cost

Graders check the final environment state, not only the reply. Check the outcome first: query the database, read the file, run the test. A code grader for the refund task can be five lines (illustrative):

def grade_refund(db):
    row = db.get("refunds", order="A-1042")
    ok = row is not None and row["amount"] == 25.00
    return ok and not db.get("refunds", order="other")

Then grade the trajectory: were the right tools called with the right arguments, in a sensible order, with no forbidden action?

Combine grader types rather than relying on one channel, as the guide advises with its code-based, model-based and human graders: deterministic checks, model-based judges and human calibration. Deterministic checks are cheap and repeatable. Model judges handle open answers but need human calibration, because a judge can be wrong in a consistent direction. Humans review a sample of transcripts and set the ground truth the judge is tuned against.

Report outcome, process, safety and efficiency separately, because one aggregate score hides which of these moved. Input: a recorded trial. Output: scores per dimension. A related practice, review by a person at risky steps, is covered in human in the loop AI agents.

Text description of the diagram

• Deterministic checks are cheap and repeatable.
• Model judges handle open answers but need human calibration.

Step 4: Aggregate into reliability numbers and gate the release

Aggregate per task first, then across the suite: pass rate per task, pass@k, pass^k, tokens per attempt, latency per attempt, and cost per success, which is tokens per attempt divided by the pass rate. Compare each number with the previous version of the agent on the same tasks.

Then connect the harness to continuous integration (CI), the automated checks that run on every change. Treat evals like tests in CI. Run the regression suite on every change to the prompt, the tools or the model, and block the release when it drops. Run the capability suite less often and read it as a trend.

Input: graded trials. Output: a report and a pass or fail gate. You now have the full harness: tasks, repeated sandbox runs, layered graders, and a gated report.

1 · Define tasks2 · Repeat in a sandbox3 · Grade4 · Report and gateReplay

Step-by-step description

1. Each task has a checkable success state.
2. Every attempt is recorded.
3. Code, judge and human review score it.
4. Reliability and cost gate the release.

• Each task is repeated in a sandbox and its transcript is recorded.
• Graders check the final environment state, not only the reply.

Worked example

Take a suite of 30 tasks with 5 trials each, where one trial uses 12,000 tokens. All of these inputs are illustrative.¹ The run is 150 trials and 1,800,000 tokens.¹

If a task has a true success rate of 0.8, its cost per success is 15,000 tokens.¹ Now the noise: treating each of the 30 tasks as one pass/fail draw at 0.8, a Wilson 95% interval for the pass rate runs from about 0.627 to 0.905; the normal approximation gives 0.1431 either side, which is weak at this size.¹ Trials within one task are correlated, so the 150 trials do not buy the precision of 150 independent tasks. A gap of a few points between two versions is inside that noise unless you compare the versions task by task on the same tasks; that paired comparison has lower variance and is the right test.

pass@k asks whether at least one of k attempts succeeds. pass^k asks whether all k attempts succeed. The first suits a task where a retry is cheap, the second suits an agent that must work every time.

Text description of the diagram

• pass@k asks whether at least one of k attempts succeeds.
• pass^k asks whether all k attempts succeed.

Multi-turn drift compounds the same way. If each of six turns is handled correctly 97% of the time (illustrative), the whole conversation is right only 0.833 of the time, which is why single-turn tests pass while the agent falls apart by turn six.¹ A mixed suite of ten tasks at 0.95, ten at 0.8 and ten at 0.5 has an average pass rate of 0.75 but a pass^3 of 0.4981, which the aggregate hides.¹

Trade-offs and edge cases

Drifting ground truth. When the underlying APIs change, an expected answer copied from yesterday's response goes stale. Grade the end state through a rule rather than a stored string, keep recorded or stubbed tool responses under version control for the repeatable suite, and run a small live-contract check separately that flags when a real API no longer matches its recording. When it does, update the recording and re-baseline the scores on the old and new agent together, so a shift in the environment is not read as a shift in the agent. This is common engineering practice, not a figure from a cited study.

Judge calibration. A model judge needs a human-labeled sample to compare against; track their agreement and re-check it when the judge model changes. This page has no source for a target agreement figure, so set one with your team.

Grader bugs and shortcuts. A failing task can mean a broken task or a broken grader, not a bad agent, and an agent can find a shortcut that satisfies the grader without doing the work. The guide's advice is to read the transcripts of failures; do this for every task whose pass rate surprises you, and tighten the grader or the forbidden-action list when you find a shortcut.

Scale. For a company with 50 workflows, share one runner, sandbox and report format, and keep graders per workflow. At 30 tasks and 5 trials each, a full pass of all 50 workflows is 90,000,000 tokens (illustrative), so run the regression slice on every change and the full suite on a schedule.¹ In production, review the failing traces and turn the real failures into new tasks, so the suite follows what the agent meets in use.

Coverage versus cost. More trials tighten the estimate and raise the bill; spend them on the tasks near the pass line. The wider design question is covered in designing an AI agent system, and the tool layer in the Model Context Protocol page.

How to answer it in an interview

The spoken structure: tasks, repeated sandbox runs, layered graders, then the gated report, and finish with how you handle drift.

Common mistakes

• "The final message looked right, so it passed." (wrong) The state of the environment decides success. (right) "I check the database or file the agent was supposed to change."
• "It passed once, so it is reliable." (wrong) A single run is one sample from a random process. (right) "I run several trials and report pass^k next to pass@k."
• "I report one accuracy number." (wrong) It hides safety and cost regressions. (right) "I score outcome, process, safety and efficiency separately."
• "The model judge is ground truth." (wrong) A judge can be consistently biased. (right) "I calibrate the judge against a human-labeled sample."
• "Re-run the same stored answers after an API change." (wrong) The expected values are stale. (right) "I grade the end state by rule and re-baseline after a change."

Follow-up questions

How would you build the eval suite to confirm an AI agent meets its accuracy and cost targets?

Write the targets as numbers first, for example a pass^k floor and a maximum cost per success. Run the suite with repeated trials, compute both from the same runs and gate on both, since a cheaper agent that fails more often is not a win.

How would you design an eval harness for an agent system where the ground truth keeps drifting because the underlying APIs change?

Grade the end state by rules, not stored strings, run the repeatable suite on versioned recordings, and add a small live-contract check that flags drift. After a change, re-run the old and new agent together to re-baseline.

How would you build an evaluation platform for a company running 50 different agent workflows?

Share the runner, the sandbox interface, the trajectory format and the report; let each workflow own its tasks and graders. Gate each workflow on its own regression suite and track cost per workflow. For a deeper research-style workflow, see designing a deep-research agent.

Your single-turn evals pass but the agent falls apart by turn six. How do you catch that before launch?

Add multi-turn tasks driven by a simulated user, a second language model that plays the person, and run each several times. Grade the end state plus constraints on the transcript. The guide's example is a support ticket: resolved (a state check), finished in under 10 turns (a transcript constraint), and tone (a model-judge rubric). Small per-turn error rates compound across turns: 97% per turn is 0.833 over six (illustrative).¹ Read the trajectories of the failures to see which turn broke.

How do you evaluate an agent's trajectory and tool-use accuracy, not just its final answer?

Grade each tool call against an expected tool and argument rule, check the order and any forbidden action, and count wasted steps. Keep the outcome check separate so a good path to a wrong end state, or the reverse, is visible. More questions of this kind are in agentic AI interview questions.

Sources

1. Demystifying evals for AI agents — Anthropic Engineering, 9 Jan 2026
2. Evaluating Large Language Models Trained on Code (arXiv 2107.03374) — arXiv (Chen et al.), 7 Jul 2021
3. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv 2406.12045) — arXiv (Yao et al.), 17 Jun 2024