How do you evaluate agent trajectories and tool-use accuracy?

In this article

1. Before you start
2. Why interviewers ask this
3. Evaluating an agent trajectory from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Read the checklist below first if the agent loop is new to you; the judge rubric in Step 3 builds on prompt writing.

Why interviewers ask this

Agents fail in ways a single answer hides, so an interviewer asking how to evaluate one is testing whether you can look inside the run. Candidates report that agent interviews often start from a design question and then ask how you would know the system works, so this is a common follow-up to designing an AI agent system.

A weak answer says "check that the final answer is correct." A strong answer separates the path from the result, names how the path is compared with a reference, and says how tool calls are scored. A Staff-level answer adds the trade-off between deterministic matching and a judge, and the fact that one good run proves little. The sections below build that answer from zero.

Evaluating an agent trajectory from first principles

Match mode

A match mode is a rule that decides how closely an agent's tool calls must follow a hard-coded reference trajectory, for example in exact order, in any order, with extra calls allowed or with no extra calls allowed.

In plain words

Think of marking a driving test. A fair examiner does not only ask whether you reached the destination. The examiner watches the route, whether you checked mirrors before turning and whether you took a needless detour. The analogy breaks in one place: a driving test has a fixed route, while an agent often has several valid routes, so the marking rule must be chosen with care.

In one precise sentence, evaluating a trajectory means scoring the sequence of actions an agent took, not only the answer it ended with. The five steps below follow one worked example, a refund request.

Step 1: Record the trajectory

Log every run as a trajectory: the ordered messages, including tool calls, that the agent produces, which is the object every later score is computed on (LangChain trajectory evaluation docs, accessed 2026-10-26).

Input: one task given to the agent. Output: an ordered log of messages, each tool call with its name and arguments, each tool result and the final answer. If the log is not exact, there is nothing precise to compare.

The running example is illustrative. The reference trajectory for a refund task is three calls in order: lookup_policy with topic refund, then check_order with order id A17, then issue_refund with order id A17 and amount 40.¹ You now have the object to score; the next step decides how to score it.

Step 2: Compare the trajectory to a reference with a match mode

Choose a match mode to compare the run's tool calls with a hard-coded reference trajectory. Trajectory matching is deterministic, fast and cheap because it needs no extra LLM calls (LangChain docs).

The documentation lists four modes. The strict mode (exact order) needs the same messages and tool calls in the same order, and suits workflows such as a policy lookup before authorization. Unordered mode (any order) accepts the same tool calls in any order. Superset mode (extras allowed) needs at least the reference tools. Subset mode (no extras) allows only tools from the reference. By default, two tool calls are equal only when the tool and its arguments both match.

Reading the table:¹ a reordered run fails the exact-order check but passes the unordered check, and an extra call passes superset while failing unordered and subset. If a policy lookup must come before a refund, run B is wrong and strict mode catches it. If order is irrelevant, run B is fine and strict mode raises a false alarm.

Text description of the diagram

• A reordered run fails the exact-order check but passes the unordered check.
• An extra call passes superset while failing unordered and subset.

Step 3: Use an LLM judge when many paths are valid

An LLM judge is a language model that reviews an agent's decisions against a prompt rubric, which can include a reference trajectory (LangChain docs).

Input: the trajectory and a rubric, for example "the agent must check policy before refunding and must not call unrelated tools." Output: a verdict or score with a reason. The documentation describes it as more flexible, able to assess nuanced aspects such as efficiency and appropriateness, but it needs an LLM call and is less deterministic. It also does not require a reference trajectory, although one can be provided.

Use the judge where several different routes are correct, such as an open-ended research task, and matching where the correct sequence is known. Because a judge is less deterministic, keep the rubric specific and spot-check its verdicts against human review. The deep-research agent design is a typical case where no single reference path exists. You now have two scorers; the next step zooms into the calls themselves.

Step 4: Score each tool call on tool and arguments

Tool-use accuracy is the share of an agent's tool calls that match the intended call in both the tool chosen and the arguments passed. A call with the right tool and a wrong argument is still a wrong call.

Two simple scores follow. Call precision is the share of the agent's calls that match a reference call; call recall is the share of reference calls the agent made. These definitions are ours, not a vendor standard. In the example, run C has precision 0.75 and recall 1.0 because of its extra call, run D has precision 1.0 and recall 0.6667 because it stopped early, and run E has precision 0.6667 and recall 0.6667 because one argument is wrong.¹

Tool-name recall alone would score run E at 1.0¹ and hide the wrong amount, so report the full-call score. Also compare the end state of the system, such as the refund recorded in the database, because a correct path can still leave the wrong state; τ-bench (Yao et al., 2024) grades by comparing the final database state with the goal state.

Step 5: Repeat the run to measure reliability

Reliability is the chance that the agent succeeds every time, not once. If a run passes with probability p and runs are independent, the chance that all k runs pass is p raised to the power k. This is the pass^k metric of τ-bench, Yao et al., 2024.

With an illustrative per-run success rate of 0.8, passing 2 runs in a row has probability 0.64, 4 runs 0.4096 and 8 runs 0.1678.¹ With 0.95 per run, 8 runs in a row still pass with probability 0.6634.¹ The independence assumption is ours: real runs share causes, so measure the repeated rate directly. The τ-bench authors report that in the retail domain even strong function-calling agents had pass^8 below 25% (Yao et al., 2024).

One passing trajectory is an anecdote. Run each task several times, report the share of tasks that pass every run, and keep the failing trajectories for review. Where a wrong action is costly, human review of agent actions is the control that scores cannot replace.

1 · Record2 · Match3 · Or judge4 · Score calls5 · RepeatReplay

Step-by-step description

1. Log the exact messages and tool calls.
2. Compare to a reference with a match mode.
3. Or use a rubric when many paths are valid.
4. Check tool and arguments together.
5. Repeat runs to measure reliability.

• A trajectory is the exact sequence of messages and tool calls from one run.
• A match mode compares the trajectory to a reference trajectory.
• An LLM judge reviews the decisions against a rubric.
• Repeated runs measure reliability.

The Staff-level answer

What makes it Staff-level: it chooses the mode from the workflow, states the cost of each scorer, and treats reliability across repeated runs as part of correctness.

The rubric

Three variants of this question

How would you evaluate an agent that has no reference trajectory?

Use an LLM judge. The documentation states that a judge does not require a reference trajectory, though one can be supplied, so you write a rubric of the behaviours you expect and score each run against it.

How do you catch an agent that calls the right tool with wrong arguments?

Compare the tool and the arguments together. By default, two calls are equal only when both match, so a refund of 45 instead of 40 is a different call and fails.¹ The argument matching can be loosened where exact values are not required.

The agent passes 80% of runs. Is that production-ready?

Not on that number alone. If runs are independent, passing 8 in a row has probability 0.1678 at that rate, so ask how often the agent passes every run of the same task and what a failure costs.¹

Common mistakes

• "The final answer is right, so the agent did well." (wrong) The path may have skipped a required step or called extra tools. (right) "I grade the trajectory and the end state as well as the answer."
• "I always use strict matching." (wrong) It fails valid alternative paths. (right) "I use strict only for well-defined sequences and a judge or unordered mode otherwise."
• "The LLM judge is objective." (wrong) The documentation describes it as less deterministic. (right) "I keep the rubric specific and spot-check the judge against human review."
• "It passed once, so it is reliable." (wrong) Success on one run says little about every run. (right) "I repeat runs and report how often all of them pass."

Follow-up questions

What is the difference between trajectory evaluation and final-answer evaluation?

Final-answer evaluation scores only the result. Trajectory evaluation scores the sequence of messages and tool calls that produced it, so it catches skipped steps, extra calls and wrong arguments that a correct answer can hide. The broader agent evaluation page covers the wider toolkit.

When should you use an LLM judge instead of trajectory matching?

Use matching for a known required sequence, because it is deterministic, fast and cheap. Use a judge when several routes are valid or when you care about efficiency and appropriateness, accepting the extra call and lower determinism.

How do you measure tool-use accuracy?

Score each call on the tool name and the arguments together, then report precision and recall over calls against the reference. Report the full-call score, not the tool-name score alone, because wrong arguments are otherwise hidden.

What if the agent finds a valid shorter path?

Matching in strict mode marks it wrong. Switch to subset or unordered mode, or let a judge decide whether the path is acceptable, and add the new path to the reference set if it is valid.

How do you check that an agent is reliable, not just right once?

Run each task several times and count the tasks that pass every run, the pass^k idea. Per-run success of 0.8 gives only 0.1678 for eight independent runs in a row.¹ More agent questions of this kind are in the agentic AI interview questions.

Sources

1. How to evaluate your agent with trajectory evaluations (LangSmith, agentevals) — LangChain Docs (undated page; accessed 2026-10-26), 26 Oct 2026
2. agentevals: evaluators for agent trajectories (official repository) — LangChain (GitHub; accessed 2026-10-26), 26 Oct 2026
3. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv 2406.12045) — arXiv (Yao et al.), 17 Jun 2024