Why do agents fail on long tasks?

In this article

1. Before you start
2. Why interviewers ask this
3. Error compounding in agents from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A design for an agent that cannot say how often a long run finishes is a demo, not a system. The question separates candidates who treat accuracy as one number from those who see that a task is a chain of steps and that the chain, not the step, is what the user experiences.

A weak answer says "the model hallucinates" or "the context gets too long." A strong answer states the arithmetic of compounding, adds that a model's own earlier mistakes in its context make later mistakes likelier, and then names the levers: shorter chains, checks on each step, saved progress, a step cap and whole-run evaluation. A Staff-level answer also says which of these claims are measured and which are engineering judgment, as the rubric below shows. The same reasoning appears in the wider set of agentic AI interview questions.

Error compounding in agents from first principles

Error compounding

Error compounding is the effect by which a small failure rate at each step of a chain multiplies into a much larger failure rate for the whole chain, because one wrong step is enough to ruin the task.

Self-conditioning

Self-conditioning is the effect in which a language model becomes more likely to make a mistake when its context already contains its own mistakes from earlier turns, so errors raise the chance of further errors.

Task horizon

Task horizon is the number of steps an agent can chain before its chance of finishing the task falls to a chosen level, such as one half.

In plain words

Picture a relay of many runners. Each hand-off has a small chance of dropping the baton, and one drop ends the race. Even careful runners make a long relay risky. Per-step accuracy is the share of single steps the agent gets right when each is judged on its own.

The analogy breaks in one place. A runner who drops the baton does not make the next runner clumsier. An agent that makes a mistake leaves it in its context, and the next step reads it.

In one precise sentence: task success depends on the whole chain, and errors in the chain feed later steps.

Step 1: Independent step errors multiply

If each step succeeds with probability p, and one wrong step ruins the run, the chance that n steps all succeed is p multiplied by itself n times, written $p^n$. Input: p and n. Output: the task success rate. The authors of the long-horizon execution paper frame the metric an agent needs as how many consecutive steps it executes correctly, not whether it answers one question.

With per-step success of 0.99 (illustrative), task success at 10 steps is 0.9044 and task success at 100 steps is 0.366. With 0.95, task success at 50 steps is 0.0769.¹ A figure that sounds excellent per step is poor per task. You now know the baseline; Step 2 shows why reality is worse.

Text description of the diagram

• With per-step success of 0.99, task success at 100 steps is 0.366.
• With 0.95, task success at 50 steps is 0.0769.

Step 2: Self-conditioning makes later steps worse than the formula says

The formula $p^n$ assumes every step has the same, independent chance of failing. Sinha et al. (2026) tested this by handing the model the knowledge and the plan, so that only execution was measured. They report that per-step accuracy degrades as the number of steps grows, and that this is not just a long-context limit: "models become more likely to make mistakes when the context contains their errors from prior turns." They also report that this effect does not shrink just by making the model larger.

The consequence is that a wrong step left in the transcript is not harmless history; it is input to the next step. As a shape, not a measurement: if per-step success starts at 0.99 and loses 0.001 with every step (an illustrative assumption), a 50-step run succeeds 0.1719 of the time, against 0.605 for a constant 0.99.¹ The context rot page covers the related problem of long contexts degrading on their own.

Step 3: Small accuracy gains stretch the task horizon a lot

Because success is $p^n$, the number of steps at which success falls to one half is $\ln 0.5/\ln p$. With constant per-step success (illustrative), the task horizon is 6.6 steps at 0.90, 13.5 at 0.95, 69.0 at 0.99, 138.3 at 0.995 and 692.8 at 0.999.¹ Halving the slip rate roughly doubles the horizon.

The same paper's abstract states that even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks a model can complete. It also reports that larger models execute significantly more turns even when small models have near-perfect single-turn accuracy, and it argues that failures on longer simple tasks come from execution mistakes rather than an inability to reason. Short benchmarks can therefore hide real progress, and a high single-step score says little about how long a run an agent can hold. A goal-based agent that plans well can still fail in execution.

Kwa et al. (Model Evaluation and Threat Research (METR), 2025) measure a related 50% time horizon, the length of task (in human working time) an agent finishes half the time, and report that it has doubled about every seven months since 2019. Their horizon counts human minutes, not agent steps, so it is a cousin of the step horizon here, not the same number.

Step 4: Break the chain with checks, saved progress and a stop rule

Four levers act on the formula. First, use fewer steps: do deterministic work in ordinary code, so the model takes fewer decisions. Second, check each step against something outside the model, such as a test, a schema or a lookup, and redo a step that fails. Third, save progress and restart from a clean, short summary rather than carrying every past mistake; context folding is one approach to long histories. Fourth, cap the steps and hand over to a person when the cap hits, as in human in the loop agents.

Take a per-step success rate of 0.95 and a check that catches a share of slips and redoes them correctly (illustrative assumptions). A check catching half the slips lifts per-step success to 0.975 and 50-step success from 0.0769 to 0.28; a check catching 90% gives 0.995 per step and 0.78.¹

1 · Every step can slip2 · Slips stay in context3 · Small gains stretch runs4 · Checks cut the slip rateReplay

Step-by-step description

1. Success over n steps is p to the power n.
2. Earlier mistakes raise the chance of later ones.
3. A little more accuracy lengthens the horizon.
4. Verify, save progress and cap the steps.

• Task success is the per-step success rate raised to the number of steps.
• A slip left in the context raises the chance of later slips.
• A small gain in per-step success lengthens the task horizon.
• Checks that catch slips cut the failure rate of each step.

The Staff-level answer

What makes it Staff-level: it gives the arithmetic, separates the independent-slip model from the measured self-conditioning effect, and ends with levers instead of a diagnosis.

The rubric

Three variants of this question

Why does a long task fail when per-step accuracy looks fine?

Per-step accuracy is measured one step at a time, while the task needs every step to be right. The chain's success is the product of its steps, and past mistakes in the context lower later steps further.

Your single-turn evals pass but the agent falls apart by turn six. How do you catch that before launch?

Single-turn tests measure one step, not the chain. Write full-run scenarios of realistic length, score end-to-end success, and report success by turn number so a drop at turn six shows up. Repeat each scenario several times, because a task that succeeds 0.605 of the time passes eight repeats in a row only 0.018 of the time (illustrative).¹ Plant a wrong step on purpose to see whether the agent recovers; the agent evaluation page covers trajectory scoring.

What if per-step accuracy rises from 0.99 to 0.995?

The 50% task horizon doubles from 69.0 to 138.3 steps, and a 50-step task rises from 0.605 to 0.778 success.¹ A small gain on the step is a large gain on the task.

Common mistakes

• "If each step is very accurate, the agent is reliable." (wrong) Reliability belongs to the whole chain. (right) "Task success is the per-step rate raised to the number of steps, so I judge whole runs."
• "Failures are independent random slips." (wrong) The paper reports that earlier errors in context raise later errors. (right) "Slips are not independent, so the plain formula is optimistic."
• "A bigger context window fixes it." (wrong) The paper says the degradation is not just long-context limits. (right) "Long context is one cause and errors in context are another, so I keep the context clean and check steps."
• "The agent stops when it thinks it is done." (wrong) That trusts the component that may be wrong. (right) "Stopping depends on an outside check plus a hard step cap."

Follow-up questions

How do agents decide a task is done?

A strong design gives the agent a stop condition it can check against something outside its own text, such as a passing test, a filled-in field or a person's approval. It also adds a hard step cap so a missed condition cannot run forever.

How do you implement termination conditions in long-running agents?

Combine three stops: a success check, a step or time budget, and a stall rule that ends the run when several steps add no new information. On any stop, return the progress made and the reason, so a person or a retry can continue from there.

How do you detect and stop infinite planning loops?

Record each action with its arguments and flag a repeat when the same action recurs with no change in the state. On a flag, change the strategy once, then stop and escalate, with the step cap as the last guard.

How to make LLM agents reliable?

Shorten the chain, verify each step, save progress, keep the context short and score whole runs, as in Step 4. Choose where a person approves the risky steps.

Does a bigger or thinking model solve it?

Partly. The paper reports that larger models execute more turns, but that self-conditioning does not shrink just by scaling model size, while thinking mitigates it and lets a model run much longer tasks in a single turn.

Is the task horizon of agents getting longer?

Yes, on one measure. Kwa et al. (METR, 2025) report that the 50% time horizon of frontier models, in human task time, has doubled about every seven months since 2019. It is a different unit from the step horizon above.

Sources

1. The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (arXiv 2509.09677v3) — arXiv (Sinha, Arun, Goel, Staab, Geiping), 13 Mar 2026
2. Measuring AI Ability to Complete Long Tasks (arXiv 2503.14499) — arXiv (Kwa et al., METR), 18 Mar 2025