What is an LLM agent coding assignment, and how do you build one?

In this article

1. Before you start
2. Why interviewers ask this
3. LLM agent coding assignment from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page belongs to the AI agent architecture learning path. The wording "LLM agent coding assignment" is used in two ways: a task where you build an agent, and a task where a coding agent does a programming job. The loop below is the same in both.

Why interviewers ask this

A coding agent can look finished while failing in ways a reviewer finds in minutes: an endless loop, a tool call nobody checked, a green test run on the wrong file. A task of this kind therefore separates people who can call a model from people who can control one. No public standard defines the format and tasks vary, so this page explains the skills such a task can exercise; it does not report any company's task.

A weak submission is a long prompt and a hope. A strong one shows a plain loop, a few well-described tools, tests as the source of truth, a stopping rule and a note on what could still go wrong. A Staff-level answer adds cost, error compounding and the point where a human reviews. The wider design view is in how to design an AI agent system.

LLM agent coding assignment from first principles

Stop reason

A stop reason is a field in the model's response that says why it stopped writing, such as tool_use when it wants a tool run or end_turn when it has finished; the loop in your code reads it to decide what to do next. These names are the Claude application programming interface (API)'s; other model APIs expose an equivalent finish or stop field.

Agent-computer interface

An agent-computer interface (ACI) is the set of tools, commands and output formats through which a language model agent reads and changes a computer environment, designed for the model's needs the way a user interface is designed for a person; for a coding agent, file viewers, editors and a test runner.

In plain words

Picture a contractor on the phone who cannot touch anything on site. You are the site assistant. The contractor says "read the log file", you read it and report back, and the contractor decides the next request. The analogy breaks in one place: the model keeps no memory between calls, so each request must carry the whole conversation again.

In one precise sentence, agents are, in Anthropic's words (2024), "typically just LLMs using tools based on environmental feedback in a loop." An influential early form is ReAct (Yao et al., 2022), which interleaves reasoning with actions so that plans can be updated and exceptions handled.

Step 1: The model asks for a tool

Your code starts the loop by sending the task, the conversation so far and a list of tool definitions. The Claude documentation on how tool use works calls the canonical shape a while loop keyed on stop_reason. The reply holds text, one or more tool_use blocks, or both. These names are the Claude API's. Per the tool-use tutorial, each block carries the tool name, a unique id and the structured input.

Input: the task and the tool definitions. Output: one or more tool calls. Nothing has run yet, because the model has only written a request. For client tools, your application runs the operation, so every tool call is a round trip: the model asks, you execute, you report back, the model continues.

Step 2: Your code runs the tool and reports back

Your code reads the tool name and input, runs the matching function, and wraps the output in a tool_result block. The block's tool_use_id must match the id from the call, as the tutorial states. Then it sends a new request holding the original messages, the model's reply and the result. When the model requests several tools in one reply, your code returns every tool_result, each with its matching id, in the next user message, as the overview and how tool use works pages describe.

Input: one or more tool calls. Output: the tool results and a new request. This is where tool design matters. Anthropic recommends investing in the agent-computer interface through tool documentation and testing. The authors of SWE-agent report that a custom interface significantly improves an agent's ability to edit code files, navigate repositories and execute programs. Run tools in a sandbox, with a timeout for each.

Step 3: Tests give the loop feedback it can trust

Coding suits agents because, per Anthropic, code solutions are verifiable through automated tests and agents can iterate using test results as feedback. The test runner is just another tool: a failing run returns the error text, and the model reads it as the next observation.

This is the ReAct pattern in practice: thought, action, observation. Reasoning text alone changes nothing outside the model; only actions do, and observations tell the model what they did. So the loop should trust test output, never the model's own claim that it succeeded. Tests are not the whole check, though: Anthropic notes that human review remains crucial for ensuring a solution fits the broader system, which is the subject of human in the loop AI agents.

Step 4: The loop stops on stop_reason, not on the model's text

The loop continues while stop_reason is tool_use. It exits on any other value: end_turn, max_tokens, stop_sequence or refusal, per the documentation. A careful loop handles each exit differently: end_turn means finished, while max_tokens means the reply was cut off and needs handling.

Add a stopping condition such as a maximum number of iterations to keep control, as Anthropic suggests. Treat it as a safety stop, not as the test of completion. When the server runs the loop for you, it has its own iteration limit and the server loop can pause and return pause_turn.

1 · The model asks2 · Your code runs it3 · Tests give feedback4 · end_turn ends itReplay

Step-by-step description

1. The request goes to the model, which replies.
2. Read stop_reason, then run the requested tool.
3. Test output is one tool_result, sent back to the model.
4. Anything other than tool_use ends the loop.

• The model asks for a tool by returning a tool_use block.
• Your code runs the matching function and wraps the output in a tool_result block.
• Agents can iterate using test results as feedback.
• The loop continues while stop_reason is tool_use.

Agent or workflow: which should the assignment use?

A workflow is a system where LLMs and tools are orchestrated through predefined code paths, while in an agent the model dynamically directs its own process and tool use. Anthropic says agentic systems often trade latency and cost for better task performance, and advises starting with the simplest solution, often a single model call with retrieval and examples. If the steps are known in advance and the same every time, such as a fixed pipeline of format, test and report, write a workflow. Reach for an agent only when the next step depends on what the last tool returned.

Text description of the diagram

• A workflow is a system where LLMs and tools are orchestrated through predefined code paths.
• In an agent the model dynamically directs its own process and tool use.

Worked example

Take a small assignment: fix one failing test in a repository. The trace below is illustrative: the tool names and the turn count are choices made for this example, but its shape follows the documented loop.¹

Turns one to four each return tool_use, so the loop runs a tool and calls again. Turn five returns end_turn, so it exits. The second test run is what turns "I edited the file" into "the edit worked".

Length costs reliability. If each step succeeds with probability p, and steps are independent with no retry, a task of n steps succeeds with p multiplied by itself n times. With an illustrative per-step probability of 0.95, ten steps succeed about 0.60 of the time and twenty steps about 0.36.¹ This is arithmetic, not a measurement, and it is a worst case for loops without checks: in a real loop, test feedback lets the model catch and retry a failed edit, which is why Step 3 matters. It shows why Anthropic warns that the autonomous nature of agents means higher costs and the potential for compounding errors.

Text description of the diagram

• At 0.95 per step, ten steps succeed about 0.60 of the time and twenty steps about 0.36.
• More steps lower the chance that every step succeeds.

More steps lower the chance that every step succeeds, so keep the tool set small and let tests catch failures early.¹

Trade-offs and edge cases

An agent buys flexibility with cost and risk. Each loop turn is another model call, so latency and spend grow with the turn count. Errors compound, as the table shows. Anthropic advises testing in sandboxes with guardrails. Log the trajectory, meaning each tool call, its result and the token count per turn, so a reviewer can see what the loop did and what it cost.

Frameworks are optional. Anthropic suggests starting with the model's programming interface directly, because many patterns take a few lines, and warns that frameworks can obscure prompts and responses and make debugging harder. For a comparison of such tools see LangGraph, CrewAI and AutoGen.

The iteration cap is a design choice. No authoritative number exists, so pick one for the task, say so, and explain what happens when it is hit (an assumption such as ten turns for a one-test fix is fine if labeled).

Scaffolding matters, but be careful with benchmarks. SWE-bench, a benchmark of real software-engineering tasks, has 2,294 problems drawn from real GitHub issues across 12 Python repositories; the model writes a patch that is graded against real tests. Its original paper reported that the best model, Claude 2, solved 1.96% of issues. SWE-agent reported in 2024 that it solves 12.5% of issues on it. Those two figures use different models, so they are not a controlled test of the interface. SWE-bench Verified is a subset of 500 problems that professional developers screened as well specified and solvable. Current leaderboard numbers are not given here. To see how such agents are scored in general, read evaluating agents.

How to answer it in an interview

The spoken structure: name the loop, say who runs the tools, say how tests give feedback, then give the stop rule and the risk.

Common mistakes

• "The model runs the code." (wrong) For client tools, the model only writes a request. (right) "The model asks, my application executes and reports back."
• "I stop when the model says it is done, or after a fixed number of turns." (wrong) The model's text is not a stop signal, and a turn cap is only a safety stop; the documented stop signal is the API's stop field, stop_reason in the Claude API. (right) "I exit on any stop reason other than tool_use; the iteration cap is a safety stop."
• "Passing tests means the change is correct and mergeable." (wrong) Tests verify function, not fit with the wider system. (right) "Tests are the feedback; a human reviews before merge."
• "More autonomy is always better." (wrong) Agents trade latency and cost for performance. (right) "I use the simplest design that works and add the loop only when the steps cannot be known in advance."

Follow-up questions

What is the difference between an agent and a workflow?

In a workflow, code fixes the path. In an agent, the model decides its own steps and tool use. Choose a workflow when the steps are known upfront.

Does the model run the tool itself?

No. For client tools the model returns a structured request and your application runs it, then sends back a tool_result. Tool definitions can also come from a shared standard, covered in the Model Context Protocol page.

How many iterations should the loop allow?

No source sets a standard number. Choose a cap from the task's size, state it as an assumption, and decide what the loop does when it is reached.

Do you need a framework to build an agent?

No. Anthropic suggests starting with direct calls to the programming interface, because frameworks can hide prompts and responses and make debugging harder.

What does SWE-bench measure?

It measures whether a model can resolve a real GitHub issue with a patch that passes the repository's tests. That differs from short self-contained puzzles. The agentic AI interview questions page lists more questions of this kind.

Why do passing tests not prove the code is ready to merge?

Tests check the behavior they cover. Human review still checks whether the change fits the broader system. For how two coding agents handle this in practice, see coding agents compared.

Sources

1. Building effective agents — Anthropic Engineering, 19 Dec 2024
2. How tool use works — Claude Platform Docs (Anthropic), 26 Oct 2026
3. Tool use with Claude: overview — Claude Platform Docs (Anthropic), 26 Oct 2026
4. Tutorial: Build a tool-using agent — Claude Platform Docs (Anthropic), 26 Oct 2026
5. ReAct: Synergizing Reasoning and Acting in Language Models (arXiv 2210.03629) — arXiv (Yao et al.), 6 Oct 2022
6. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv 2310.06770) — arXiv (Jimenez et al.), 10 Oct 2023
7. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (arXiv 2405.15793) — arXiv (Yang et al.), 6 May 2024
8. Introducing SWE-bench Verified — OpenAI, 13 Aug 2024