In this article
Before you start
Why interviewers ask this
An AI feature fails differently from a web service. The server stays up, the request returns a success status, and the answer is wrong, slow or expensive. If you cannot say which step caused it, you cannot fix it, and the interviewer learns that you have never operated one of these systems in production.
The question is usually phrased as "how do you build observability for an LLM application, and how does it differ from traditional monitoring?" or "how do you debug an agent that misbehaves?". A weak answer names a dashboard tool. A strong answer describes what is recorded per step, how traces are sampled and redacted, and which signals trigger an alert. This page builds that answer in five steps.
LLM observability from first principles
- Trace
- A trace is the complete record of one request as a tree of timed steps, all sharing one trace id, from the first call to the final answer.
- Span
- A span is one timed step inside a trace, such as a model call, a retrieval or a tool call, carrying a name, start and end times, a parent link and key-value attributes.
- Sampling
- Sampling is the rule that decides which traces are stored, because at high traffic storing every trace gets expensive and a careless rule drops the traces you need.
In plain words
Picture a parcel tracked through a depot. Every scan, at every shelf and belt, is logged with a time. When a parcel arrives late, you read its scans and see where it sat. A trace is that scan history for one user request, and each scan is a span. The analogy breaks in one place: a parcel's route is fixed, while an agent decides its own route at run time, so the same question can produce a different trace tomorrow.
In one sentence: observability is the ability to explain a system's behaviour from the records it emits, and for a language-model application the records must include what the model saw, chose and cost at each step.
Step 1: Wrap every step in a span
A span is created around each unit of work: the guardrail check, the retrieval, each model call, each tool call. According to the OpenTelemetry trace documentation, a span has a name, a parent span id (empty for the root), start and end timestamps, a span context, attributes, events, links and a status.
The input to this step is a function call in your code; the output is a timed record around it. Instrumenting at the library layer, so that every model client and tool wrapper emits spans, is how teams avoid forgetting a step. The Dapper paper (Google, 2010) says two design choices were key to its success: sampling, and restricting instrumentation to a small number of common libraries.
Step-by-step description
- Wrap the guardrail, retrieval, each model call and each tool call.
- Record model, prompt version, tokens, finish reason and tool name.
- A shared trace id and parent ids join the spans into one tree.
- Keep every error and slow trace; mask personal data before storing.
- Count every request before sampling; alert on percentiles, rates, cost and quality.
- A span has a name, a parent span id, start and end timestamps, attributes, events, links and a status.
- Spans share a trace id and each child stores its parent's span id, which forms a tree.
- In the simulation, head sampling at 10% retained 9.2% of error traces; a tail rule (all errors, all requests over the p99, 5% of the rest) kept every error and slow request and stored 6.9%.
- Count rates on every request before sampling; stored traces explain, they do not measure.
Step 2: Attach the facts that explain behaviour
A bare timing says a step was slow but not why. Each span therefore carries attributes: the model name and version, the prompt template version, input and output token counts, the finish reason, tool name and arguments, retrieved document ids and any cost computed from tokens. OpenTelemetry's GenAI conventions repository describes standard names for "spans, metrics, and events for GenAI clients" and for the Model Context Protocol; its README does not state a stability status, so check the current definitions before depending on exact attribute names.
Two attributes pay for themselves quickly. The prompt version lets you match a quality drop to a deployment, and the token counts let you attribute cost to a feature. The output of this step is the same span, now explanatory.
Step 3: Link the spans into one trace tree
Every span in a request shares the trace id, and each child stores its parent's span id, which forms a tree; spans with the same parent are siblings (OpenTelemetry). For an agent the tree shows the loop: plan call, tool call, answer call, with branches where the agent retried or chose another tool.
The tree answers the question "where did the time and the tokens go?" and, for a wrong answer, "what did the model see when it decided?". Propagate the trace id across every service boundary, including calls to a vector database or a tool server, or the tree breaks into disconnected pieces. This is the same discipline as tracing any distributed system; only the span contents are new.
Step 4: Sample the traces and redact what must not be stored
At high traffic, storing everything gets expensive. In an illustrative service with 1,000,000 requests a day and 14 KB per trace, keeping all traces is 14 GB a day, and keeping 10% is 1.4 GB.1 A blind 10% rule keeps only about 10% of the failures too. In a simulation of 100,000 requests with a 1% error rate, head sampling at 10% retained 92 of 1,005 error traces, a recall of 9.2%.1
The fix is a tail rule: decide after the request finishes, keep every error and every slow trace, and keep a small random share of the rest. In the same simulation, the rule "every error, every request slower than the p99, and a random 5% of the rest" stored 6,865 of 100,000 traces (6.9%) and kept 100% of the 1,005 errors and 100% of the 1,000 slow requests.1
Before storage, redact. Prompts and tool arguments can contain personally identifiable information (PII), secrets and customer data. Store hashes, ids or masked text, and keep full payloads only under a short retention and tight access.
Step 5: Count every request for signals; use stored traces to explain
Signals are counted on every request, before the sampler, so they stay true: a latency histogram, error and refusal counts, and tokens and cost per request. Stored traces then explain the cases the signals flag. Counting from stored traces would mislead: in the simulation above, the error rate read from the tail-stored traces is 1,005 of 6,865, or 14.6%, against a true 1.0%.1 The Google site reliability engineering (SRE) book names four golden signals: latency, traffic, errors and saturation, and says to track failed and successful latency separately. It also warns that averages hide a slow tail and recommends percentiles; its example is a 100-millisecond average with 1% of requests taking 5 seconds.
For a language-model feature add quality signals: refusal rate, a judge-model score on a sampled slice, retrieval hit rate and cost per request. User thumbs up or down is feedback you attach to a trace, not something computed from it. Alert on percentiles and rates, not means. Finally, route failed traces into your evaluation set, so today's incident becomes tomorrow's regression test.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
One agent request, with illustrative numbers, produces five spans that run one after another.1
| Span | Kind | Input tokens | Output tokens | Duration (milliseconds) |
|---|---|---|---|---|
| guardrail_check | model call | 420 | 12 | 310 |
| retrieve_docs | retrieval | 0 | 0 | 140 |
| plan_call | model call | 1,850 | 160 | 1,900 |
| tool:search_orders | tool | 0 | 0 | 620 |
| answer_call | model call | 2,600 | 220 | 2,400 |
Text description of the diagram
- The three model calls take 4,610 of the 5,370 milliseconds in the illustrative trace.
- The tool call takes 620 milliseconds and retrieval takes 140 milliseconds.
The totals are 4,870 input tokens, 392 output tokens and 5,370 milliseconds of wall time. The three model calls account for 85.8% of the time; the tool call takes 620 milliseconds and retrieval takes 140 milliseconds. With illustrative rates of 3 cost units per million input tokens and 15 per million output tokens, the request costs about 0.0205 cost units, and the 392 output tokens, only 7.5% of all tokens, cause 28.7% of that cost. The reading for an engineer: shorten the prompts of the two large calls and cap output length before touching the tool.
Now the tail. In a seeded illustrative sample of 10,000 request latencies, the mean was 2,887 milliseconds, the median 2,502, the p95 6,166 and the p99 8,628.1
Text description of the diagram
- In the illustrative sample the mean latency is 2,887 milliseconds and the p99 is 8,628 milliseconds.
- The median is 2,502 milliseconds and the p95 is 6,166 milliseconds.
A dashboard showing only the mean would look healthy while one request in a hundred waited 8.6 seconds or more in this illustrative sample.
Trade-offs and edge cases
Privacy against debuggability. The payload that explains a failure is often the payload you should not keep. Reported practice differs between teams that store full prompts for a short period under access control and teams that store only hashes and metadata; the choice depends on the data your users send and on the rules that apply to it.
Sampling bias. A tail rule over-represents failures, so rates computed from stored traces are wrong unless you record the sampling weight. Compute headline rates from counters that see every request, as Step 5 does, and use traces for explanation.
Non-determinism. Replaying a stored prompt may not reproduce the failure, because sampling temperature and model updates change outputs. Store the model version and the exact inputs, and treat replay as evidence, not proof.
Standards move. The OpenTelemetry GenAI conventions are an active repository; pin the version you use and expect attribute names to change.
Quality is not an error code. A wrong answer returns a success status, which the SRE book calls an implicit error. Only a quality signal catches it.
How to answer it in an interview
How do you build observability for an LLM application, and how does it differ from traditional monitoring?
I trace every request as a tree of spans: guardrail, retrieval, each model call, each tool call. Each span records model and prompt version, token counts, finish reason and redacted inputs. I sample with a tail rule that keeps all errors and slow traces plus a small random share. On top I alert on latency percentiles, error rate, cost per request and quality signals such as refusals and sampled judge scores. It differs from classic monitoring because a wrong answer returns a success status, so I need quality signals, not only uptime.
The spoken structure is: spans and attributes, then sampling and redaction, then signals, ending on the one difference.
Common mistakes
"We log the final answer and the thumbs rating." (wrong) That cannot show which step failed. (right) "We log a span per step with its inputs, tokens and version."
"Average latency looks fine." (wrong) The mean hid a p99 three times larger in our sample. (right) "I alert on p95 and p99, and on error and cost per request."
"We keep every trace for a year." (wrong) The storage bill and the privacy exposure both grow with it. (right) "Tail-sample, redact, and keep payloads for a short window."
"A 10% random sample is enough." (wrong) It keeps about 10% of the failures too. (right) "Keep every error and slow trace, and sample the healthy ones."
Follow-up questions
How do you debug an agent that is misbehaving in production?
Pull the trace of a bad request and read the tree top to bottom: what the model saw at each decision, which tool it chose and what came back. Then reproduce with the stored inputs and model version, and add the case to your evaluation set. For the failure side of the same problem, see human-in-the-loop controls for agents.
What is your observability stack for LLM features?
Describe layers, not brands: instrumentation emitting OpenTelemetry-style spans, a trace store, a metrics store for rates and percentiles, a place for sampled quality scores, and dashboards with alerts. Name your own tools only if you have run them.
How do you monitor and evaluate AI agents in production, and how do evals differ from observability?
Observability records what happened on live traffic; evaluation scores behaviour against expected results. Production monitoring feeds evaluation with real failures, and evaluation gives monitoring a quality score to alert on.
What do you log, and what must you never log?
Log ids, versions, token counts, timings, tool names and redacted text. Never keep secrets, raw credentials or unmasked personal data in a long-retention store.
What changes if traffic grows tenfold?
Storage and cost scale with requests, so the sampling rate must fall while the rule keeping every error stays. Move the headline metrics to counters that see all traffic. Related reading: the machine learning operations (MLOps) hub and the system design hub.
- A trace is a tree of spans, one per model call, retrieval or tool call, joined by a trace id.
- Attributes such as prompt version and token counts are what make a span explain behaviour.
- Tail sampling can keep every error and every slow trace; blind head sampling keeps about the same share of errors as of everything.
- Count rates on every request before sampling; stored traces explain, they do not measure.
- Alert on percentiles, error rate, cost and quality signals, never on the mean alone.
- A wrong answer returns a success status, so quality signals are required.
Sources
- Traces (OpenTelemetry concepts) — OpenTelemetry, 24 Oct 2026
- OpenTelemetry GenAI semantic conventions (repository README) — OpenTelemetry (GitHub), 24 Oct 2026
- Monitoring Distributed Systems, in Site Reliability Engineering — Google, 1 Apr 2016
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure — Google Research, 1 Apr 2010
