LLM latency incident debugging: percentiles, queue wait, prefill and decode

In this article

1. Before you start
2. Why interviewers ask this
3. LLM latency incident debugging from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference optimization guide, and it assumes you know how a transformer produces text one token at a time.

Why interviewers ask this

A latency incident question tests whether a candidate has a method before they have a theory. Candidates report this kind of prompt in model-serving and machine learning (ML)-infrastructure rounds: a chatbot that was fast yesterday is slow today, and the interviewer wants your first moves.

A weak answer lists remedies at random: add graphics processing units (GPUs), shrink the model, cache more. A strong answer measures first, separates the time to first token (TTFT) from the inter-token latency (ITL), names the phase that grew, and only then picks a fix. A Staff-level answer also says how to verify the fix and which trade-off it buys, such as per-user speed against total throughput. The rest of this page gives that method in four steps.

LLM latency incident debugging from first principles

Inter-token latency

Inter-token latency (ITL) is the time between two consecutive streamed output tokens. It measures how fast the decode phase runs once the first token has appeared, so a rising ITL makes text stream slowly even when the first token arrived on time.

Preemption

Preemption is the pausing of a running request to free KV cache space for other requests. In vLLM V1 (recompute by default) the preempted request is recomputed when enough space returns, so the cost is extra latency, not a dropped request.

In plain words

Picture a restaurant. Customers wait at the door. That is queue wait. The kitchen reads each order and preps everything. That is prefill. Then plates come out one at a time. That is decode.

A slow evening has three possible causes: a long line, a huge order or a slow kitchen. Each needs a different fix. The analogy breaks in one place. A real server cooks for many tables in one batch, so one huge order can slow every other table.

In one precise sentence: a latency incident is a rise in the time requests spend in one of three phases, and debugging finds which phase.

Prefill and decode, the two phases of a request

Prefill is the first phase, in which the server reads the whole prompt in parallel and builds the KV cache. Decode is the second phase, in which the server writes the answer one token at a time.

Agrawal et al. (2024) describe the difference: prefill iterations have high latency but saturate GPU compute, while decode iterations have low latency and low compute utilization because each processes a single token per request. Prefill is compute-bound because many prompt tokens go through large matrix multiplications together, while decode is memory-bound because each step must read the model weights and the KV cache from memory to produce one token per request. That asymmetry is why one slow prompt can disturb many running answers, as Step 3 shows.

Which numbers show each phase

The vLLM metrics design doc lists histograms for time to first token, inter-token latency and end-to-end request latency. Time per output token (TPOT) is the average decode time per token for one request, computed as (end-to-end latency minus TTFT) divided by (output tokens minus one). TPOT and ITL are related but not identical, because ITL is measured between streamed outputs and TPOT is derived per request.

Server-level histograms aggregate all requests. Current vLLM also exposes them per phase: vllm:request_queue_time_seconds, vllm:request_prefill_time_seconds and vllm:request_decode_time_seconds, plus the gauges vllm:num_requests_waiting and vllm:num_requests_running. A preemption counter exists as well; the v0.8.2 design doc names it vllm:num_preemptions_total, so check your version's metrics page for the exact name, and for the prefix-cache hit-rate gauge. The per-request metrics then isolate a single slow request, such as its queue time and its mean inter-token latency.

For an illustrative request with an end-to-end latency of 13.8 seconds, a TTFT of 7.5 seconds and 100 output tokens, TPOT is about 63.6 milliseconds.¹

Step 1: Confirm the slowdown with percentiles, not averages

A latency incident is confirmed with percentiles of per-request timings, because a mean hides the slow tail. NVIDIA's benchmarking blog puts it this way: "a server with a healthy mean TTFT and an outlier p99 looks fine in aggregate and fails in production".

Take 1,000 illustrative requests, where 985 take 700 milliseconds to the first token and 15 take 9,000. The mean is 824.5 milliseconds, which looks healthy, while p99 is 9,000.¹ Chart p50, p95 and p99 for both TTFT and ITL, scoped by model, route and start time, and compare with the same hour last week.

Measure on streamed responses, because a non-streamed response arrives whole and has no separate first token or gaps between tokens. Name the benchmarking tool when you compare numbers, since NVIDIA's documentation notes results can vary between tools.

Step 2: Split a slow request into queue wait, prefill and decode

A slow request spends its time in three places, queue wait, prefill and decode, and the first-token delay covers the first two plus network time. NVIDIA's documentation says that "time to first token generally includes both request queuing time, prefill time and network latency", and that the longer the prompt, the larger the TTFT.

So the split starts with the aggregate numbers. Compare TTFT and ITL against their baselines: the one that moved tells you which side of the first token to inspect. If TTFT moved, compare the queue-time and prefill-time histograms and the waiting-requests gauge against their baselines: queue time that grew with a rising waiting count means requests waited; prefill time that grew means the prompts or the cache hits changed. A drop in the prefix-cache hit rate is a classic cause of a TTFT spike with flat ITL. Then read one slow request's own queue time: if that is most of its TTFT, it waited; if small, the cost was prefill or the network.

You now know which phase grew. Step 3 turns that into a cause.

Step 3: Match the symptom to a cause

A slow first token with steady gaps between tokens points to queueing or prefill. Slow gaps between tokens with a healthy first token point to the decode phase; as the NVIDIA blog puts it, "high ITL means the decode phase is struggling, even if TTFT looks healthy".

Three causes are the first to check. First, load: more requests than capacity grow the queue, so read the waiting count and the batch size. Second, prompt mix: longer prompts mean longer prefill, so read the prompt length percentiles. Third, KV cache pressure: vLLM can preempt requests to free cache space and recomputes them later, which adds latency without dropping them.

Decode slows when long prefills share the batch. Agrawal et al. (2024) note that batching multiple requests leads to an interleaving of prefill and decode iterations, and their Sarathi-Serve system builds chunked-prefill schedules that add new requests "without pausing ongoing decodes". This is the multi-tenant noisy-neighbour pattern; see multi-tenant model serving.

Text description of the diagram

• A slow first token with steady gaps between tokens points to queueing or prefill.
• Slow gaps between tokens with a healthy first token point to the decode phase.

Step 4: Apply the smallest safe mitigation, then verify

The smallest safe mitigation is the one you can undo quickly and that targets the phase you found. The triage order on this page is a working method, not a standard.

For queue growth, shed or rate-limit traffic, add replicas, or roll back the deploy that changed traffic. For cache pressure, the vLLM tuning guide lists raising gpu_memory_utilization, lowering max_num_seqs or max_num_batched_tokens, and raising tensor_parallel_size, with a warning that the last may cause excessive synchronization overhead. In vLLM V1, chunked prefill is on by default whenever possible, and the scheduler prioritizes decode requests. Older documentation said a smaller max_num_batched_tokens gave better ITL and a larger one better TTFT; defaults change between versions, so read your version's page.

Verify by re-reading p99 for both TTFT and ITL, plus the preemption counter.

The order is percentiles, then the phase split, then the cause, then the mitigation. Each step uses the result of the step before it.

1 · Percentiles2 · Split3 · Cause4 · FixReplay

Step-by-step description

1. Confirm the slowdown with p99.
2. Separate queue, prefill and decode.
3. Match the symptom to a cause.
4. Apply the smallest safe change.

• The order is percentiles, then the phase split, then the cause, then the mitigation.
• Each step uses the result of the step before it.

Worked example

An illustrative incident shows the method. A typical slow request has a TTFT of 800 milliseconds in healthy traffic and an ITL of 50 milliseconds. After an incident its TTFT is 6,000 milliseconds and ITL is still 50. Because only the first token moved, the fault is before decode. A slow request's queue time is 5,300 milliseconds, so the wait is the cause and prefill is unchanged at 600.¹

Tokens per second per user is about one divided by ITL for long outputs, as NVIDIA's documentation states, so unchanged ITL means the same speed once text starts. The users complain because every answer starts late.

The illustrative queue wait grows from 100 to 5,300 milliseconds while prefill stays at 600 milliseconds, and network stays at 100. The first-token total rises from 800 to 6,000 milliseconds.

Text description of the diagram

• The illustrative queue wait grows from 100 to 5,300 milliseconds while prefill stays at 600 milliseconds.
• The first-token total rises from 800 to 6,000 milliseconds.

Trade-offs and edge cases

Every fix trades something. NVIDIA's documentation states that as concurrent requests increase, total tokens per second for the system rises while tokens per second per user falls. Raising concurrency to drain a queue can therefore slow every user's stream.

• Chunked prefill mainly protects ITL by scheduling decodes first; expect a long prompt's own first token to arrive later.
• Preemption is not a drop: vLLM recomputes the request, so the symptom is a latency spike, which the preemption counter confirms.
• Memory-bound decode can look like a decode slowdown with no software change; see high bandwidth memory (HBM) and paged attention.
• Other causes such as GPU throttling, the network or the tokenizer matter when the phases look healthy; this page gives no numbers for them.
• Tool and version drift: names and defaults change, so quote the version and the tool.

For a target-driven version of this reasoning, see serving a 70B model at a p95 target.

How to answer it in an interview

The spoken structure: confirm with percentiles, split the phases, name the cause, apply the smallest fix, verify, and state the trade-off.

Common mistakes

• "The average latency looks fine, so there is no incident." (wrong) A healthy mean can hide a failing tail. (right) "I check p99 for both first-token time and inter-token latency."
• "Time to first token is just model speed." (wrong) It also includes queue wait and network time. (right) "I read the queue time of a slow request before blaming prefill."
• "Two dashboards' inter-token latency numbers are comparable." (wrong) Tools define it differently. (right) "I check each tool's definition before comparing."
• "More concurrency is free speed." (wrong) It raises total throughput but lowers per-user speed. (right) "I report total and per-user throughput separately."
• "Preemption means requests were dropped." (wrong) vLLM recomputes them later. (right) "Preemption shows up as extra latency."

Follow-up questions

Why is p99 time to first token high when the average looks fine?

A small share of requests is very slow, for example behind a queue or a long prompt, and the mean averages them away. Check the per-request queue time and prompt length of the slowest requests.

How do you tell queueing from slow prefill?

Read the queue time of a slow request. If it accounts for most of the TTFT, requests waited; if the TTFT grew only for long prompts, prefill is the cost.

What does preemption in vLLM mean for latency?

The server pauses a request to free KV cache space and recomputes it when space returns, so the request is delayed, not lost. Frequent preemption suggests cache pressure; the tuning guide lists the remedies.

Is inter-token latency the same as time per output token?

Tools disagree. NVIDIA's benchmarking documentation treats ITL and TPOT as synonyms, while vLLM defines TPOT per request as (end-to-end latency minus TTFT) divided by (output tokens minus one) and ITL per streamed output. They are close for long, steady outputs, so name the tool and its definition.

How does chunked prefill change latency?

It splits a long prompt into chunks so running decodes keep going, which protects ITL, according to Agrawal et al. (2024). The long prompt's own first token can arrive later.

Sources

1. Metrics (design doc), vLLM documentation (stable) — vLLM project, 26 Oct 2026
2. Per-Request Metrics, vLLM documentation (stable) — vLLM project, 26 Oct 2026
3. Optimization and Tuning, vLLM documentation (stable) — vLLM project, 26 Oct 2026
4. Metrics design doc, vLLM documentation (v0.8.2) — vLLM project, 26 Oct 2026
5. Performance and Tuning, vLLM documentation (v0.4.2) — vLLM project, 26 Oct 2026
6. Metrics, NVIDIA NIM LLM Benchmarking (latest) — NVIDIA, 26 Oct 2026
7. Metrics, NVIDIA NIM LLM Benchmarking (1.0.0) — NVIDIA, 26 Oct 2026
8. Benchmarking LLM Inference at Scale with AIPerf — NVIDIA Technical Blog, 26 Oct 2026
9. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2403.02310) — arXiv (Agrawal et al.), 4 Mar 2024