What is the LLM serving throughput latency tradeoff?

In this article

1. Before you start
2. Why interviewers ask this
3. The throughput latency tradeoff from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference optimization learning path. It assumes only the ideas below.

Why interviewers ask this

A serving design that meets a throughput target can still feel broken to users, and interviewers use this topic to see whether a candidate knows why. A design that reaches model serving, such as serving a large model under a p95 latency target, leads to it quickly. A p95 target means 95 in 100 requests must finish faster than the limit, and p99 means 99 in 100.

A weak answer says "add GPUs" or "batch more." A strong answer separates throughput from latency, splits latency into time to first token (TTFT) and inter-token latency (ITL), and explains that the two respond differently to scheduling choices. A Staff-level answer adds the scheduler: a long prompt can stall everyone's output, and a token budget decides how much prefill work is allowed in each iteration. It also names the metric to report, which is a tail percentile and not a mean.

The throughput latency tradeoff from first principles

Generation stall

A generation stall is a pause in the output of requests that are already running, caused when the scheduler spends an iteration on the prefill of new requests. A prefill iteration can be slow, taking seconds in the example of Agrawal et al. (2024), so every running response freezes for that time.

Token budget

A token budget is the maximum number of tokens, prompt chunks and decode tokens together, that the scheduler allows in one model iteration. In vLLM it is the setting max_num_batched_tokens, and it moves the balance between time to first token and inter-token latency.

In plain words

Picture a shuttle bus that runs from a hotel to a conference. The bus burns almost the same fuel with few riders or many. So carrying more riders per trip moves more people per hour. But each rider's trip gets longer if the bus makes more stops.

A GPU serving a language model behaves like that bus. The batch is the riders. Each trip is one model iteration that outputs one token for every request in the batch. The analogy breaks in one place. A bus trip is not slowed by each extra rider's own luggage, but every extra sequence in a batch adds its own KV-cache reads to the iteration.

In one precise sentence, the LLM serving throughput latency tradeoff says that a batch large enough to use the GPU well makes each iteration slower for every request inside it.

Step 1: A request has two phases with opposite costs

Prefill reads the whole prompt in one pass, and decode outputs one token per iteration. The first phase is called prefill. The second is called decode. An iteration is one pass of the batch through the model.

Agrawal et al. (2024) report that prefill iterations have high latency but saturate GPU compute, while a decode iteration "processes only a single token per request" and so leaves most compute idle. Yu et al. (2022) describe the same loop: each iteration generates a single output token for the request.

Prompt length sets the prefill cost, and answer length sets the number of decode iterations. A long prompt is enough to keep the GPU's compute units busy; a single decode token is not. You now know why the two phases need different handling.

Step 2: Batching decodes raises throughput and slows each user

Batching decodes raises throughput because every request shares the cost of reading the weights. Throughput is the total number of output tokens per second that a server produces across all users.

Reading the model's weights from GPU memory is the main fixed cost of a decode iteration. A batch pays it once, so adding requests adds little time at first. Agrawal et al. (2024) put it this way: "This makes batching highly effective for decodes and consequently for overall throughput."

Each added request still brings work of its own, such as reading its own KV cache. So iterations lengthen as the batch grows, and every user's tokens arrive more slowly. NVIDIA says the trade-off is driven by the number of concurrent requests and the latency budget, and that serving many users concurrently often means higher latency for each request.

Aggregate throughput equals batch size times each user's tokens per second. Those are two different quantities, and the next step shows how scheduling can hurt the second one even when the first looks healthy.

Step 3: Prefill-heavy iterations cause generation stalls

A prefill-heavy iteration freezes running decodes, which is called a generation stall. Agrawal et al. (2024) show that iteration-level batching improves throughput but comes at the cost of high time between tokens because of these stalls.

Their example has requests A and B already decoding. A scheduler that eagerly admits the prefills of new requests C and D delays the decodes of A and B, because, in their example, an iteration that computes prefills can take several seconds. The user watching A sees the text stop mid-sentence.

Average inter-token latency hides this. Most gaps stay short, and the freeze appears only in the tail, which is why tail percentiles matter. The opposite policy avoids stalls. FasterTransformer finishes all ongoing decodes before it schedules a new prefill, so it has no stalls but lower throughput, because its decode batch stays small. Orca schedules prefills eagerly, which raises throughput but causes stalls (Yu et al., 2022; Agrawal et al., 2024). Each policy picks one point on the trade-off.

Step 4: A token budget and chunked prefill balance the two

A token budget splits prefill into chunks so decodes keep running. Chunked prefill is a scheduling method that cuts a prefill request into near-equal chunks and adds new requests to a batch without pausing ongoing decodes (Agrawal et al., 2024). Their system, Sarathi-Serve, uses it to build stall-free schedules.

In vLLM the knob is max_num_batched_tokens. The vLLM documentation says that with chunked prefill enabled the scheduler prioritizes decode requests, and that smaller values such as 2048 give better ITL because fewer prefills slow the decodes. Higher values give better TTFT because more prefill tokens fit in a batch, and the documentation recommends a value above 8192 for throughput, especially for smaller models on large GPUs.

As of October 2026 the same documentation says chunked prefill is enabled by default in V1 whenever possible. Defaults changed between versions, so read the page for the version you run. You now have the whole picture: two phases, batching, the stall it causes, and the budget that tames it.

1 · Two phases2 · Batch the decodes3 · Stall4 · Token budgetReplay

Step-by-step description

1. Prefill reads the prompt; decode outputs one token per iteration.
2. Requests share the weight-reading cost, so throughput rises.
3. A prefill-heavy iteration freezes the running decodes.
4. Chunked prefill keeps decodes running between chunks.

• Prefill reads the whole prompt in one pass, and decode outputs one token per iteration.
• Batching decodes raises throughput because every request shares the cost of reading the weights.
• A prefill-heavy iteration freezes running decodes, which is called a generation stall.
• A token budget splits prefill into chunks so decodes keep running.

Worked example

Start with the metrics. NVIDIA's benchmarking documentation defines end-to-end latency as TTFT plus generation time, and ITL as the end-to-end latency minus TTFT, divided by the number of output tokens minus one. Take an illustrative request with a TTFT of 500 milliseconds, an end-to-end latency of 6.5 seconds and 301 output tokens. The generation time is 6.0 seconds spread over 300 gaps, so ITL is 20 milliseconds per token, which is 50 tokens per second for that user.¹ In a separate illustration, if 32 simultaneous requests each stream at 50 tokens per second, the server's aggregate throughput is 1,600 tokens per second.¹

Now make the batch grow. The illustrative model below says one decode iteration takes 18 milliseconds plus half a millisecond per sequence in the batch, so throughput follows from the batch size.¹

Aggregate throughput rises from 54 tokens per second at batch 1 to 1,561 at batch 128, so a bigger batch multiplies total throughput. One user's speed falls from 54.1 to 12.2 tokens per second over the same range. The curve flattens as the batch grows, so each extra request buys less throughput and costs more latency.¹

Text description of the diagram

• Aggregate throughput rises from 54 tokens per second at batch 1 to 1,561 at batch 128.
• A bigger batch multiplies total throughput.

Text description of the diagram

• One user's speed falls from 54.1 to 12.2 tokens per second over the same range.
• One user's speed falls to under a quarter of its batch-1 value.

A latency target then picks the batch size. If the product needs at least 25 tokens per second per user, the iteration must take at most 40 milliseconds, which allows a batch of 44 and 1,100 tokens per second in total. If it needs 50 tokens per second per user, only a batch of 4 fits, and the total is 200. The stricter target gives 5.5 times less total throughput.¹

Finally, the stall. Suppose illustratively that 32 decodes are running and an 8,000-token prompt arrives, with prefill costing a quarter of a millisecond per token. Admitted whole, one iteration takes 2,034 milliseconds, and every running response pauses that long. With a prefill chunk of 512 tokens per iteration the longest pause falls to 162 milliseconds, 12.6 times shorter, and the long prompt finishes its prefill 25.1% later.¹ In the sense defined above, the token budget is this chunk plus the 32 decode tokens that run in the same iteration, so the chunk is the prefill share of the budget.

Trade-offs and edge cases

Memory limits the batch. Every running sequence holds a KV cache, so the batch cannot grow past what GPU memory holds; the GPU memory page and the high bandwidth memory page cover the hardware side. Kwon et al. (2023) built vLLM with PagedAttention for near-zero waste in KV cache memory and report 2 to 4 times higher throughput at the same latency than FasterTransformer and Orca, with larger gains for longer sequences and larger models. See paged attention.

Static batching wastes time. Yu et al. (2022) point out that with request-level batching, finished requests cannot return to the client and new requests wait for the whole batch. Their iteration-level scheduling, the idea behind continuous batching, reports in the paper's abstract 36.9 times higher throughput at the same latency on a Generative Pre-trained Transformer 3 (generative pre-trained transformer (GPT)-3) 175B model against FasterTransformer. That figure depends on the baseline, so quote it with the baseline.

Text description of the diagram

• FasterTransformer finishes all ongoing decodes before it schedules a new prefill, so it has no stalls but lower throughput.
• Orca schedules prefills eagerly, which raises throughput but causes stalls.
• Sarathi-Serve uses chunked prefill to avoid stalls and keep throughput high.

FasterTransformer finishes all ongoing decodes before it schedules a new prefill, so it has no stalls but lower throughput. Orca schedules prefills eagerly, which raises throughput but causes stalls. Sarathi-Serve uses chunked prefill to avoid stalls and keep throughput high.

Chunking is not free. Agrawal et al. note fixed overheads in practice, and a small chunk makes a long prompt wait longer for its first token, as the table above shows.

Disaggregation is the other current answer to stalls. Zhong et al. (2024), in DistServe, disaggregate the prefill and decoding computation onto separate GPU pools, because running both phases in one shared batch causes strong interference between them. With separate pools a prefill cannot stall a decode, and each pool can use its own parallelism plan. The price is that the KV cache built during prefill must be moved to a decode GPU, so the transfer time becomes part of time to first token.

Chunked prefill keeps one pool, while disaggregation fits strict targets on both metrics at traffic large enough to keep two pools busy. Metric definitions also differ: NVIDIA's documentation treats ITL and time per output token (TPOT) as the same quantity but warns that tools differ on whether TTFT is included. Other levers, such as quantization, change the cost of each iteration and are covered on their own pages.

How to answer it in an interview

The spoken structure: define both sides, say why batching helps, name the stall, give the token budget as the fix, and finish with what you would measure.

Common mistakes

• "Higher throughput means a faster experience for users." (wrong) Throughput is the total across users, and per-user speed can fall as it rises. (right) "Throughput is tokens per second for the whole server; inter-token latency is what one user feels."
• "Latency is one number." (wrong) TTFT, ITL and end-to-end latency respond differently to scheduling. (right) "I track time to first token, inter-token latency and end-to-end latency separately."
• "Continuous batching removes the trade-off." (wrong) It raises throughput, but eager prefill still causes generation stalls. (right) "Continuous batching plus chunked prefill with a token budget manages the stalls."
• "Chunked prefill is free." (wrong) It trades a long prompt's first-token wait for smoother streaming for everyone else. (right) "The token budget sets that trade, and I tune it against my targets."
• "Mean latency is enough." (wrong) A healthy mean can hide a bad tail. (right) "I report p95 and p99 for each metric."

Follow-up questions

What is the difference between throughput and latency in LLM serving?

Throughput is total output tokens per second across all users, and latency is the delay one request sees. They conflict because a larger batch raises the first and usually worsens the second.

What is the difference between time to first token and inter-token latency?

Time to first token covers queueing, prefill and network time before anything appears. Inter-token latency is the average gap between tokens after that, so long prompts hurt the first and long decode batches hurt the second.

Does continuous batching remove the throughput latency tradeoff?

No. It removes the waiting of static batches and raises throughput, but eager prefill admission still produces stalls. See the continuous batching explainer for the mechanism.

What does chunked prefill do, and is it free?

It splits a long prefill into near-equal chunks mixed into decode iterations, so running responses are not paused. It is not free: it delays the long prompt's first token and adds overhead at small chunk sizes.

How do you choose the token budget in vLLM?

Start from the product target. Smaller values of max_num_batched_tokens favor ITL, larger values favor TTFT and throughput, and the documentation recommends going above 8192 for throughput on smaller models on large GPUs. Then measure on your own traffic.

When would you disaggregate prefill and decode?

When both targets are strict and traffic is large. Running prefill and decode on separate GPU pools removes their interference, as in DistServe, at the cost of moving each request's KV cache between pools. For modest traffic, chunked prefill on one pool is simpler.

Sources

1. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2403.02310) — arXiv / USENIX OSDI 2024 (Agrawal et al.), 4 Mar 2024
2. Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI 2022 (Yu et al.), 11 Jul 2022
3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (arXiv 2401.09670) — arXiv / USENIX OSDI 2024 (Zhong et al.), 20 Jan 2024
4. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — ACM SOSP 2023 (Kwon et al.), 12 Sep 2023
5. Optimization and Tuning (chunked prefill, max_num_batched_tokens) — vLLM documentation, 26 Oct 2026
6. Metrics: NVIDIA NIM LLMs Benchmarking — NVIDIA documentation, 26 Oct 2026
7. Optimizing inference efficiency for LLMs at scale with NVIDIA NIM microservices — NVIDIA Technical Blog, 26 Oct 2026