What is continuous batching in LLM serving?

In this article

1. Before you start
2. Why interviewers ask this
3. Continuous batching from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the LLM inference optimization learning path.

Why interviewers ask this

Continuous batching is the reason one GPU can serve many chat users at once without most of its time going to waste. Interviewers use it to separate candidates who know that a reply is produced one token per step from candidates who can say what that does to scheduling, cost and delay.

A weak answer says "it groups requests to use the GPU better." A strong answer names the unit of scheduling (one iteration, not one request), explains why static batches waste slots when reply lengths differ, and says what ultimately limits the batch. It also turns up as a design prompt about batching queries onto GPUs, so be ready to draw the loop.

Continuous batching from first principles

Static batching

Static batching is a scheduling method that fixes a batch of requests at the start and runs it until every request in it has finished, so a slot freed early stays idle.

Iteration

An iteration is a single forward pass of the model that adds one new token to every request in the batch that is generating.

In plain words

Picture a city bus and a long-distance coach. The coach leaves with a fixed group and does not take anyone new until it returns, so a seat freed halfway stays empty. The bus stops at every stop, lets people off and lets waiting people on, so seats rarely stay empty. Static batching is the coach and continuous batching is the bus. The analogy breaks on boarding: a new passenger's prompt (prefill) takes far longer than one stop, and the bus's seat count depends on luggage, which here is cache memory.

In one precise sentence: the server decides who is in the batch once per iteration instead of once per batch. This idea, called iteration-level scheduling, was introduced for transformer serving in Orca (Yu et al., 2022).

Step 1: A reply is many one-token iterations

An LLM writes a reply one token at a time, so one request needs many iterations. The first pass reads the whole prompt (prefill) and each later pass adds one token (decode); the KV cache page explains both. The input to an iteration is every running request's newest token plus its cached keys and values, and the output is one new token per request.

Because replies differ in length, one request may need 4 iterations and another 26. A model server cannot know a reply's length in advance, since the model decides when to stop by emitting an end-of-text token. You now know the key fact: the work is a loop of equal-sized steps, and requests leave that loop at unpredictable times.

Step 2: Static batching waits for the slowest reply

Static batching groups requests, runs the group together and returns results only when the longest reply ends. A request that finishes early keeps its slot but does no useful work, so the GPU computes padding for it, meaning filler tokens that are calculated and thrown away.

A concrete case from the simulation below: the first batch of four holds replies of 26, 19, 7 and 4 tokens. The batch runs 26 iterations, which is 104 slot-iterations, but only 56 of them produce a real token, a utilisation of 53.8%.¹ The 4-token request is finished after 4 iterations yet its answer waits 22 more. Waiting requests also cannot start until the whole batch finishes. You now know the cost: idle slots and delayed answers, both caused by uneven lengths.

Step 3: Continuous batching refills a slot after every iteration

Continuous batching checks the queue after every iteration. When a request emits its last token, its slot is freed and a waiting request takes it before the next iteration, so the batch stays full while work is queued and memory allows.

The input to this step is the set of requests that just finished; the output is the same number of new requests admitted. A new request first needs a prefill pass over its prompt, and Orca's authors solve the awkward mix of requests at different positions with selective batching: the matrix multiplications and layer normalisation are batched across all tokens, while attention is computed separately for each request (Orca §4.2).

Orca (Yu et al., July 2022) reports up to 36.9 times the throughput of NVIDIA FasterTransformer on a 175-billion-parameter generative pre-trained transformer (GPT) 3 model at the same latency. Throughput there is requests completed per second, and latency is the median time per generated token; at 190 milliseconds a token, Orca served 6.81 requests a second against 0.185 (Yu et al., July 2022, §6).

Step 4: Memory, not compute, limits the slots

The number of slots is limited by GPU memory, not by arithmetic. Every running request holds its own KV cache, and the cache grows by one entry per token, so a request admitted now needs more memory later. Orca reserved that memory up front for each request's maximum length (Yu et al., 2022, §4.2), which wastes it; paged systems such as vLLM allocate it block by block and pause a request when memory runs out (Kwon et al., 2023).

Decoding is largely limited by memory bandwidth, meaning how fast the weights and cache move out of GPU memory, so throughput depends mostly on batch size, as the Anyscale analysis (June 2023) puts it. Kwon et al. (2023) report that existing systems waste cache memory through fragmentation (reserved-but-unused memory and unusable gaps) and duplication (identical data stored more than once), which limits batch size; their PagedAttention method, which stores the cache in fixed-size blocks, gives vLLM 2 to 4 times the throughput of FasterTransformer and Orca at the same latency. The size formula is in the KV cache page.

1 · One token per iteration2 · A short reply ends3 · The slot is refilled4 · Memory caps the slotsReplay

Step-by-step description

1. Every running request gains one token each iteration.
2. Its slot frees while longer replies keep running.
3. A waiting request takes the slot after that iteration.
4. KV cache memory sets how many requests fit.

• Each iteration adds one token to every request in the batch.
• A short request finishes and frees its slot while longer requests keep running.
• Continuous batching refills the freed slot from the waiting queue after that iteration.
• KV cache memory, not arithmetic, caps the number of slots.

Worked example

A small simulation shows the effect. It uses 16 requests, all waiting at the start, and 4 slots, with one token per slot per iteration and prefill ignored. The output lengths come from a skewed random draw with a fixed seed: 26, 19, 7, 4, 10, 7, 21, 5, 9, 12, 33, 9, 4, 19, 13 and 4 tokens, 202 tokens in total (illustrative, not a measurement of any real server).¹

Static batching takes the requests in groups of four in arrival order. The groups run for 26, 21, 33 and 19 iterations, which is 99 iterations and 396 slot-iterations, so 202 of 396 slot-iterations do useful work, a utilisation of 51.0%.¹ Continuous batching refills a slot after every iteration and finishes the same 202 tokens in 59 iterations (236 slot-iterations), a utilisation of 85.6%.¹ The mean iteration at which a request finishes falls from 63 to 29.44, because short requests no longer wait for long ones.

Text description of the diagram

• Static batching needs 99 iterations to finish 16 requests on 4 slots.
• Continuous batching needs 59 iterations for the same requests.
• The mean finish iteration falls from 63 to 29.44.

The gap grows with the spread of reply lengths. In its own test on the Open Pre-trained Transformer (OPT) 13B model and one 40 GB NVIDIA A100, Anyscale (June 2023) reports that naive static batching dropped to 81 tokens per second at the highest length variance, and that continuous batching alone reached 8 times the throughput of naive batching.

Trade-offs and edge cases

Continuous batching trades scheduler complexity and memory pressure for much better GPU use. Three cases matter.

First, prefill disturbs decoding. A new request's prompt is processed in one large pass, which can stall every running request for that iteration. SARATHI (Agrawal et al., 2023) splits each prefill into equal chunks and fills the rest of every batch with decode work, reporting up to 1.33 times higher end-to-end throughput for LLaMA-13B on one NVIDIA A6000 and 1.91 times for GPT-3 with pipeline parallelism, which splits the model's layers across several GPUs.

Second, memory fragmentation caps the batch long before compute does. When each request reserves one contiguous block for its longest possible reply, much of that block sits unused, so fewer requests fit. PagedAttention avoids this by storing the cache in small fixed-size blocks that are handed out as tokens arrive, which is why vLLM can run larger batches in the same memory (Kwon et al., September 2023).

Third, the policy has a latency price, and both static and continuous batching are bounded by KV cache memory. Admitting every waiting request raises throughput but makes each running request's iterations slower, so serving stacks cap the batch or the waiting time against a latency target. The number format sets the bytes per cached value: 16-bit formats use 2 bytes (16-bit number formats), and an 8-bit cache or quantized weights free memory for more slots.

Text description of the diagram

• Static batching reuses a slot only when the whole batch ends.
• Continuous batching reuses a slot after every iteration.
• Both are bounded by KV cache memory.

How to answer it in an interview

The spoken structure: say the unit of scheduling, say what static batching wastes, say what refills the slot, then name the memory limit.

Common mistakes

• "Continuous batching makes each request faster." (wrong) A single request's iteration is not quicker; it waits less and the GPU wastes less. (right) "It raises total throughput and lowers waiting, not per-token speed."
• "The batch size is limited by compute." (wrong) Decoding is limited by memory traffic and cache memory. (right) "KV cache memory sets how many requests fit."
• "It is the same as grouping requests that arrive close together." (wrong) That is request-level grouping; the batch members still run to the end together. (right) "It changes the batch after every iteration."
• "A new request joins with no cost." (wrong) Its prompt needs a prefill pass that can stall running requests. (right) "Chunked prefill spreads that cost."

Follow-up questions

Who invented continuous batching?

The idea of scheduling at the granularity of one iteration was published as iteration-level scheduling in Orca at a systems conference in July 2022 by Yu and colleagues. Later open-source servers popularised it, and the name "continuous batching" is the one the Anyscale post (June 2023) uses alongside "dynamic batching" and "iteration-level scheduling".

Continuous batching vs dynamic batching?

Continuous batching and dynamic batching are not the same thing in every system. NVIDIA's Triton dynamic batcher, built for stateless models, merges separate incoming requests into one batch and may wait up to a configured delay to fill it, and its documentation says batches typically complete in a single execution; the same page names continuous, iteration-level and in-flight batching as terms for LLMs, which Triton handles through iterative sequences (NVIDIA documentation, read October 2026). Continuous batching changes membership between the many iterations of one reply.

Does vLLM support continuous batching?

Yes. The Anyscale post (June 2023) lists vLLM as combining continuous batching with its PagedAttention memory management, and the vLLM paper (Kwon et al., September 2023) builds the serving system around that cache layout. Check the current vLLM documentation for the scheduler settings of your version.

How to implement continuous batching?

Keep a waiting queue and a list of running requests. In a loop, admit waiting requests while cache memory allows, run one iteration for every running request, append each new token, and remove requests that emitted the end token. Add a cap on batch size and a policy for long prompts, as in SARATHI (Agrawal et al., August 2023).

What happens when the KV cache is full?

The scheduler stops admitting requests and may pause (preempt) a running request, all of it at once. Its cache is either swapped out to central processing unit (CPU) memory and copied back later, or dropped and recomputed from the prompt and tokens so far (Kwon et al., September 2023). Real servers handle this with preemption policies and paged allocation; which policy is best depends on the latency target. See the machine learning operations hub and the transformer overview for the neighbouring topics.

Sources

1. Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI 2022 (Yu et al.), 11 Jul 2022
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv (Kwon et al.), 12 Sep 2023
3. How continuous batching enables 23x throughput in LLM inference while reducing p50 latency — Anyscale (Daniel et al.), 22 Jun 2023
4. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — arXiv (Agrawal et al.), 31 Aug 2023
5. Dynamic Batcher, Triton Inference Server user guide — NVIDIA (date = read date), 24 Oct 2026