LLM Inference & Serving

How does vLLM work, and why is it fast?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 12 min read

How vLLM works: vLLM is an open-source engine that serves large language models (LLMs) by batching requests continuously over a KV cache stored in small blocks, so little graphics processing unit (GPU) memory is wasted. The part explanations skip is the scheduler, which decides at every step how many tokens each request gets, and the second step below shows the rule.

In this article

Before you start
Why interviewers ask this
vLLM from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

vLLM is one engine within the wider field of LLM inference optimization, and understanding it builds on the transformer architecture.

Why interviewers ask this

An answer that stops at "vLLM uses PagedAttention" costs the point, because the follow-up is always what the rest of the engine does. Interviewers ask how vLLM works to see whether a candidate understands that serving throughput is limited by memory and scheduling, not only by arithmetic speed.

A weak answer names the library and one buzzword. A strong answer explains why the KV cache fragments memory, how fixed-size blocks fix that, and why more free memory means a bigger batch. A Staff-level answer adds the scheduler: each step is shared out under a token budget, repeated prompt prefixes can be reused, and for small models the central processing unit (CPU) work between GPU steps can become the bottleneck. The sources for all of this are the vLLM paper and the vLLM team's V1 architecture post; this page is built from both.

vLLM from first principles

Block table: A block table is a per-request list that maps a request's token positions to the fixed-size memory blocks holding its KV cache, so the blocks need not sit next to each other in GPU memory, in the way a page table maps virtual memory to physical memory.
Prefix cache: A prefix cache is a store of KV-cache blocks keyed by a hash of the tokens they hold, so a new request that starts with text already processed can reuse those blocks instead of recomputing them.

In plain words

Picture a car park. The old way reserves one long row of bays for each car. The row is sized for the longest car that could ever arrive. Most bays sit empty, so few cars fit. The vLLM way gives each car single bays, wherever they are free. It writes the bay numbers on a ticket. The analogy breaks in one place. A parked car never needs more space, but a language model's cache grows by one token at a time.

In one precise sentence, vLLM keeps each request's KV cache in small fixed-size blocks tracked by a block table, and a scheduler packs as many requests as fit into every step of the model.

Step 1: The KV cache lives in small blocks, not one long reservation

A language model's KV cache grows with every token a request generates, and its final length is not known in advance. Kwon et al. (2023) argue that managing this memory badly wastes it through fragmentation and duplication, which limits the batch size. Their fix is PagedAttention, an attention algorithm inspired by virtual memory and paging in operating systems, covered in depth on its own page.

Input: a request whose cache grows token by token. What changes: the engine takes one new block from a shared pool whenever the last one fills, and records it in the block table. Output: a cache whose waste is at most part of one block. The paper reports near-zero KV-cache waste and sharing of cache within and across requests. You now know where the memory saving comes from; the next step is who uses it.

Step 2: The scheduler splits a token budget across requests

The scheduler decides, at every step, which requests run and how many tokens each one processes. In V1 it does not separate a prefill phase (reading the whole prompt) from a decode phase (writing one token at a time). Per the V1 post (January 2025), it treats prompt tokens and generated tokens alike, and a decision is a simple map from request id to a number of tokens.

Input: the running and waiting requests plus a fixed token budget. What changes: each running request gets its next token, and the leftover budget goes to waiting prompts. Output: one batch for the GPU. The same post says chunked prefill falls out of this naturally: with a fixed budget the scheduler dynamically chooses how many tokens each request receives. A long prompt is therefore split over several steps instead of stalling everyone else's output.

A budget example uses a budget of 2,048 tokens per step, 48 running requests and a 3,000-token prompt (all illustrative). The first step gives the 48 decoders one token each and the prompt 2,000 tokens. The second finishes the last 1,000 prompt tokens beside the same 48 decoders, 1,048 tokens in all.¹

Step 3: A prefix cache reuses blocks that were already computed

Many requests begin with the same text, such as a system prompt or a shared document. The prefix cache stores finished blocks under a hash of their tokens, and a new request whose first tokens match can point its block table at those blocks. The V1 post says V1, like V0, uses hash-based prefix caching and eviction based on least recently used (LRU) order, meaning the block unused for longest is dropped first.

Input: a new request's token blocks. What changes: matching blocks are shared and not recomputed. Output: less GPU work before the first new token. The idea is the engine-level cousin of caching LLM calls at the application level. This is a saving only when prompts repeat; with no repetition the lookup finds nothing and the benefit is zero.

Step 4: A lean engine loop keeps the GPU fed

Between two GPU steps the engine runs CPU work: updating request states, building inputs and returning tokens. The V1 post (January 2025) says this CPU work is part of vLLM's core loop and is especially visible with smaller models such as Llama-8B on NVIDIA H100 GPUs, where a GPU step can take as little as about 5 milliseconds. It reports up to 1.7 times higher throughput than V0 (without multi-step scheduling) from the re-architecture, per the same post. That figure is from the January 2025 alpha announcement and this page was last checked in October 2026, so check the current vLLM documentation for the present state of V1 before relying on it.

The engine loop drives each step. Input: the scheduler's decision. What changes: tokenization, detokenization and streaming run in a separate process, so they overlap with the engine core's loop instead of stalling the GPU, as the V1 post describes. Output: tokens streamed to the client. The engine also exposes an application programming interface (API) server compatible with the OpenAI interface.

vLLM is four parts working together: a block pool, a token-budget scheduler, a prefix cache and a lean engine loop.CC BY 4.0 — source: How does vLLM work, and why is it fast?

Step-by-step description

The KV cache lives in small blocks.
A token budget is shared per step.
Blocks are reused by hash.
A lean loop keeps the GPU fed.

The KV cache lives in small blocks in a shared pool.
The scheduler shares a token budget across requests at every step.
A prefix cache reuses blocks by hash.
The engine loop drives each step.

Worked numbers: what paging buys on one GPU

All figures here are illustrative inputs and exact arithmetic.¹ Take a model with 32 layers, 8 KV heads, a head size of 128 and 16-bit numbers (illustrative). Each token then needs 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB of cache.¹ On an 80 GiB GPU (illustrative) with 90% usable and 16 GiB of weights (illustrative), the cache pool is 56 GiB.¹

If the engine reserves room for 8,192 tokens per request, each request takes 1 GiB and 56 requests fit. If real requests hold 1,000 tokens, paging with a block size of 16 (illustrative) uses 63 blocks, 1,008 token slots, so 8 slots are wasted, and 455 requests fit, about 8.12 times as many.¹ The worst waste per request is 15 slots, under 2 MiB.¹

Paging turns 87.8% waste per request into at most 15 token slots, which raises the illustrative capacity from 56 to 455 requests.
Quantity	Value (illustrative setup)
Cache per token	131,072 bytes
Cache pool	56 GiB
Reserve 8,192 tokens, requests that fit	56
Paged blocks, requests that fit (1,000 tokens each)	455
Token slots wasted per request, reservation	7,192 (87.8%)
Token slots wasted per request, paged	8 (at most 15)

Allocating cache in small blocks instead of reserving the maximum length fits several times more requests in the same memory.CC BY 4.0 — source: How does vLLM work, and why is it fast?

Text description of the diagram

Reserving 8,192 tokens fits 56 requests in the illustrative pool.
Paged blocks fit 455 requests in the illustrative pool.

A bigger batch matters because the paper reports a 2 to 4 times throughput gain over FasterTransformer and Orca at the same level of latency (Kwon et al., 2023).

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.