In this article
Before you start
This page is part of the LLM inference optimization learning path and assumes the transformer architecture at the level of "tokens go in, the next token comes out".
Why interviewers ask this
The question separates a candidate who has run a model from one who has only called an application programming interface (API). A person who says "inference is slow because the model is big" cannot choose a fix, because each bottleneck has a different one. Questions like this are reported by candidates in interviews at AI labs and large tech companies, usually as the opening of a serving or system design discussion.
A weak answer lists tricks such as batching, quantization and caching with no reason for any of them. A strong answer names the limiting resource of each phase and gives the evidence for it. A Staff-level answer adds numbers: how many bytes move per token, how many requests fit in memory, and where adding batch size stops paying off.
LLM inference bottlenecks from first principles
- Bottleneck
- A bottleneck is the one resource, such as arithmetic speed, memory transfer speed or memory capacity, that runs out first and so sets the speed of the whole system. Speeding up anything else changes nothing until that resource is relieved.
In plain words
Think of a chef and a pantry. The chef can chop very fast, but every ingredient must be fetched from a pantry across the room. If the chef cooks one dish at a time, most of the time goes to walking, not chopping. Cooking many dishes per trip uses the same walk for more food. The analogy breaks in one place: the pantry also has to hold a notebook for every open order, and that notebook grows with each dish.
In one precise sentence, serving an LLM is limited by arithmetic speed in one phase, by memory transfer speed in another, and by memory capacity throughout.
The example model and hardware
Every number below uses one illustrative setup, so you can redo the arithmetic.1 The model has 70 billion parameters stored in 16-bit form, which is 2 bytes each, so the weights take 140 GB. It has 80 layers, 8 key-value heads and a head size of 128, the published shape of Llama 2 70B (Touvron et al., 2023).
The hardware is four illustrative GPUs, each with 80 GB of memory, 3.35 TB/s of memory bandwidth and 989 trillion operations per second of 16-bit compute. Together that is 320 GB, 13.4 TB/s and about 3.96 quadrillion operations per second, assuming perfect scaling across the four. These values are close to the published peak figures of a current data-center GPU, used here as illustrative inputs, and real chips reach only a fraction of their peak.
Step 1: Prefill is limited by compute
Prefill is the first pass in which the model reads the whole prompt in parallel and builds the cache for it. Because all prompt tokens are processed together, each weight read from memory is reused for every token, so the arithmetic intensity is high and prefill is compute-bound.
Each parameter takes part in about two operations per token, one multiply and one add. A 2,000-token prompt through 70 billion parameters is about 280 trillion operations, which takes about 71 milliseconds at perfect use of the illustrative compute. Reading the weights once takes only about 10.4 milliseconds.1 Compute, not memory, is the floor. Prefill sets the time to first token. Time to first token is the wait before the first output token appears. Longer prompts raise it, and attention work grows faster than linearly with prompt length. You now know the first limit: prefill waits on arithmetic.
Step 2: Decoding is limited by memory bandwidth, so batching helps
Decode is the phase after prefill in which the model emits one token per step, and each step must read every weight and the whole KV cache from memory. One token does only two operations per parameter, so the arithmetic intensity is about 1 floating-point operation (FLOP) per byte at batch size 1, while the illustrative chip needs about 295 to become compute-bound.1 One step's arithmetic takes about 35 microseconds, but reading the weights and a 4,096-token cache takes about 10.5 milliseconds, roughly 300 times longer. NVIDIA and Databricks describe decode in these memory-bandwidth terms, and Shazeer (2019) built multi-query attention for the same reason.
Batching helps because one weight read serves every request in the batch.
| Batch size (4,096-token contexts) | Step time (milliseconds) | Tokens per second, all requests | Tokens per second, one request | KV cache (GB) |
|---|---|---|---|---|
| 1 | 10.55 | 94.8 | 94.8 | 1.34 |
| 32 | 13.65 | 2,343.7 | 73.2 | 42.95 |
| 134 (memory limit) | 23.87 | 5,613.8 | 41.9 | 179.85 |
Step 3 explains where 134 comes from.1
Text description of the diagram
- Illustrative decoding throughput is about 95 tokens per second at batch 1 and about 5,610 tokens per second at the memory limit of 134 sequences.
- Each request slows down as the batch grows.
Illustrative decoding throughput is about 95 tokens per second at batch 1 and about 5,610 tokens per second at the memory limit of 134 sequences. Each request slows down as the batch grows.
Step 3: The KV cache sets the capacity of the batch
The KV cache is the third limit because it takes memory for every token of every open request, and the batch can only be as large as the memory that is left. For each token the cache holds a key and a value for every layer and KV head: 2 × 80 layers × 8 heads × 128 values × 2 bytes is 327,680 bytes per token. A 4,096-token request therefore takes about 1.34 GB.1
After the weights take 140 GB of the illustrative 320 GB, 180 GB is left, which fits 134 such requests. Kwon et al. (2023) show that memory wasted by fragmentation shrinks the usable batch, which is what paged attention reduces. The cache also explains why batching tops out well short of the compute-bound point: at 134 requests the weight matrices reach about 134 FLOPs per byte, but counting the cache reads the whole step is about 59, far under 295, so decode stays memory-bound. Attention reads each request's own cache, which no other request can share.1
Step 4: Communication and scheduling overhead
A model that does not fit on one GPU is split across several, and the split adds a fourth kind of cost: communication between GPUs. Tensor parallelism, which divides each layer's matrices across GPUs, makes the GPUs exchange partial results twice per layer in the forward pass, so the speed of the links between them joins the budget. The arithmetic above assumed perfect scaling, so real step times are longer. Adding GPUs adds capacity and bandwidth, but also more communication.
Scheduling overhead is the other cost. The processor (CPU) tokenizes and detokenizes text, schedules requests and launches GPU work. If it falls behind, the GPU sits idle between steps. Static batching, which waits for the slowest request in a batch before starting new ones, also wastes GPU time; continuous batching refills finished slots at once.
Prefill is compute-bound and sets time to first token. Decoding is memory-bandwidth-bound. The KV cache limits how many requests fit on the GPU. Communication between GPUs and scheduling overhead can leave the GPU idle.
Step-by-step description
- Reading the prompt is limited by compute.
- Each new token waits on memory bandwidth.
- Cache memory caps the batch size.
- Communication and scheduling add cost.
- Prefill is compute-bound and sets time to first token.
- Decoding is memory-bandwidth-bound.
- The KV cache limits how many requests fit on the GPU.
- Communication between GPUs and scheduling overhead can leave the GPU idle.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →The Staff-level answer
What kind of bottlenecks might you face when running inference on an LLM?
Inference has two phases with different limits. Prefill processes the whole prompt at once and is compute-bound, so it sets time to first token. Decoding emits one token at a time and is memory-bandwidth-bound, because every step reads all the weights plus the KV cache. The KV cache also caps how many requests fit on the GPU, so batching raises throughput only until memory runs out, and each extra request slows the others. If the model spans GPUs, communication adds cost, and slow CPU scheduling can leave the GPU idle. I would measure first: time to first token, time per output token and memory in use, then fix whichever binds.
What makes this Staff-level is that each bottleneck is tied to a phase, the batch limit is traced to memory, and the answer ends with how to find the binding limit instead of a list of tricks.
The rubric
| Dimension | Weak answer | Strong answer | Staff answer |
|---|---|---|---|
| Correctness | "The model is big, so it is slow." | Prefill is compute-bound; decoding is memory-bandwidth-bound. | Adds the KV cache and communication, and says which binds for a given workload. |
| Depth | Names no mechanism. | Says every decode step reads all weights and the cache. | Computes bytes per token and the request count that fits in memory. |
| Trade-offs | Says "use a bigger batch." | Batching raises throughput but slows each request. | Shows where memory caps the batch and what that does to latency. |
| Communication | Lists tricks in random order. | Orders the answer by phase. | Opens with the limits, closes with how to measure them. |
| Follow-up readiness | Stalls on "why". | Explains why quantization helps decoding. | Moves to fixes: paging, continuous batching, chunked prefill, grouped-query attention (shared key-value heads), quantization. |
Three variants of this question
How do you tell whether your LLM server is compute-bound or memory-bound?
Compare arithmetic intensity with the chip's ratio of compute to bandwidth, then confirm with a profiler, a tool that records where the GPU spends its time. Decoding at small batch is almost always memory-bound; long-prompt prefill is usually compute-bound. The GPU fundamentals page covers the compute-versus-bandwidth picture, and the high-bandwidth memory page covers why the memory next to the GPU is the key number.
What limits the batch size when serving an LLM?
GPU memory, mostly the KV cache. Each request's cache grows with its length, so the weights plus all caches must fit. In the worked example, 134 requests of 4,096 tokens fill the memory that is left.1
What changes if the prompt is very long?
Prefill takes longer, so time to first token rises, and each request's cache takes more memory, so fewer requests fit. Both limits tighten together.
Common mistakes
- "LLM inference is compute-bound." (wrong) That is true only of prefill. (right) "Prefill is compute-bound; decoding is memory-bandwidth-bound."
- "More FLOPs means faster token generation." (wrong) Decoding speed follows memory bandwidth and batch size. (right) "At small batch, a chip with more compute but the same bandwidth decodes at about the same speed."
- "The weights are the only memory cost." (wrong) The cache grows with context length and batch size. (right) "Weights are fixed; the cache is what limits how many requests I can serve."
- "A bigger batch always helps." (wrong) Memory caps it and per-request latency worsens. (right) "Throughput rises with batch size until memory runs out; I set the batch from the latency target."
Follow-up questions
Why does adding GPUs not always make LLM inference faster?
More GPUs add memory and bandwidth, but splitting a model makes the GPUs exchange results twice per layer in the forward pass. When that communication takes longer than the compute it saves, the step does not speed up.
How does quantization change the bottleneck?
Quantization stores weights in fewer bits. At 8 bits the illustrative weights shrink from 140 GB to 70 GB, so the batch-1 decode step falls from about 10.55 to about 5.32 milliseconds, and the space left for the cache fits about 186 requests instead of 134.1
How do you fix a slow time to first token?
Prefill sets it, so attack compute: shorter prompts, caching a shared prompt prefix, faster attention kernels such as FlashAttention, or more compute through parallelism. The LLM call caching page covers the prompt-reuse route.
Why does a long prompt slow down other users' decoding?
A long prefill batched with ongoing decodes occupies the GPU for its whole compute time, so those requests wait and their time per output token spikes; prefill throughput and decode latency compete. Chunked prefill splits the prompt into pieces mixed with decode steps (Agrawal et al., 2024), and disaggregation runs prefill and decode on separate GPUs (Zhong et al., 2024).
What is the difference between latency and throughput in LLM serving?
Latency is how long one request waits; throughput is how many tokens per second the server produces in total. Batching trades one for the other, as the table in Step 2 shows. The 70B serving design page works through the trade under a latency target.
- Prefill is compute-bound; decoding is memory-bandwidth-bound.
- Every decode step reads all the weights and the KV cache, so bytes moved set the speed.
- Batching shares the weight read and raises throughput until KV memory caps the batch.
- Splitting a model across GPUs adds communication cost; slow scheduling leaves the GPU idle.
- Name the limiting resource first, then choose the fix.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al.), accessed 2026-10-26, 12 Sep 2023
- Llama 2: Open Foundation and Chat Models (arXiv 2307.09288) — arXiv (Touvron et al.), accessed 2026-10-26, 18 Jul 2023
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv 2403.02310) — arXiv (Agrawal et al.), accessed 2026-10-26, 4 Mar 2024
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (arXiv 2401.09670) — arXiv (Zhong et al.), accessed 2026-10-26, 17 Jan 2024
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv 2205.14135) — arXiv (Dao et al.), accessed 2026-10-26, 27 May 2022
- Fast Transformer Decoding: One Write-Head Is All You Need (arXiv 1911.02150) — arXiv (Shazeer), accessed 2026-10-26, 6 Nov 2019
- Mastering LLM Techniques: Inference Optimization — NVIDIA Technical Blog, accessed 2026-10-26, 17 Nov 2023
- LLM Inference Performance Engineering: Best Practices — Databricks, accessed 2026-10-26, 12 Oct 2023
