LLM inference optimization: the complete guide

In this article

1. Where LLM inference time goes
2. Start here: the learning path
3. The KV cache and paged attention
4. Quantization
5. How a model picks the next word
6. Attention kernels and GPU fundamentals
7. Speculative decoding
8. Continuous batching and scheduling
9. Inference engines
10. Caching and routing LLM calls
11. Serving at scale: capacity and gateways
12. Production: measuring what you serve
13. Where things stand in October 2026
14. Questions interviewers ask on LLM inference
15. All 30 guides in LLM inference and serving
16. Follow-up questions
17. Sources

Where LLM inference time goes

LLM inference spends most of its time moving data, not computing. A model generates text in two phases. In the first, the model reads the whole prompt in parallel, which keeps the chip busy with arithmetic. In the second, it generates one token per step, and each step has to read all of the model's weights and the stored attention data of every sequence in the batch, for very little arithmetic per byte read. That second phase is memory-bound, which means its speed is set by memory bandwidth rather than arithmetic; the FlashAttention page defines the term in full. It is where most of the cost of a long answer sits.

A worked estimate makes this concrete. A 70-billion-parameter model in 16-bit numbers is 140 GB (gigabytes) of weights. The SXM version of the H100, Nvidia's data-centre graphics processing unit (GPU), has 80 GB of memory and 3.35 TB (terabytes) per second of memory bandwidth (Nvidia, read 2026-10-24), so four such GPUs offer 320 GB and 13.4 terabytes per second combined.

If a decode step only has to read data, one sequence of 8,192 tokens takes about 10.6 milliseconds per step, roughly 94 tokens per second, while a batch of 32 takes about 16.9 milliseconds per step and produces 1,898 tokens per second in total.¹ The weights are read once per step whatever the batch size, so total throughput rises about 20 times, from 94 to 1,898 tokens per second, while each sequence slows from 94 to 59 tokens per second, until the stored attention data or the arithmetic becomes the limit. This is an idealised estimate with perfect bandwidth scaling; real numbers are lower.

Text description of the diagram

• In an idealised memory-bound estimate, one decode step reads the weights once plus each sequence's KV cache.
• Four H100-class GPUs and a 70B model give about 94 tokens per second at batch 1 and about 1,898 at batch 32.

Published systems work shows the same shape at scale. Pope et al. (2022) report a low-batch latency of 29 milliseconds per token for PaLM 540B with 8-bit weights, 76% model FLOPS (floating-point operations per second) utilization on large-batch prompt processing, and up to 32 times larger context lengths when multi-query attention is combined with their partitioning.

Start here: the learning path

The topics run in a sensible learning order, from the transformer basics to the serving system around the model, with the key-value (KV) cache underlying most of them. The map below shows the order, and the sections after it give a short answer for each topic. If the words "attention" and "layer" are new, read the transformers guide first.

LLM inference

LLM inference is the process of running a trained language model to produce output, in which the model reads a prompt in one pass and then generates one token at a time.

Throughput

Throughput is the total number of tokens a system produces per second across all requests, as opposed to the speed of any one request.

Batching

Batching is processing several requests together in one step, so that the model's weights are read from memory once and used for all of them.

Text description of the diagram

• The topics run from transformer basics to the serving system around the model.
• The KV cache underlies the inference topics: quantization, attention kernels, decoding, batching and scheduling, and production.

1. What is a KV cache in an LLM, and how big does it get?
2. What is paged attention in vLLM, and why does it matter?
3. What is LLM quantization, and how much memory does it save?
4. How an LLM picks the next word, step by step
5. GPU fundamentals for ML engineers: memory, compute and the roofline
6. What is speculative decoding, and how does it work?
7. What is continuous batching in LLM serving?
8. Caching LLM calls: how prompt and semantic caching cut cost
9. Model routing and cascades: how to cut LLM cost per query
10. How do you serve a 70B model at p95 under 500 milliseconds?
11. What does an inference engine do?
12. Multi-tenant model serving: how to design it
13. How do you run LLM inference over a billion documents as cheaply as possible?
14. How do you design an LLM gateway?
15. What bottlenecks do you face when running LLM inference?
16. How would you optimize large-scale training/inference?
17. What is KV caching in the context of transformer inference?
18. How to optimize an ML kernel that underutilizes the GPU

The KV cache and paged attention

The KV cache is the stored keys and values of every token already processed, which lets the model compute only the newest token at each step instead of recomputing the whole prompt. It removes repeated work but adds memory: its size, 320 KiB per token for Llama 3 70B, which has 80 layers, 8 key-value heads and a head size of 128 (Meta-Llama-3-70B config.json, read 2026-10-24), is about 2.7 GB for one 8,192-token sequence.

That is why long contexts and large batches strain memory. Read the sizing formula and a worked example at the KV cache page. Paged attention stores the cache in fixed-size blocks so that almost no memory is wasted (only part of each sequence's last block); Kwon et al. (2023) report 2 to 4 times higher throughput at the same latency.

Quantization

Quantization cuts the bytes read per step, which is the main cost in decode. The groundwork is the choice of 16-bit format; the fp16 versus bf16 page explains why one is the standard for training and serving. Going lower, Dettmers et al. (2022) describe LLM.int8(), which they report cuts inference memory by half compared with 16-bit weights by multiplying most values in 8-bit and isolating rare outlier dimensions in 16-bit. The risk is quality: a smaller number format can lose accuracy, so the check after quantizing is always the same task evaluation as before.

How a model picks the next word

Decoding strategies decide which token is emitted from the model's probability scores. Greedy decoding takes the most likely token; sampling draws from the distribution, reshaped by temperature, top-k (only the k likeliest tokens) or top-p (the smallest set of tokens whose probabilities sum to at least p). Temperature near zero makes output almost deterministic; higher values make it more varied. These choices affect quality and variety, not speed, but they matter for inference because they decide what the speed techniques must preserve: some speed-ups, such as speculative decoding below, are designed to leave the output distribution unchanged.

Attention kernels and GPU fundamentals

Attention kernels are the GPU programs that compute attention, and their speed depends on how often they move data between slow and fast memory. FlashAttention is an exact method, not an approximation, that tiles the computation to cut those transfers; Dao et al. (2022) report 3 times faster end-to-end training on a small generative pre-trained transformer (GPT) language model, GPT-2, at sequence length 1,024.

That figure is for training; for inference the gain is mainly in prompt processing and long-context attention, where many queries meet many keys, because a decode step has only one query. FlashAttention-3 (Shah et al., 2024) reports 1.5 to 2.0 times faster attention than its predecessor on H100 GPUs. The mechanism and interview version are on the FlashAttention page; to see why moving data costs more than arithmetic, read GPU fundamentals for machine learning engineers.

Speculative decoding

Speculative decoding makes generation faster by having a small draft model propose several tokens that the large model then checks in one pass, so one expensive step can yield several tokens. Leviathan et al. (2022) report a 2 to 3 times acceleration on the extra-extra-large (XXL) size of the T5 model with output identical to standard decoding. It helps most when decode is memory-bound and the draft model guesses well; it adds little when the batch is already large and the chip is busy.

Continuous batching and scheduling

The naive batch waits for its slowest request. Orca's iteration-level scheduling changes the batch between steps so finished requests leave and new ones join; Yu et al. (2022) report 36.9 times higher throughput than Nvidia FasterTransformer at the same latency on a GPT-3 175B model. Schedulers also have to mix the two phases. Chunked prefill splits a long prompt into pieces so that waiting decode steps are not stalled behind it; as of October 2026 the vLLM tuning guide says chunked prefill is enabled by default whenever possible, that a smaller per-step token budget improves inter-token latency, and a larger one improves time to first token.

Inference engines

An inference engine is the program that does the scheduling, memory management and kernel launching so that application code does not have to. vLLM, Nvidia TensorRT-LLM and Nvidia Triton Inference Server are examples; they package the techniques above (paged KV memory, continuous batching, optimized kernels, quantized formats) behind one serving interface. The practical interview question is what an engine decides for you, such as when a request is preempted, and what you still choose, such as the memory fraction and the token budget. Per the vLLM guide above, a preempted request is recomputed once cache space frees up.

Caching and routing LLM calls

Not every saving happens on the GPU. Prompt caching reuses the stored work for a prompt prefix that repeats across calls; semantic caching returns a stored answer when a new question means the same as an earlier one, which is cheaper but can return a stale or slightly wrong answer.

Model routing sends easy requests to a small, cheap model and hard ones to a large one, and a cascade tries the small model first and escalates when its answer looks unreliable. Prompt caching is exact, so it saves cost with no change to the answer; semantic caching, routing and cascades trade a little quality risk for a lower cost, so each needs the measurement described in the production section below.

Serving at scale: capacity and gateways

Capacity planning turns the memory arithmetic above into a count of GPUs: (weights + cache per sequence × concurrent sequences) ÷ memory per GPU, rounded up. Equivalently, the sequences that fit = (total memory − weights) ÷ cache per sequence, which is 67 sequences of 8,192 tokens in the illustrative estimate above. The result is then checked against a latency target such as the 95th-percentile response time. An LLM gateway is the service in front of the models that applies rate limits, routing, retries and logging for many applications at once. Both appear in system-design interviews, where the strong answer states its assumptions and shows the arithmetic.

Production: measuring what you serve

An optimization that cannot be measured is a guess. In production you track latency percentiles, throughput, cost per token and answer quality together, because a change that helps one often costs another. LLM observability covers what to log and alert on, and the machine learning operations (MLOps) guide places it in the wider machine learning workflow. Interviews often ask for this in design form; see machine learning system design.

Where things stand in October 2026

The foundational papers above date from 2022 to 2023 and their ideas are now standard in serving engines. Current engine documentation, dated 2026-08-20, treats chunked prefill and recompute-on-preemption as defaults, and the attention kernel line has moved on to FlashAttention-3 (H100, with 8-bit floating-point support) and FlashAttention-4 for Nvidia's Blackwell GPUs (Zadouri et al., March 2026). Treat the paper speed-ups as results on the papers' own models and hardware, not as promises for yours.

Questions interviewers ask on LLM inference

Follow-up questions

How does LLM inference work?

The model reads the prompt in one parallel pass, storing keys and values for each token, then generates one token per step, reading the stored data each time and appending one new row. Generation stops at an end token or a length limit. Read the full explanation, with sizes, at the KV cache page.

How do you optimize LLM inference?

Find the bottleneck first. If decode is memory-bound, avoid recomputation with the KV cache; cut bytes per step with grouped-query attention (GQA), which shares each key-value head among a group of query heads, and quantization; and read each byte for more sequences with batching. If prompt processing dominates, look at attention kernels and prompt length. The reason moving data is the cost is on the GPU fundamentals page.

What is inference optimization?

Inference optimization is making a trained model cheaper or faster to run without retraining it from scratch, by changing how it is executed: number formats, scheduling, caching, kernels and decoding. It differs from training optimization, which changes how the model is learned.

Is inference a method to optimize LLM performance?

No. Inference is the act of running the model; optimization is what you do to make that run faster or cheaper. The phrase "LLM performance" can mean speed or answer quality, so state which one when you answer.

Which LLM inference optimization is best?

None is best in general; the right one depends on the bottleneck, the quality bar and the hardware. Batching and KV-cache management are where the largest serving gains in the systems papers above came from (Orca and vLLM report 36.9 and 2 to 4 times), and quantization is the common next step once memory is the limit. As of October 2026, current engines already enable the scheduling ones (continuous batching, chunked prefill) by default, so the choices left to you are mostly memory fraction, number format and whether a draft model pays off.

Sources

1. Efficiently Scaling Transformer Inference — arXiv (Pope et al.), 9 Nov 2022
2. Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI '22 (Yu et al.), 1 Jul 2022
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv (Kwon et al.), 12 Sep 2023
4. Fast Inference from Transformers via Speculative Decoding — arXiv (Leviathan et al.), 30 Nov 2022
5. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — arXiv (Dao et al.), 27 May 2022
6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — arXiv (Dettmers et al.), 15 Aug 2022
7. Meta-Llama-3-70B config.json (mirror of the model configuration) — Hugging Face (date = read date), 24 Oct 2026
8. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — arXiv (Shah et al.), 11 Jul 2024
9. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling — arXiv (Zadouri et al.), 5 Mar 2026
10. vLLM documentation: Optimization and Tuning (chunked prefill, scheduling, preemption) — vLLM project (date = read date; page footer 2026-08-20), 24 Oct 2026
11. NVIDIA H100 Tensor Core GPU — NVIDIA (date = read date), 24 Oct 2026