What does an inference engine do?

In this article

1. Before you start
2. Why interviewers ask this
3. Inference engine from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the large language model (LLM) inference optimization learning path, and it assumes only the ideas below.

Why interviewers ask this

LLM features in production run on an engine, so a candidate who says "the engine runs the model" has told the interviewer nothing about cost, speed or failure. Candidates report questions on converting a model into a TensorRT engine, on deploying behind Triton Inference Server, and on designing a vLLM-style serving platform; all three reduce to this one question.

A weak answer lists product names. A strong answer names the four jobs (prepare the model, schedule requests, manage cache memory, run kernels and stream tokens) and says which resource limits each. A Staff-level answer adds the central fact: generating text is limited by how fast weights and cache are read from memory, so the engine's real product is utilization of that bandwidth and of GPU memory. The transformer architecture page covers the model itself; this page covers what runs around it.

Inference engine from first principles

Scheduler

A scheduler is the component of an inference engine that decides before each step which waiting and running requests are processed together, so the GPU stays busy and waiting requests join as soon as batch slots and cache memory free up.

In plain words

Inference means using a trained model to produce outputs, as opposed to training it. Picture a restaurant. The recipe book is the model. The kitchen is the engine. Order tickets are requests, counter space is memory, stoves are GPU kernels, and the pass is where finished dishes leave.

The analogy breaks in one place. Dishes are independent, but each new token of a reply depends on every earlier token. So a request stays in the kitchen for hundreds of rounds, and the engine must keep its working notes the whole time.

In one precise sentence, an engine repeats a loop of scheduling, memory allocation, kernel execution and streaming around fixed weights.

Step 1: The engine prepares the model for the hardware

The engine first turns a checkpoint file into weights sitting in GPU memory in a chosen number format. Input: the trained checkpoint. Output: weights resident on the GPU, plus any compiled artifacts.

Precision is the main choice. A 13-billion-parameter model (an illustrative size) takes 26 GB at 16 bits per weight and 13 GB at 8 bits.¹ Lower precision is called quantization, and it trades a little accuracy for memory and speed. Ahead-of-time builders may also fuse adjacent operations into one kernel, so intermediate results are not written back to memory between them. The TensorRT variant below describes that build step.

You now know the starting state: fixed weights in memory. Everything else in the engine decides how requests share them.

Step 2: The scheduler batches requests at every step

A scheduler is the component that decides, before each step, which requests run together. A decode step is one forward pass that produces one new token for every request in the batch. Input: all waiting and running requests. Output: the batch for this step.

Batching works because decoding is limited by memory bandwidth: each step reads all the weights once, whether the batch holds one request or many. With illustrative values of 26 GB of weights and 2,000 GB per second of bandwidth, the weight-only ceiling is about 76.9 tokens per second for one request and 2,461.5 for thirty-two sharing each read.¹ That 2,461.5 ignores KV-cache reads.

Each step also reads every running request's cache: at 400 tokens each, thirty-two requests add about 10.49 GB to the 26 GB, so a step costs about 1.4 times the single-request step, and the cache-aware rate is about 1,754.1 tokens per second against 75.97 for one request.¹ With the 27 sequences of 2,048 tokens that fit on the GPU in Step 3, the cache read is about 45.30 GB, more than the 26 GB of weights, and the rate falls to about 757.4.¹

That answers the follow-up: one decode step for thirty-two requests costs far less than thirty-two separate steps, because the weight read is shared.

Continuous batching refills freed slots

Continuous batching refills a freed slot at every step; Yu et al. (2022) introduced this iteration-level scheduling in the Orca serving system. A static batch must wait for its longest request, so slots whose requests finished early sit idle until the whole batch ends. Orca's scheduler instead runs a single iteration of the model on the current batch, then re-forms the batch, so finished requests leave and waiting ones join between iterations. In a toy run of eight requests with four slots, static batches need 160 steps and continuous batching needs 112 for the same 286 tokens.¹

Text description of the diagram

• Static batches need 160 steps for eight toy requests.
• Continuous batching needs 112 steps for the same 286 tokens.

Step 3: The cache manager gives each request memory for its keys and values

The KV cache manager allocates GPU memory for the keys and values of every token each running request has produced. Input: the batch from the scheduler. Output: a memory location for each request's cache.

For the illustrative 13-billion-parameter model with 40 layers, 40 KV heads, a head size of 128 and 16-bit values, one token costs 819,200 bytes (800 KiB), so a 2,048-token sequence holds 1.678 GB.¹ On an 80 GB GPU with 90% usable (illustrative), 46 GB is left for cache after the weights. Reserving 2,048 tokens per request fits 27 sequences; allocating 16-token blocks as needed, at an average of 400 tokens, fits 140.¹ The 140 assumes every sequence sits at the 400-token average; as sequences grow, the engine must preempt, as the follow-up below describes.

Allocating cache in small blocks as tokens arrive is the idea of paged attention. Kwon et al. (2023) report that their vLLM system reaches near-zero cache waste and improves throughput by 2 to 4 times at the same latency against earlier systems. It changes where cached values live, not what the model learned.

Separately, grouped-query attention, which gives a model fewer KV heads than attention heads, shrinks the cache itself; the illustrative model uses 40 KV heads, so its cache is large.

Text description of the diagram

• Reserving 2,048 tokens per request fits 27 sequences in 46 GB of cache.
• Allocating 16-token blocks at an average of 400 tokens fits 140 sequences.

Step 4: Kernels run the forward pass and a stream returns tokens

The engine calls GPU kernels for every layer, then a sampler picks the next token from the output scores. Input: the batch and its caches. Output: one new token per request, sent to the client as soon as it exists, so users see text growing instead of waiting for the full reply.

A request first goes through prefill, one pass over its whole prompt that fills the cache; the wait for the first token is mostly prefill. After that each step is a decode step. A request leaves the batch when it emits an end token or hits a length limit, which frees its cache blocks for the scheduler to reuse. The GPU fundamentals page explains why kernel speed depends on memory traffic as much as arithmetic.

1 · Prepare model2 · Schedule3 · Allocate cache4 · Run and streamReplay

Step-by-step description

1. Weights are loaded in a chosen precision.
2. The scheduler picks the batch for each step.
3. Each request gets KV cache blocks.
4. Kernels run and tokens stream back.

• The engine loads fixed weights into GPU memory first.
• A scheduler picks the batch, a cache manager allocates KV cache blocks, kernels run the forward pass and tokens are streamed back.

The Staff-level answer

What makes it Staff-level is the last two sentences: it names the limiting resource and the trade-off, not just the components.

The rubric

Three variants of this question

How would you build a toy LLM inference engine from scratch?

Write the loop in four parts: load weights, keep a queue of requests, run one decode step for the whole batch with a per-request cache, and stream each token. Start with static batches, then refill freed slots every step, which is the 160-to-112 step gain in the toy run above.¹ Add block-based cache allocation last.

How do you convert a trained model into a TensorRT engine?

Candidates report this question. As of October 2026, the NVIDIA TensorRT documentation says TensorRT takes trained models from frameworks such as PyTorch and the Open Neural Network Exchange (ONNX) format and compiles them into engines, optimized executable artifacts for a specific deployment configuration, with support for mixed precision and dynamic input shapes.

In general, such a build step also fuses adjacent layers, chooses a lower precision such as 16-bit floating point, 8-bit floating point or 8-bit integer (post-training 8-bit integer quantization usually needs sample data to calibrate value ranges), and times candidate kernels on the target GPU. Because the engine is built for one configuration, rebuild it when the GPU or the software version changes. Check the version-specific build options in the documentation for the release you use.

For the deployment workflow, the Triton documentation describes a model repository that holds the models the server serves, a model configuration (config.pbtxt) for each model, required for some backends, and batchers, including a dynamic batcher that groups incoming requests. So the engine is the optimized model, and the server takes requests, batches them and calls the engine. Say which version you have used.

How would you run a 3-billion-parameter model on an edge device?

Start with memory: 3 billion weights take 6 GB at 16 bits, 3 GB at 8 bits and 1.5 GB at 4 bits (illustrative sizes).¹ Lower precision makes the model fit and cuts memory reads, which dominate decoding. Weights can be memory-mapped from storage so they load without a second copy, and on a device where the central processing unit (CPU) and the accelerator share one memory pool, there is no copy across a bus.

Per-token time is roughly the weight bytes divided by memory bandwidth; at an illustrative 100 GB per second, 3 GB of 8-bit weights give about 30 milliseconds per token and 1.5 GB of 4-bit weights about 15 milliseconds.¹ Then measure time to first token and per-token time against the target, say 100 milliseconds, instead of assuming the numbers.

Common mistakes

• "An inference engine is the model." (wrong) The engine is the runtime around it. (right) "The model is fixed weights; the engine schedules requests, manages memory and runs kernels."
• "TensorRT and Triton Inference Server are the same thing." (wrong) They play different roles. (right) "TensorRT compiles a trained model into an engine optimized for a specific deployment configuration. Triton serves requests from a model repository with a model configuration (config.pbtxt) for each model, required for some backends, and can batch requests dynamically."
• "vLLM changes the model's answers." (wrong) Paging changes where cache lives. (right) "It manages memory differently; the weights and the math are the same."
• "Bigger batches are always better." (wrong) They raise throughput but can slow each stream and need more cache memory. (right) "I set the batch limit from a latency target."

Follow-up questions

Why does one decode step cost far less than thirty-two separate steps?

Each step reads all the weights from memory once, and that shared read dominates at short contexts. Extra requests add their own KV-cache reads and a little arithmetic, so throughput rises with batch size but less than in proportion, and the gain shrinks as contexts grow.

Does PagedAttention change the model's outputs?

No. It changes how cache memory is allocated, which lets more requests share a GPU; Kwon et al. (2023) frame it as a memory-management technique. Read more in paged attention.

What happens when GPU memory runs out?

The engine must refuse or queue new requests, or pause a running one and redo its work later. As of October 2026, the stable vLLM optimization and tuning page describes preemption when cache space runs out, and describes chunked prefill as a scheduling feature; the automatic prefix caching page covers reuse of cache for shared prompt prefixes. Which policy applies differs between engines and versions, so check the release you run.

How would you design an LLM inference platform, a vLLM-style service?

Put a gateway and queue in front of a pool of engine replicas, route by load, autoscale on queue depth, and track time to first token and tokens per second. The LLM inference guide covers the pieces.

Sources

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al., SOSP 2023), 12 Sep 2023
2. Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI 2022 (Yu et al.), 11 Jul 2022
3. NVIDIA TensorRT documentation, version 11.4.0 (overview) — NVIDIA documentation, 26 Oct 2026
4. Triton Inference Server: Model Repository — NVIDIA documentation, 26 Oct 2026
5. Triton Inference Server: Model Configuration — NVIDIA documentation, 26 Oct 2026
6. Triton Inference Server: Batchers — NVIDIA documentation, 26 Oct 2026
7. vLLM stable documentation: Optimization and Tuning — vLLM project documentation, 26 Oct 2026
8. vLLM stable documentation: Automatic Prefix Caching — vLLM project documentation, 26 Oct 2026