GPU memory: what it holds and how much an LLM needs

In this article

1. Before you start
2. Why interviewers ask this
3. GPU memory from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page sits inside the large language model (LLM) inference optimization guide.

Why interviewers ask this

A model that does not fit on the card cannot be served at all, so memory arithmetic is the first check in any serving design. Questions of this kind are reported in interviews at AI labs and large tech companies, for example: your model and its intermediate data do not fit in GPU memory on long inputs, so how do you design an offloading policy?

A weak answer says "buy a bigger GPU." A strong answer splits memory into weights, KV cache and overhead, does the arithmetic out loud, and separates capacity from bandwidth. A Staff-level answer adds what to do when the budget fails: quantize the weights, shard across cards, page the KV cache, or move cold data to the central processing unit (CPU) and pay for the transfer. The worked example below shows each of those moves with numbers, and the 70B serving walkthrough places them in a full design.

GPU memory from first principles

GPU memory

GPU memory is the pool of on-card memory, called device memory, that a graphics processing unit reads and writes directly, holding a model's weights, its KV cache and temporary buffers during inference. It is separate from the host's CPU memory and is a different spec from compute speed.

Offloading

Offloading is a technique that moves data which does not fit in GPU memory, such as part of the weights or the KV cache, to CPU memory or a solid-state drive (often NVMe, non-volatile memory express), then brings it back across a link when needed, which trades capacity for transfer time.

In plain words

Think of a workshop. The GPU is the worker, the bench is GPU memory, and the warehouse across town is CPU memory. Everything the worker needs often must sit on the bench, because a trip to the warehouse takes far longer than reaching for a tool. The analogy breaks in one place. A real bench is limited by two numbers. One is its size (capacity). The other is how fast the worker's hands can pick things off it (bandwidth). For language models the second often matters more.

In one precise sentence, a model can run only if its weights, its KV cache and its buffers fit in GPU memory together, and its decode speed is limited by how fast those bytes can be read.

Step 1: The GPU has its own memory, separate from the CPU's

NVIDIA's Compute Unified Device Architecture (CUDA) programming guide describes the host and the device as keeping separate memory spaces, called host memory and device memory. Inside the device, a program sees private local memory per thread, shared memory per block of threads, and global memory visible to all threads; the guide says shared memory "is expected to be much faster than global memory".

"GPU memory" in everyday speech means the global memory. On a data-center card it is built from high bandwidth memory (HBM). The NVIDIA H100 specification table lists 80 GB of GPU memory for one H100 variant and 94 GB for another. You now know where the data lives: on the card, apart from the CPU. The GPU fundamentals page covers the full hierarchy.

Step 2: The weights fill GPU memory first

The weights are the model's learned numbers, and in standard serving they must all be in GPU memory before the first token. Their size is the parameter count times the bytes per parameter: two bytes at 16 bits, one at 8 bits, half a byte at 4 bits. A 70-billion-parameter model at 16 bits is 140 GB, which does not fit on one 80 GB card, while the same model at 8 bits is 70 GB.¹

Input: parameter count and precision. Output: a fixed number of gigabytes that does not change with traffic. Lowering the bits per parameter is called quantization, and it is the first lever when weights do not fit. You now know the fixed cost; what remains on the card is the budget for everything that varies.

Step 3: The KV cache grows with every token and every request

The KV cache is stored per request, and it is large. Kwon et al. (2023) state that the KV cache memory of each request is huge and grows and shrinks dynamically, and that managing it badly wastes memory through fragmentation and duplication, which limits the batch size. A batch here means the set of requests the GPU processes together.

Per token the cache stores a key and a value for every layer: two, times layers, times KV heads, times head dimension, times bytes per number. For an illustrative model with 80 layers, 8 KV heads, a head dimension of 128 and 16-bit numbers, that is 327,680 bytes per token, or 1.3422 GB for one 4,096-token sequence.¹ Ten such sequences take 13.4 GB. On one 80 GB card with 8-bit weights and an illustrative 4 GB reserve, that leaves 6 GB, room for only 4 such sequences.¹ Weights are fixed; the cache scales with requests times context length. Paged attention reduces the waste Kwon et al. describe.

Step 4: Bandwidth sets how fast the bytes are read

Generating one token needs the weights, and the cache of that request, to be read from GPU memory. A simple lower bound for the time per token is the bytes read divided by the bandwidth. The H100 specification table lists 3.35 TB/s for the 80 GB variant. A 7B model at 16 bits is 14 GB, so the floor is about 4.179 milliseconds per token; a 70B model at 8 bits is 70 GB and the floor is 20.896 milliseconds.¹

These are estimates, not measurements: they ignore the cache reads, and real systems are slower. The point is the shape of the limit, which is why the same 80 GB of capacity can run at different speeds on cards with different bandwidth. The continuous batching page shows how serving systems amortize each read over many requests.

1 · Two memories2 · Weights3 · KV cache4 · BandwidthReplay

Step-by-step description

1. Host and GPU memory are separate spaces.
2. Weights fill GPU memory first.
3. The cache grows with each request.
4. Each token reads the bytes at bandwidth speed.

• Host memory and GPU memory are separate memory spaces joined by a link.
• The model weights fill GPU memory first.
• The KV cache grows with every token and every request.
• Memory bandwidth sets how fast the bytes are read for each token.

Worked example

Take one 80 GB card, the 70B model above, and the illustrative architecture from Step 3. Reserve 4 GB per card for buffers and overhead; that figure is an illustrative assumption, not a measurement. The free memory then pays for 4,096-token sequences at 1.3422 GB each.¹

Free memory is card capacity, minus the 4 GB reserve, minus the weights. Halving the weight bytes does not merely fit the model: it multiplies the batch, and the batch drives throughput, as Kwon et al. (2023) argue. Quantizing also halves the bytes read per token, so the decode floor for the 70B model drops from 20.896 to 10.448 milliseconds at 4 bits.¹

Text description of the diagram

• A 70B model at 16 bits needs 140 GB of weights.
• The same model at 8 bits needs 70 GB and at 4 bits needs 35 GB.

Text description of the diagram

• With 8-bit weights on one 80 GB card, 4 sequences of 4,096 tokens fit.
• With 4-bit weights, 30 sequences fit.
• With 16-bit weights on two cards, 8 sequences fit.

Trade-offs and edge cases

Capacity and bandwidth are separate specs, so a bigger card is not automatically a faster one. NVIDIA's H100 specification table lists 80 GB at 3.35 TB/s for H100 SXM and 94 GB at 3.9 TB/s for H100 NVL (as of October 2026), so check the exact variant before you do arithmetic.

Offloading moves data to CPU memory or a solid-state drive. Because host and device are separate memory spaces (NVIDIA CUDA guide), every offloaded byte crosses a link before the GPU can use it. A reasonable policy keeps what is read on every token, the weights of the active layers and the cache of running requests, on the card, and offloads what is cold, such as the cache of paused conversations; transfers should overlap with computation on other layers. This is design reasoning, not a sourced benchmark, so state the assumption and say you would measure it.

Activations are the third consumer of GPU memory: the temporary outputs each layer produces while it computes. Inference keeps only the current layer's activations, not the whole stack as training does. This working memory scales with the number of tokens processed at once, so the prefill of a long prompt is the peak, which is why the flat reserve above must be measured for your longest inputs.

Pre-allocating the cache for the maximum length also wastes memory, because most requests stop earlier (Kwon et al., 2023). Finally, the 4 GB reserve above varies with the serving stack, so treat it as a parameter to measure.

How to answer it in an interview

The spoken structure: name the three consumers, do the weights arithmetic, say the cache sets the batch, list the levers, then mention bandwidth.

Common mistakes

• "GPU memory only holds the weights." (wrong) The cache of every request is large and grows with each token (Kwon et al., 2023). (right) "Weights are the fixed cost; the KV cache is the variable cost."
• "More memory means faster." (wrong) Capacity and bandwidth are separate specs. (right) "Capacity decides what fits, bandwidth decides how fast it is read."
• "CPU memory and GPU memory are one pool." (wrong) They are separate host and device spaces. (right) "Offloading means moving data across a link, so I only offload cold data."
• "I reserve cache for the maximum length up front." (wrong) Reserving the maximum length wastes memory through fragmentation. (right) "I allocate cache in pages as tokens arrive."

Follow-up questions

Is KV cache stored in GPU memory?

Yes in standard serving: Kwon et al. (2023) treat the KV cache as a major consumer of GPU memory that limits the batch size. It can be moved elsewhere by offloading, at a transfer cost.

Should I offload the KV cache to CPU memory?

Only when the cache would otherwise stop you serving the request, and only for cold data such as paused conversations. Kwon et al. (2023) describe swapping some requests' cache out of GPU memory under pressure. Data read on every token should stay on the card, because the transfer link is slower than device memory.

How would you design an offloading policy when the model and KV cache do not fit in GPU memory?

First name the consumers: weights, KV cache and activations, the last peaking when a long prompt is prefilled. Shrink what you can: quantize the weights and page the cache. Then rank data by how often it is read, keep the hot part on the card, move the cold part to CPU memory or a solid-state drive, and start the transfer for the next layer while the current one computes. Measure before committing.

How much GPU memory does LLM inference need?

Weights plus KV cache plus overhead. Weights are parameters times bytes per parameter; the cache is bytes per token times context length times concurrent requests. The worked example above does both sums for a 70B model.

Does more GPU memory make a model faster?

Not by itself. More capacity lets a larger batch fit, which raises throughput, but the speed of one token is bounded by bandwidth. The LLM inference guide covers the other levers.

Sources

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv 2309.06180) — arXiv (Kwon et al., SOSP 2023), 12 Sep 2023
2. NVIDIA H100 Tensor Core GPU: specifications — NVIDIA (read 2026-10-26), 26 Oct 2026
3. CUDA C++ Programming Guide, Memory Hierarchy (version 13.0) — NVIDIA (read 2026-10-26), 26 Oct 2026