In this article
Before you start
Why interviewers ask this
Interviewers ask about HBM because it explains why a faster chip does not always mean a faster model. A candidate who says "the GPU has more floating-point operations per second (FLOPS)" has missed that single-stream generation spends most of its time waiting for memory. The question separates people who can estimate a speed limit from a bandwidth number from people who recite specifications. A weak answer lists HBM as "fast memory"; a strong answer divides the bytes read per token by the bandwidth and says what that implies about batching. It shows up in large language model (LLM) inference, serving and GPU fundamentals questions at AI labs and large technology companies.
High bandwidth memory from first principles
- High bandwidth memory
- High bandwidth memory is a DRAM design that stacks several memory dies vertically, connects them with through-silicon vias and a very wide interface, and sits next to the processor on the same package, so it delivers much more bandwidth than a conventional memory module.
- Through-silicon via
- A through-silicon via (TSV) is a vertical electrical connection that passes through a silicon die, so stacked dies can send signals to each other and to the base without long wires around the edge.
- Memory bandwidth
- Memory bandwidth is the amount of data a memory system can move to or from the processor per second, equal to the interface width times the data rate per wire divided by eight bits per byte.
In plain words
HBM is like replacing a single-lane road with a motorway of a thousand lanes at a lower speed limit. Each lane is not faster than before. The capacity comes from having many lanes at once. The road is also built right next to the factory, so the trucks do not travel far. The analogy breaks in one place. A motorway can be extended anywhere. HBM must sit right beside the processor. That limits how many stacks fit and how much memory a GPU can carry. Precisely, HBM trades the narrow, long, high-speed interface of ordinary memory for a very wide, short, low-speed one.
Step 1: Stack the memory dies
HBM stacks several DRAM dies on top of each other, so each stack holds a lot of memory in a small footprint. The input is a set of ordinary memory dies; the output is one stack, a short tower that behaves as one memory device. The memory standards body JEDEC (the JEDEC Solid State Technology Association) sets the HBM standard, and its release for the third generation of HBM (HBM3) supports stacks of 4, 8 and 12 dies, with a provision for 16 later (JEDEC, 27 January 2022).
Stacking is what lets a single device on the package hold many gigabytes without taking much board area. Height saves board area, not cost: the dies share one footprint and one connection to the processor, but taller stacks are harder to build and to cool. What you now know: capacity comes from height.
Step 2: Connect the layers with TSVs and a wide interface
Through-silicon vias carry signals up and down the stack, and the interface to the processor is very wide. The input is the stack of dies; the output is a channel to the processor made of many parallel wires. The same release states a data rate of up to 6.4 gigabits per second per pin and 819 gigabytes per second per device (JEDEC).
Multiplying 6.4 gigabits per second by an interface of 1,024 data wires and dividing by 8 gives 819.2 gigabytes per second, the 819 GB/s the release states; read the other way, the two figures imply 1,024 data wires per HBM3 stack, because command and address pins are extra.1 That width, and not a high clock speed, is where the bandwidth comes from. What you now know: bandwidth comes from width.
Step 3: Place the stacks beside the GPU
Each HBM stack sits next to the processor die on the same package, joined by a short, dense wiring layer called an interposer, which keeps the wires short enough to run so many in parallel. The input is the finished stack; the output is a GPU package with several stacks around the processor. Short wires make the high pin count of the interface practical. The cost is that only so many stacks fit around a die.
NVIDIA lists, as of October 2026, 80 GB of memory and 3.35 terabytes per second for the H100 SXM (NVIDIA H100) and, for the A100 80GB with HBM2e memory (a faster revision of the second-generation HBM standard), 2,039 gigabytes per second in the SXM form (NVIDIA A100). What you now know: capacity and bandwidth are capped by the number of stacks times what each stack holds.
Step 4: Read the speed limit off the bandwidth
Single-stream generation of a dense model reads every weight once per token, so the memory-bound speed limit is the weight bytes divided by the bandwidth. The input is the model size and the bandwidth; the output is the fastest possible token time. An 8-billion-parameter model in fp16 has 16 gigabytes of weights, and at 3,350 gigabytes per second that is 4.8 milliseconds per token, or about 209 tokens per second.1
The arithmetic is two floating-point operations for each two bytes read, or one operation per byte, which is far too little work to keep the arithmetic units busy. That is the sense in which decoding is memory-bound; a similar argument underlies the analysis of latency and memory in Pope et al. (2022, arXiv:2211.05102v1). What you now know: the number on the memory datasheet, not the arithmetic peak, sets single-stream speed.
Step-by-step description
- Dies are stacked into one tower.
- Vias and a wide interface join them.
- Stacks sit beside the GPU.
- Bandwidth sets the speed limit.
- HBM3 supports stacks of 4, 8 and 12 dies.
- In HBM3, 1,024 wires at 6.4 gigabits per second give 819.2 gigabytes per second per stack.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The speed limit of one stream is the bytes of weights read per token divided by the memory bandwidth. Take two illustrative models, 8 billion and 30 billion parameters, stored in fp16 at two bytes per parameter, and the two NVIDIA bandwidths above. The model sizes are illustrative, not products, and the results ignore the KV cache, compute time and software overhead, so real systems run slower.1 The 30-billion-parameter model has 60 gigabytes of weights, which fits in the 80 GB of one GPU with room left for the KV cache.
| Fact | A100 80GB SXM (2,039 GB/s) | H100 SXM (3,350 GB/s) |
|---|---|---|
| 8B model, weights read per token | 16 GB | 16 GB |
| 8B model, time per token | 7.8 milliseconds | 4.8 milliseconds |
| 8B model, tokens per second | 127 | 209 |
| 30B model, weights read per token | 60 GB | 60 GB |
| 30B model, time per token | 29.4 milliseconds | 17.9 milliseconds |
| 30B model, tokens per second | 34 | 56 |
The ratio of the two bandwidths is 1.64, and the table's two columns differ by that ratio, up to rounding.1 Batching changes the picture. Several requests in one step share a single read of the weights, so on the H100 the 30-billion-parameter ideal total throughput across the batch rises from 56 tokens per second at a batch of one to 447 at eight and about 1,800 at 32. Per-request speed does not rise: each request still gets at most about 56 tokens per second. The ideal also ignores KV-cache reads, which grow with batch times context, and arithmetic eventually becomes the limit.1
Text description of the diagram
- An 8B model in fp16 reads 16 GB per token, giving 127 tokens per second on A100 and 209 on H100.
- A 30B model reads 60 GB per token, giving 34 and 56 tokens per second.
Trade-offs and edge cases
HBM trades cost, capacity growth and packaging complexity for bandwidth, so it is used where bandwidth is the limit. Four cases matter in an interview.
First, bandwidth limits decoding but not prefill. Prefill processes the whole prompt in parallel, reuses each weight many times, and is usually limited by arithmetic, so extra bandwidth helps it less than it helps token-by-token generation.
Second, capacity and bandwidth are separate limits. The 80 GB of an H100 caps the model and cache you can hold; the 3.35 terabytes per second (NVIDIA H100) caps how fast you can read them. A model that does not fit must be split across GPUs, which adds communication.
Third, a mixture-of-experts model breaks the textbook answer. It reads only the experts each token is routed to, so the single-stream limit uses the active weight bytes, not the total: Mixtral has 47 billion parameters but uses 13 billion per token (Jiang et al., January 2024). The weights must still fit in memory, so capacity follows the total while speed follows the active share.
Fourth, the numbers move with each generation. The third-generation standard doubled the per-pin rate of the second-generation HBM (HBM2), and doubled the number of independent channels from 8 to 16 (JEDEC, January 2022). In April 2025 JEDEC published the fourth-generation standard, HBM4 (high bandwidth memory 4), which widens the interface to 2,048 bits at up to 8 gigabits per second, about 2 terabytes per second per stack, with 32 channels (JEDEC release, 16 April 2025). State the generation and the date with any bandwidth you quote. Conventional modules are easier to extend and cost less per gigabyte, which is why HBM is not used everywhere.
| Dimension | HBM | Conventional memory module |
|---|---|---|
| Interface | Very wide (1,024 wires per stack in HBM3), lower speed per wire | Narrow, higher speed per wire |
| Placement | On the package beside the processor | In a socket on the board |
| Bandwidth | Terabytes per second per GPU | Far less per processor |
| Capacity growth | Limited by stacks around the die | Easy to add modules |
| Cost per gigabyte | Higher | Lower |
Text description of the diagram
- HBM3 uses 1,024 wires per stack at a lower speed per wire; HBM4 uses 2,048.
- Conventional modules are easier to extend and cost less per gigabyte.
How to answer it in an interview
The answer states what HBM is, why it exists and the consequence for inference, in a sentence each. Say the definition, then the cost, then the trade-off.
What is high bandwidth memory, and why does it matter for LLM inference?
High bandwidth memory is stacked DRAM on the same package as the GPU. It reaches terabytes per second with a very wide interface, about a thousand wires per stack in HBM3, not a very fast narrow one. Single-stream decoding of a dense model reads every weight once per token, so the speed limit is weight bytes divided by bandwidth; a mixture-of-experts model reads only its active weights. For an 8-billion-parameter model in fp16 on an H100, that is 16 gigabytes over 3.35 terabytes per second, about 4.8 milliseconds per token. The trade-off is capacity and cost. I would add that batching raises total throughput, not one user's speed.
Common mistakes
- Calling HBM "faster memory". (wrong) "HBM has a higher clock speed." (right) "HBM has a much wider interface, so it moves more bytes per second even at a lower speed per wire."
- Quoting FLOPS as the inference speed limit. (wrong) "A GPU with twice the FLOPS generates tokens twice as fast." (right) "Single-stream decoding is limited by weight bytes divided by bandwidth."
- Forgetting the bytes per parameter. (wrong) "A 30B model reads 30 GB per token." (right) "At two bytes per parameter in fp16, it reads 60 GB."
- Treating bandwidth as capacity. (wrong) "The H100 has 3.35 terabytes of memory." (right) "It has 80 GB of memory and 3.35 terabytes per second of bandwidth."
- Applying the limit to prefill. (wrong) "Prefill is memory-bound for the same reason." (right) "Prefill reuses each weight across many tokens, so it is usually limited by arithmetic."
Follow-up questions
What is HBM memory used for in a GPU?
HBM holds the model weights, activations and the KV cache, and supplies them to the compute units at terabytes per second. Training and inference both read it constantly, which is why bandwidth appears on every datacenter GPU datasheet.
Why is LLM decoding limited by memory bandwidth?
In a dense model each generated token needs every weight once, and each weight is used for only about one operation per byte read at a batch of one. The arithmetic units finish long before the next weights arrive. See the LLM inference overview for the full pipeline.
What is the difference between HBM and GDDR?
GDDR (graphics double data rate) memory is made of separate chips soldered on the board around the GPU, each with a narrow interface at a high speed per pin. HBM stacks dies on the package and uses a very wide interface at a lower speed per pin. Neither can be added to later.
How does batching change the bandwidth limit?
Batching shares one read of the weights across requests, so the ideal total throughput across the batch rises almost in proportion to the batch size, while each request's own speed stays at most the batch-of-one limit. The ideal ignores KV-cache reads, which grow with batch times context, and arithmetic eventually becomes the limit. FlashAttention reduces the memory traffic of the attention step itself (FlashAttention).
Is the KV cache stored in HBM?
Yes, on a datacenter GPU the KV cache normally lives in the same HBM as the weights, so it competes with them for both capacity and bandwidth. It is read at every decoding step and grows with batch times context. See KV cache.
Does more bandwidth help prefill?
Less than it helps decoding, because prefill is usually compute-limited and a higher arithmetic rate helps it more. The two 16-bit formats, fp16 versus bf16, both store two bytes per parameter, so switching between them leaves the decoding limit unchanged. Quantising the weights to 8 or 4 bits halves or quarters the bytes read per token, which raises that limit by two or four times.
- HBM is stacked DRAM beside the GPU with a very wide interface.
- Bandwidth is width times data rate divided by eight: In HBM3, 1,024 wires at 6.4 gigabits per second give 819.2 gigabytes per second per stack; HBM4 widens the interface to 2,048 bits.
- Single-stream decoding speed is weight bytes divided by bandwidth.
- Capacity and bandwidth are separate limits.
- Batching shares the weight read; the KV cache grows with it.
Sources
- JEDEC Publishes HBM3 Update to High Bandwidth Memory (HBM) Standard (JESD238; reprint of the 27 January 2022 release) — JEDEC Solid State Technology Association, via Design & Reuse, 27 Jan 2022
- JEDEC and Industry Leaders Collaborate to Release JESD270-4 HBM4 Standard (release of 16 April 2025) — JEDEC Solid State Technology Association, 16 Apr 2025
- NVIDIA H100 Tensor Core GPU (product page, undated, read 2026-10-24) — NVIDIA, 24 Oct 2026
- NVIDIA A100 Tensor Core GPU (product page, undated, read 2026-10-24) — NVIDIA, 24 Oct 2026
- Mixtral of Experts (arXiv:2401.04088v1, 8 January 2024) — arXiv (Jiang et al.), 8 Jan 2024
- Efficiently Scaling Transformer Inference (arXiv:2211.05102v1) — arXiv (Pope et al.), 9 Nov 2022
