GPU fundamentals for ML engineers: memory, compute and the roofline

In this article

1. Before you start
2. Why interviewers ask this
3. GPU fundamentals from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Questions about GPUs come in many forms: why is a central processing unit (CPU) slower than a GPU for deep learning, what does memory-bound mean, what is kernel fusion, how would you tell whether a kernel is limited by compute or memory. They all test one skill, which is estimating where time goes before running anything. A weak answer lists hardware words: more cores, higher clock, Tensor Cores.

A strong answer counts operations and bytes, compares the ratio with the machine's own ratio and concludes what to change. That skill is also what makes large language model (LLM) serving questions answerable, since decoding is a memory problem and prefill is a compute problem. This page builds the estimate in five steps and checks it with numbers.

GPU fundamentals from first principles

Arithmetic intensity

Arithmetic intensity is the number of floating-point operations (FLOPs) a piece of code performs divided by the number of bytes it moves to and from memory, measured in FLOPs per byte.

Roofline

The roofline is a performance model that plots attainable throughput against arithmetic intensity; throughput rises with intensity until it hits the processor's peak compute, the flat roof.

In plain words

A CPU is a small team of brilliant specialists. A GPU is a stadium of clerks who each do one simple step from the same instruction sheet. Neural networks mostly multiply large tables of numbers, which splits into millions of independent small sums, so the stadium wins. The analogy breaks in one place. The clerks are only as fast as the warehouse that supplies their paper, and in modern ML the warehouse is usually the slow part.

In one sentence: a GPU is fast when there is enough parallel work and enough arithmetic per byte fetched; otherwise it waits for memory.

Step 1: Thousands of simple threads, grouped in warps

NVIDIA's Compute Unified Device Architecture (CUDA) programming guide says a CPU typically has several cores, while a GPU has many streaming multiprocessors (SMs) that run huge numbers of threads in parallel. Threads are grouped into warps of 32 that execute the same code under the single-instruction, multiple-threads model.

Each thread may take its own branch, but when threads in a warp take different branches the non-matching threads are masked off and sit idle; this is warp divergence, and it lowers utilization. The input to this step is a data-parallel job, such as one output element per thread; the output is a GPU kept busy only if every warp follows one path. Matrix multiplication is the ideal job: no thread's work depends on another's.

1 · Many simple threads2 · Memory hierarchy3 · Count FLOPs and bytes4 · Compare to the ridge5 · Fuse kernelsReplay

Step-by-step description

1. Threads run in warps of 32; divergent branches idle part of a warp.
2. Small fast SRAM sits on chip; large HBM holds weights at 3.35 TB/s.
3. Intensity = FLOPs / bytes; one-token decode is about 1.0.
4. Ridge = peak / bandwidth, about 295 FLOPs per byte on an H100.
5. Fewer round trips to HBM: 12 GB of traffic becomes 4 GB.

• Threads are grouped into warps of 32 that execute the same code.
• HBM is large but slower and on-chip SRAM is small but fast.
• Arithmetic intensity is FLOPs divided by bytes moved; one-token decoding is about 1.0.
• The ridge point of an H100 in 16-bit is about 295 FLOPs per byte.
• Fusing three elementwise operations cuts the bytes moved from 12 gigabytes to 4 gigabytes.

Step 2: Data lives in a memory hierarchy

The arithmetic units read operands from a hierarchy: registers; shared memory, a small block of static random-access memory (SRAM) on the chip that one block of threads shares; a cache; and the large off-chip high-bandwidth memory (HBM) that holds weights and activations. The FlashAttention paper describes HBM as large but slower and on-chip SRAM as small but fast, and argues the bottleneck is often memory access rather than arithmetic.

NVIDIA's H100 page lists 80 GB of memory and 3.35 terabytes per second of bandwidth for the SXM version, as of October 2026. Every weight used in a layer must travel from HBM to the arithmetic units at that rate. Whatever the compute units could do in the time is wasted if the bytes have not arrived.

Text description of the diagram

• HBM is large but slower and on-chip SRAM is small but fast.
• The H100 SXM has 80 GB of memory and 3.35 terabytes per second of bandwidth.

Step 3: Count the work and the bytes

For any operation, count its FLOPs and the bytes it must read and write. NVIDIA's performance guide defines arithmetic intensity as operations divided by bytes accessed, and shows a linear layer at batch size 512 near 315 FLOPs per byte but at batch size 1 near one, with ReLU at 0.25.

Take a matrix multiplication of an M by K input with a K by N weight, in 16-bit numbers. FLOPs are 2 times M times N times K (a multiply and an add per term). Bytes are 2 times the sum of the three matrix sizes. For one token (M = 1) against a 4,096 by 4,096 weight, FLOPs are 33.6 million and bytes 33.6 million: intensity 1.0.¹ Each weight is read once, used once, and discarded.

Step 4: Compare with the ridge point

A machine has its own ratio, its peak compute divided by its memory bandwidth, called the ridge point. NVIDIA's H100 page prints 1,979 teraFLOPS for 16-bit Tensor Core work and marks it as the figure with sparsity; NVIDIA's H100 datasheet carries the footnote "Shown with sparsity. Specifications 1/2 lower without sparsity", so dense peak is 989.5 teraFLOPS. Divided by 3.35 terabytes per second, the ridge point is about 295 FLOPs per byte.¹

The rule from the roofline model (Williams, Waterman and Patterson, 2008): below the ridge point an operation is memory-bound and attainable throughput is intensity times bandwidth; above it is compute-bound and limited by peak. One-token decoding at intensity 1.0 reaches 3.3 teraFLOPS, 0.34% of peak.¹

Matrix multiplications run on Tensor Cores. A Tensor Core is a unit that multiplies small blocks of a matrix in one instruction. NVIDIA's performance guide notes that other math runs on the ordinary CUDA cores, which have a far lower roof (67 teraFLOPS in 32-bit on the H100 page).

Step 5: Fuse kernels to raise intensity

If a computation is memory-bound, doing less arithmetic does not help; moving fewer bytes does. Kernel fusion combines several operations into one kernel so that intermediate results stay in registers or SRAM instead of making a round trip to HBM.

Take three elementwise operations on one billion 16-bit values. Run separately, each reads and writes the tensor, 12 gigabytes in all; fused, the tensor is read and written once, 4 gigabytes. The FLOPs are unchanged, so intensity rises from 0.25 to 0.75 and the memory-bound time falls threefold.¹ FlashAttention applies the same idea to attention, tiling so intermediate scores stay in SRAM; its abstract reports a 3x speedup when training a generative pre-trained transformer (GPT), GPT-2, at sequence length 1K. See the FlashAttention page.

Worked example

Apply steps 3 and 4 to a 4,096 by 4,096 weight at several batch sizes, on the H100 numbers above.¹

Text description of the diagram

• Decoding one token reaches 0.34% of peak and batch 64 reaches 21%.
• Decode at batch 256 reaches 77% of peak and a 2,048-token prefill reaches 100%.

Reading the table: going from batch 1 to batch 256 multiplies the work by 256 but the time only from 10.0 to 11.3 microseconds, because the time is the time to read the weights. That is the arithmetic behind batching requests in a serving system, covered in the LLM inference hub.

A second consequence: an 8-billion-parameter model in 16-bit weights is 16 gigabytes. Reading it once per token at 3.35 terabytes per second takes 4.78 milliseconds, so a single H100 cannot decode faster than about 209 tokens per second at batch 1, ignoring the key-value (KV) cache and overhead.¹ The ceiling comes from bandwidth, not from peak FLOPs.

Trade-offs and edge cases

Peaks are ceilings. Real kernels read inputs more than once and rarely reach the roof; NVIDIA says profiler data is more reliable than the first-order estimate.

Parallelism can be the limit. If a kernel launches too few thread blocks, latency dominates whatever the intensity; NVIDIA's rule of thumb is about four times as many thread blocks as SMs.

Elementwise layers are always memory-bound. Activations, scale, bias and add do very little work per byte.

Precision moves the ridge. In 8-bit floating point (FP8), NVIDIA's H100 page lists 3,958 teraFLOPS with sparsity (as of October 2026), so dense peak is about 1,979 and the ridge point about 591. A matrix multiplication whose inputs and output are all FP8 moves half the bytes, so its intensity doubles too: it stays on the same side of the ridge and a memory-bound decode step takes about half the time.¹ Only data that stays 16-bit, such as outputs written in BF16 or elementwise layers, falls further below the ridge.

Misaligned dimensions waste Tensor Cores. NVIDIA's matrix multiplication guide says that with 16-bit data, matrix dimensions should be multiples of 8 elements for best efficiency (64 on the A100).

How to answer it in an interview

The spoken structure is: define intensity, compare with the ridge, give the decode and prefill examples, then name the fix.

Common mistakes

"A GPU is faster because its clock is higher." (wrong) The advantage comes from parallel work, not from a faster clock. (right) "It runs thousands of simple threads on data-parallel work."

"More teraFLOPS means faster inference." (wrong) Decoding is limited by memory bandwidth. (right) "Decode is bandwidth-bound; compare bytes to bandwidth."

"Fusion reduces the number of FLOPs." (wrong) The FLOPs are unchanged. (right) "Fusion reduces bytes moved, so intensity rises."

"Aim for 100% occupancy." (wrong) Occupancy, the ratio of active warps on a multiprocessor to the most it can hold, is not the only metric that matters. (right) "Check the roofline position and the profiler first."

Follow-up questions

Why is a CPU slower than a GPU for deep learning?

A CPU has a few complex cores built for low latency on varied work; a GPU has many SMs running thousands of simple threads for throughput, which suits matrix multiplication. The CPU is built to finish one varied task quickly, the GPU to finish a very large number of identical tasks.

What is warp divergence?

Threads in a warp share an instruction stream; when they branch differently, the non-matching threads are masked and idle, so a half-taken branch can halve utilization of that warp.

What is kernel fusion, and why does it matter for ML workloads?

It merges operations into one kernel so intermediates stay on chip; it cuts bytes moved, which is what limits memory-bound operations like activations and normalization.

How would you get the roofline of a kernel?

Measure its FLOPs and bytes moved with a profiler, divide for intensity, and plot it against the machine's bandwidth slope and compute roof; the position says which limit applies. See deep learning fundamentals for the layers involved.

What changes if the model runs in FP8?

Bytes per weight halve, so a memory-bound decode step takes about half the time. Peak compute also doubles, so the ridge rises to about 591, but an all-FP8 matrix multiplication's intensity doubles as well and its bound does not change; only tensors kept in 16 bits fall further below the ridge.

Sources

1. CUDA Programming Guide: Programming Model — NVIDIA, 24 Oct 2026
2. GPU Performance Background User's Guide — NVIDIA Deep Learning Performance documentation, 24 Oct 2026
3. NVIDIA H100 Tensor Core GPU — NVIDIA, 24 Oct 2026
4. NVIDIA H100 Tensor Core GPU Datasheet (copy hosted by Microway) — NVIDIA (earlier revision, rounded figures), 1 Apr 2024
5. Matrix Multiplication Background User's Guide — NVIDIA Deep Learning Performance documentation, 24 Oct 2026
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — arXiv (Dao et al.), v2, 23 Jun 2022
7. Roofline: An Insightful Visual Performance Model for Multicore Architectures (technical report UCB/EECS-2008-134) — UC Berkeley EECS (Williams, Waterman, Patterson), 17 Oct 2008