How to optimize an ML kernel that underutilizes the GPU

In this article

1. Before you start
2. Why interviewers ask this
3. Kernel underutilization from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the large language model (LLM) inference optimization guide, and the model behind it is explained in how transformers work.

Why interviewers ask this

A candidate who answers "launch more threads" often fixes the wrong problem, and the interviewer hears it in one sentence. Candidates report this question in machine learning systems and GPU performance interviews. It separates people who diagnose from people who tune by habit.

A weak answer lists tricks: fuse kernels, raise occupancy, use Tensor Cores. A strong answer says the first move is to find why the GPU is idle, and names the test that tells the causes apart. A Staff-level answer also says what each class of fix cannot help, shows a number, and finishes by re-measuring. The page below gives you the diagnosis, the numbers behind it and a sayable answer.

Kernel underutilization from first principles

Memory-bound kernel

A memory-bound kernel is a kernel whose run time is set by how fast it can load inputs and write outputs rather than by how fast it can calculate, so making its math faster does not make it faster. The opposite case is a math-bound kernel, limited by calculation rate.

In plain words

Picture a factory with very fast machines and one narrow delivery door. If parts arrive slowly, the machines wait, and buying faster machines changes nothing. If the factory has too few jobs, the machines also sit idle, but now the fix is more work, not a wider door. The analogy breaks in one place: a GPU has thousands of machines, and the job must be cut into many small pieces before any of them can start.

A GPU is underutilized for one of two reasons. Either its kernel moves too many bytes for the math it does, or it exposes too little parallel work to keep every streaming multiprocessor (SM) busy. NVIDIA's guide describes a GPU as SMs, an on-chip L2 cache and high-bandwidth dynamic random-access memory (DRAM). It also lists too little parallelism as a cause of underuse alongside memory limits.

Step 1: Measure, then compare arithmetic intensity with the ops:byte ratio

Profile the kernel first, then classify it with two measurements. Classify by intensity and by achieved bandwidth or math rate against peak. A profiler such as Nsight Compute reports how close compute and memory are to their peaks (the speed-of-light percentages), the achieved occupancy and the reasons warps stall. The ops:byte ratio is a processor's math rate divided by its memory bandwidth. A kernel is math limited if its intensity is higher than that ratio and memory limited if it is lower. For a V100, NVIDIA gives 900 GB/s of memory bandwidth, and 138.9 FLOPs per byte for 16-bit floating point (FP16) Tensor Core work loaded from device memory. NVIDIA calls the comparison a simplified rule of thumb.

Intensity only sets the ceiling, so three outcomes follow. Near the bandwidth roof, only fewer bytes help. Below both roofs, add parallelism and coalesce, because too few bytes are in flight. Near the math roof, use Tensor Cores.

What the intensity of common kernels looks like

A general matrix multiply (GEMM) of M by K times K by N in 16-bit numbers does twice M times N times K FLOPs and moves two bytes for every number in its two inputs and its output. At 8192 on every side, that is about 2,730 FLOPs per byte, the same figure NVIDIA's matrix guide reports, so it is math limited.¹ A general matrix-vector product (GEMV) is always memory limited, with intensity below 1. An elementwise add of two 32-bit arrays does 1 FLOP per 12 bytes, an intensity of 0.0833.¹ Activation functions, pooling and batch normalization are nearly always memory-bound.

A 4096 by 4096 half-precision layer has intensity near 1 at batch 1. Intensity passes the V100 ratio of 138.9 between batch 128 and batch 256.¹ It follows from the matrix-vector rule that language model decoding at batch size 1 is memory-bound: each token multiplies a vector by weight matrices. That is why continuous batching matters. For the other side of the memory bill, see the key-value (KV) cache.

Text description of the diagram

• A 4096 by 4096 half-precision layer has intensity near 1 at batch 1.
• Intensity passes the V100 ratio of 138.9 between batch 128 and batch 256.

Step 2: Move fewer bytes by fusing and tiling

If the kernel is memory-bound and near peak bandwidth, the fix is to move fewer bytes, because speeding up calculation does not improve performance. Kernel fusion does this: each separate kernel writes its result to off-chip memory and the next one reads it back. Take add, then an activation, then a scale by a constant, on 100 million 32-bit numbers (illustrative). As three kernels they move 2.8 GB; fused into one they move 1.2 GB, because only the two inputs are read and the output is written once.¹ At the 900 GB/s of a V100, that is 3.11 milliseconds against 1.33, a gain of 2.33 times with the same math.¹

Tiling works on a block of data held in on-chip memory, so each byte leaves off-chip memory once. FlashAttention uses tiling and fusion so the large N×N attention matrix is never written to HBM, and reports a 3 times end-to-end training speedup on generative pre-trained transformer (GPT)-2 (Generative Pre-trained Transformer 2) at sequence length 1K (Dao et al., 2022), although recomputation adds FLOPs. Fewer bytes beat fewer FLOPs.

Step 3: Make each memory read useful with coalescing and fewer bits

Memory-bound kernels also waste bytes when a warp, the group of threads that execute together, reads data in a scattered pattern. For recent GPUs, the accesses of a warp are combined into as few 32-byte transactions as possible, and this combining is called coalescing; NVIDIA marks coalesced global memory access as high priority. The programming guide states the goal as maximizing the ratio of bytes used to bytes transferred. To measure the waste, compare requested bandwidth with actual bandwidth.

Fewer bits per value cut bytes directly: a 16-bit number takes half the bytes of a 32-bit one. If the fused chain above ran in 16-bit numbers, it would move 0.6 GB and take 0.67 milliseconds at the same bandwidth.¹ The page on LLM quantization covers how far this goes without hurting accuracy.

Step 4: If the GPU is idle rather than waiting, add parallel work

If the kernel sits below both roofs, too little work is in flight, so a starved kernel needs more parallel work. NVIDIA says a processor is under-utilized when the work is not large enough or lacks parallelism. Occupancy is the ratio of active warps per multiprocessor to the maximum possible active warps, and running other warps is the only way to hide the latency of a stalled warp.

Registers per thread and shared memory per block limit how many warps fit on an SM, as the occupancy section of the Compute Unified Device Architecture (CUDA) best practices guide explains. So a memory-bound kernel that runs well below peak bandwidth often gains from more warps or wider loads, because more bytes are in flight; the stall reasons in the profiler tell you which. NVIDIA's point is narrower: for a memory-bound layer, tuning its parameters for math efficiency is ineffective.

Practical fixes: launch a grid with many more blocks than SMs, and watch wave quantization, where the last wave of blocks leaves some SMs idle, as NVIDIA's matrix guide describes. Batch requests so each launch has more rows. Fuse many tiny kernels, or record them in a Compute Unified Device Architecture (CUDA) graph, so launch overhead stops dominating. Check that the central processing unit (CPU) feeds data fast enough. Inside the kernel, shared-memory bank conflicts and warp divergence are two other common wastes.

Step 5: Re-profile and stop near the roofline

Re-profile after every change and stop when the kernel sits near the limit the hardware sets. A kernel cannot finish faster than the larger of its math time and its memory time, a bound called the roofline (Williams, Waterman and Patterson, 2009). Compare achieved bandwidth with peak for a memory-bound kernel, and achieved math rate with peak for a math-bound one. If the kernel is now math-bound, Tensor Cores are the lever, and FP16 dimensions work best as multiples of 8 elements. The deeper hardware picture is in GPU fundamentals for ML engineers.

Fewer FLOPs is not the goal: FlashAttention is faster despite more FLOPs, and its authors note that approximate methods that cut FLOPs often do not achieve wall-clock speedup.

In short, profile first and classify the kernel. One near the bandwidth roof needs fewer bytes moved, one below both roofs needs more parallel work in flight, and one near the math roof needs Tensor Cores.

1 · Profile and classify2 · Fuse and tile3 · Coalesce loads4 · Add parallel work5 · Tensor Cores, re-profileReplay

Step-by-step description

1. Intensity and achieved rate vs peak.
2. Near bandwidth roof: fewer bytes.
3. Below both roofs: useful reads.
4. Below both roofs: more in flight.
5. Near math roof; then measure again.

• Classify by intensity and by achieved bandwidth or math rate against peak.
• Near the bandwidth roof, only fewer bytes help.
• Below both roofs, add parallelism and coalesce.
• Near the math roof, use Tensor Cores.

The Staff-level answer

What makes it Staff-level: it diagnoses before fixing, names the test, and says what each class of fix cannot help.

The rubric

Three variants of this question

How would you speed up a slow elementwise operation?

An elementwise operation has very low arithmetic intensity, so it is memory-bound. Fuse it with its neighbors into one kernel, by a compiler or by hand, so intermediate results never reach off-chip memory.

Why is language model decoding at batch size 1 slow on a fast GPU?

Each token multiplies a vector by large weight matrices, and matrix-vector products are memory limited, so the GPU waits on weights rather than calculating. Batching more requests raises intensity; see continuous batching.

What if the kernel is already math-bound and the GPU looks busy?

Then bytes are not the limit. Use Tensor Cores with aligned dimensions and lower precision where accuracy allows, as in 8-bit floating point (FP8) training, and accept that you are near the roofline.

Common mistakes

• "I would raise occupancy first." (wrong) Occupancy only hides latency, and it cannot help a kernel that already sits at the bandwidth roof. (right) "I check achieved bandwidth first; occupancy helps only if the kernel is below the bandwidth roof."
• "Tensor Cores will fix it." (wrong) They speed up math, which only helps math-bound kernels. (right) "I enable them when intensity is above the ops:byte ratio."
• "Fewer FLOPs means faster." (wrong) FlashAttention does more FLOPs and is faster. (right) "I count bytes moved as well as FLOPs."
• "The monitor says the GPU is busy, so the kernel is efficient." (wrong) A coarse busy figure does not show useful work. (right) "I compare achieved bandwidth and math rate with the hardware peaks."
• "Every matrix multiply is compute-bound." (wrong) Small, skinny and matrix-vector products are memory limited. (right) "I check the shape, because intensity depends on it."

Follow-up questions

How do you know a kernel is memory-bound?

Compute its arithmetic intensity, FLOPs divided by bytes moved, and compare it with the GPU's ops:byte ratio; a lower value means the ceiling is the bandwidth roof. Confirm with a profiler, since NVIDIA calls the comparison a simplified rule of thumb: achieved bandwidth near peak means fewer bytes is the fix, and well below peak means too few bytes are in flight.

Why does FlashAttention run faster while doing more FLOPs?

It tiles the work in on-chip memory and fuses the attention steps, so the N×N matrix is not written to or read from HBM. Most of standard attention's steps, such as the softmax, masking and dropout, are memory-bound, so the saved bytes outweigh the recomputed FLOPs.

Does a larger batch size always fix low GPU utilization?

No. A larger batch raises arithmetic intensity and exposes more parallel work, which helps memory-bound and starved kernels. It does not help a kernel already limited by math, and it needs more memory.

When does kernel fusion not help?

Fusion removes bytes moved, so it helps little when the kernel is already math-bound. It also cannot fix a kernel whose problem is too little parallel work.

How does the answer change at cluster scale?

Communication between GPUs becomes another place where devices sit idle, on top of kernel efficiency. The large-scale training and inference page covers that layer.

Sources

1. GPU Performance Background User's Guide — NVIDIA Deep Learning Performance documentation (accessed 2026-10-26), 26 Oct 2026
2. Matrix Multiplication Background User's Guide — NVIDIA Deep Learning Performance documentation (accessed 2026-10-26), 26 Oct 2026
3. Get Started With Deep Learning Performance — NVIDIA Deep Learning Performance documentation (accessed 2026-10-26), 26 Oct 2026
4. CUDA C++ Best Practices Guide, release 13.4 — NVIDIA, 12 Oct 2026
5. Writing kernels, CUDA Programming Guide — NVIDIA (accessed 2026-10-26), 26 Oct 2026
6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv 2205.14135) — arXiv (Dao, Fu, Ermon, Rudra, Ré), 27 May 2022
7. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Communications of the ACM (Williams, Waterman, Patterson), 1 Apr 2009