GPU kernel questions in ML coding rounds: coalescing and fusion, step by step

In this article

1. Before you start
2. Why interviewers ask this
3. GPU kernel questions from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the PyTorch and ML coding round learning path.

Why interviewers ask this

A kernel question separates candidates who can write correct code from candidates who can explain why that code is slow. Candidates report these prompts in interviews at GPU makers and AI inference companies, plus cross-company prompts such as a fused softmax in Triton; they are candidate reports, not statements by any company.

The prompts vary: why vectorized loads help, a quantized linear layer, a reduction or softmax in Compute Unified Device Architecture (Compute Unified Device Architecture (CUDA), NVIDIA's GPU programming platform), a dot product, a tiled matrix multiply. The idea underneath barely changes. A weak answer writes a working loop and stops. A strong answer states what limits the kernel, fixes that first, and says what the fix costs. This page teaches that single habit with numbers you can redo by hand, so each variant of the question becomes familiar.

GPU kernel questions from first principles

GPU kernel

A GPU kernel is a function that runs on a graphics processing unit, is started once, and is executed by thousands of threads at the same time, each working on its own part of the data. Its speed is usually set by how it reads and writes memory.

Warp

A warp is a fixed-size group of threads, thirty-two on NVIDIA GPUs, that normally execute in lockstep. The hardware judges the memory addresses of a whole warp as one request, which is why access patterns matter.

Coalesced access

Coalesced access is a load or store pattern in which the threads of one warp touch neighboring addresses, so the hardware merges their requests into as few memory transactions as possible, as the NVIDIA best practices guide describes.

Kernel fusion

Kernel fusion is the practice of doing several steps of a computation inside one kernel, so intermediate results stay in fast on-chip memory instead of being written to global memory and read back by the next kernel.

In plain words

Picture thirty-two workers sent to a warehouse for one box each. If the boxes sit side by side on one shelf, one trip serves everyone. If the boxes are scattered across aisles, the trips multiply. The analogy breaks in one place: real hardware fetches fixed-size pieces of memory, called sectors. Any bytes in a sector that no thread asked for are wasted. The warehouse also has no cost for walking a longer aisle, but a GPU does pay for every extra sector it must fetch.

In one precise sentence, a GPU kernel is fast when each warp's loads land in as few memory sectors as possible, each loaded byte is used many times, and intermediate results never leave the chip.

Step 1: Find out whether the kernel waits on memory or on math

Find out first whether the kernel waits on memory or on math. A kernel is called memory-bound when its run time is set by how many bytes it moves rather than by how many arithmetic operations it performs. Arithmetic intensity is the number of arithmetic operations a kernel performs per byte it moves to or from global memory; the roofline idea compares it with the machine's balance, the peak operations per second divided by the peak bytes per second, and a kernel whose intensity is below that balance is memory-bound.

Input: the kernel's loop body. What changes: you count operations against memory accesses. Output: a named bottleneck. Adding two vectors does one addition for three memory operations per element, two loads and one store, which is 12 bytes of float traffic per addition, an intensity of about 0.083 operations per byte, so memory sets its speed.¹ Softmax and a dot product are the same kind of problem.

A matrix multiply is different, because each input value is needed many times, so the question is how often you reload it. A naive kernel does about 0.25 operations per byte, while a tile of thirty-two raises that to 8, because intensity grows with the tile size.¹ A GPU's balance comes from its vendor datasheet, so quote it with its source and date rather than from memory. You now know what to fix; the next step fixes the access pattern.

Step 2: Coalesced access makes a warp's loads one tidy block

A warp that reads thirty-two consecutive floats moves one tidy 128-byte block (illustrative sizes, computed in the script),¹ while the same warp reading with a large stride moves many sectors to use the same bytes. NVIDIA's guide sets the priority: "High Priority: Ensure global memory accesses are coalesced whenever possible." On devices of compute capability 6.0 or higher, it says concurrent accesses coalesce into a number of transactions "equal to the number of 32-byte transactions necessary to service all of the threads of the warp". The unit of waste is therefore the 32-byte sector.

Input: the address each thread reads. What changes: how many distinct sectors those addresses touch. Output: bytes moved against bytes used. Consecutive floats touch four sectors and use all 128 bytes.¹ Stride 8 or 32 touches thirty-two sectors, moving 1,024 bytes to use 128, which is 12.5%.¹

Wider loads do not repair this on their own. If each thread loads a float4 but the threads sit 32 bytes apart, the warp touches thirty-two sectors, moves 1,024 bytes and uses 512, which is 50%, against 100% for consecutive float4 loads; the scalar stride-8 case stays at 12.5%.¹ Widening helps only partly, because coalescing is about how the warp's addresses line up. The real fix is to let thread index map to the fastest-moving array index, so neighbors read neighbors.

Step 3: Vectorized loads cut instructions, not bytes

Vectorized loads are better because one instruction moves more bytes, not because the memory is faster. NVIDIA's blog says that using vectorized loads "reduces the total number of instructions, reduces latency, and improves bandwidth utilization". Widths of 64 or 128 bits are available.¹

Input: a loop in which each thread loads one float. What changes: each thread loads a float4, four floats packed into sixteen bytes, through a pointer cast. Output: the same bytes in a quarter of the load and store instructions. For a vector add over 1,048,576 floats, that is 3,145,728 memory instructions against 786,432.¹

The cost is alignment. The blog notes that pointers must be aligned to the vector type's size, so a float4 address must be a multiple of sixteen bytes, and a length that is not a multiple of four needs a scalar tail. With consecutive threads, thirty-two sixteen-byte loads still cover 512 consecutive bytes, sixteen sectors, so coalescing is kept.¹

Step 4: Fusion and tiling reuse data instead of reloading it

Reuse is the last fix: load a value into fast on-chip memory once and use it several times before it is evicted. Two forms matter in interviews, fusion and tiling.

For fusion, the Triton fused-softmax tutorial counts the unfused version, as 5MN + 2M elements read and 3MN + 2M written for an M-by-N matrix. It fuses the steps into one kernel in which each program processes a set of rows, strided by the number of programs, which is faster for rows that fit in the on-chip static random-access memory (SRAM).

For matrix multiply, the method is called tiling. Each block copies a small square tile of each input into shared memory, the on-chip memory its threads share, and every thread reuses the tile many times. Coalescing cuts wasted bytes, vectorizing cuts instructions, and reuse cuts reloads.

1 · Find the bottleneck2 · Coalesce3 · Vectorize4 · Reuse dataReplay

Step-by-step description

1. Decide whether memory or math limits the kernel.
2. Neighboring threads read neighboring addresses.
3. Wider aligned loads cut instruction count.
4. Fuse and tile so data is loaded once.

• Find out first whether the kernel waits on memory or on math.
• Coalescing cuts wasted bytes, vectorizing cuts instructions, and reuse cuts reloads.

Worked example

Every figure below comes from a short script, so you can redo each with a calculator.¹

Coalescing: how many bytes does a strided warp waste?

Take a warp of thirty-two threads, each loading one four-byte float, with sectors of thirty-two bytes. With stride 1, the warp moves 128 bytes and uses all 128 bytes it moves.¹ With stride 2 it moves 256 bytes, so 50% is useful; with stride 4 it moves 512 bytes, so 25%.¹ At stride 8 a warp moves 1,024 bytes to use 128, which is 12.5%, and stride 32 is no better.¹ Past stride 8, every thread already has a sector to itself, so spreading further adds nothing.

Text description of the diagram

• With stride 1, the warp moves 128 bytes and uses all 128 bytes it moves.
• At stride 8 a warp moves 1,024 bytes to use 128, which is 12.5%.

Softmax: how much traffic does fusion remove?

Take a 4,096 by 1,024 matrix of four-byte floats. Using the tutorial's counts, unfused softmax moves 128.1 mebibytes of global memory traffic.¹ A fused kernel reads each element once and writes it once, as the tutorial describes, so it moves 32.0 mebibytes, a ratio of about four.¹ The math is unchanged; only the traffic shrinks. The usual stability step of subtracting the row maximum stays, since the tutorial notes softmax is invariant to that shift; see numerically stable softmax.

Text description of the diagram

• Unfused softmax moves 128.1 mebibytes of global memory traffic.
• A fused kernel that reads each element once and writes it once moves 32.0 mebibytes.

Dot product, reduction and matrix multiply: how many steps or loads?

A sum of 1,024 numbers takes 1,023 additions. Done one after another, those are 1,023 dependent steps; as a tree, where pairs add in each round, it takes ten rounds, and the last five happen inside one warp, where threads exchange values with warp shuffle instructions and need no block-wide synchronization.¹ A dot product is an elementwise multiply followed by exactly this sum.

For a 1,024 by 1,024 matrix multiply, a naive kernel has each output read one row and one column, 2,147,483,648 global loads in total. Tiles of thirty-two by thirty-two cut that by a factor of thirty-two to 67,108,864, using 8,192 bytes of shared memory for two float tiles.¹

Quantized linear layer: what does one scale per channel do?

A per-channel scale is one scaling number for each output channel, that is, each row of the weight matrix, chosen so the row's largest weight maps to 127 in 8-bit integer (INT8) storage. The output is the float input multiplied by the row's INT8 weights, then by that row's scale, then the bias is added. INT8 weights take a quarter of the bytes of float32 weights, which is the traffic saving; the per-channel scale protects accuracy. Illustrative toy: input (1.0, -2.0, 0.5), weight rows (0.5, -1.0, 0.25) and (2.0, 1.0, -4.0), bias (0.1, -0.2).

Row one quantizes to 64, -127, 32 and row two to 64, 32, -127.¹ The float outputs are 2.725 and -2.2; the INT8 outputs are 2.7299 and -2.2.¹ With one scale for the whole matrix, row one would collapse to 16, -32 and 8, losing precision.¹ This is weight-only quantization; a full INT8 kernel also quantizes the activations with their own scale and accumulates in 32-bit integers.

Trade-offs and edge cases

Each fix has a price, and naming it is what an interviewer listens for. Vector add is limited by memory traffic. Softmax is fixed by fusing it into one kernel. Tiling works because matrix multiply reloads the same values, so it is limited by data reuse.

Text description of the diagram

• Vector add is limited by memory traffic.
• Softmax is fixed by fusing it into one kernel.

Edge cases to say out loud:

• Fusion is not general. The Triton tutorial's fused softmax targets matrices whose rows fit in SRAM; longer rows need another design.
• Vectorizing needs alignment. A misaligned cast is a bug, and lengths that are not a multiple of the vector width need a scalar tail.
• Tiling spends shared memory. Bigger tiles reuse more data but leave less shared memory per block, so pick the tile with that in mind. Tiles can also cause shared-memory bank conflicts, where several threads of a warp hit the same memory bank and are served one after another; padding the tile is the usual remedy.
• Tensor Cores. Prompts sometimes ask to optimize for memory throughput and Tensor Core use together. As the CUDA C++ Programming Guide describes in its warp matrix functions section, a warp computes a small matrix tile with dedicated matrix-multiply-accumulate instructions. The inputs are 16-bit floating point (FP16), brain floating point 16 (BF16) or TensorFloat-32 (TF32), with accumulation in 32-bit floating point (FP32). The tile shapes are fixed, for example 16 by 16 by 16. So the tiled matrix multiply's shared-memory tiles must be built to feed those shapes, and the memory fixes above still come first.
• Attention. A FlashAttention-style kernel combines a matrix multiply and a softmax in one kernel; the ideas here are the foundation, and the multi-head attention page shows the layer it speeds up.

How to answer it in an interview

The spoken structure: name the bottleneck, fix the access pattern, widen the loads, reuse the data, then state the cost of each.

Common mistakes

• "Vectorized loads are faster because the memory is faster." (wrong) The hardware moves the same bytes. (right) "Vectorized loads cut the instruction count and improve bandwidth use, and they need aligned addresses."
• "A 128-bit load fixes my strided access." (wrong) Coalescing depends on how the warp's addresses line up. (right) "I fix the index mapping first, then widen the loads."
• "Fusion makes softmax faster because the math is cheaper." (wrong) The math is the same. (right) "Fusion removes global memory reads and writes."
• "Fused softmax works for any matrix." (wrong) The tutorial's version is meant for rows that fit in SRAM. (right) "I would check row length before choosing it."

Follow-up questions

Why are vectorized loads better than non-vectorized loads?

One wide instruction does the work of several narrow ones, so fewer instructions are fetched and executed and bandwidth is used better, as NVIDIA's blog explains. The bytes moved stay the same, and addresses must be aligned.

How do you implement a quantized linear layer with INT8 weights and per-channel scales?

Store each weight row as INT8 integers with its own scale, multiply the float input by those weights, then by that row's scale, then add the bias; a full INT8 kernel also quantizes the activations and accumulates in 32-bit integers. In the toy example this gave 2.7299 against a float result of 2.725.¹

How do you write and optimize a CUDA kernel for reduction, softmax or matrix operations?

Write the simple version, name its limit, then apply the steps above in order: coalesce, vectorize, reuse. For a reduction, use a tree, which needs ten rounds for 1,024 values instead of 1,023 dependent additions.¹

How do you optimize CUDA GEMM with tiling and coalescing?

In general matrix multiply (GEMM), load square tiles of both inputs into shared memory with coalesced reads, then reuse each tile across the block. In the example, tiles of thirty-two cut global loads from 2,147,483,648 to 67,108,864.¹ The code review page shows how to narrate such trade-offs.

How do you implement and optimize a dense vector dot product?

Multiply elementwise with coalesced loads, then sum with a tree reduction instead of a sequential loop. The multiply is memory-bound, so the loads matter more than the arithmetic. Related ground: 2D convolution from scratch, building generative pre-trained transformer (GPT) from scratch and fully sharded data parallel.

Sources

1. CUDA C++ Best Practices Guide, section Coalesced Access to Global Memory (read 2026-10-26) — NVIDIA, 26 Oct 2026
2. Fused Softmax, Triton tutorial 02 (read 2026-10-26) — Triton documentation, 26 Oct 2026
3. CUDA Pro Tip: Increase Performance with Vectorized Memory Access (read 2026-10-26) — NVIDIA Technical Blog, 4 Aug 2025
4. CUDA C++ Programming Guide, sections on warps and Warp Matrix Functions — NVIDIA, 26 Oct 2026