FP16 vs BF16

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

The wrong format costs a run that diverges or silently stalls. The question arrives as "FP16 or BF16?", as "your loss is NaN (not a number, the result of invalid arithmetic such as zero divided by zero) after a few epochs, why?" or as "why does mixed precision not ruin accuracy?". All three test the same idea: a floating-point format is a budget of bits split between range and precision, and the split decides which failures you meet.

A weak answer says "BF16 is newer and better". A strong answer names the exponent and mantissa bits, says which failure each format produces, explains loss scaling and the 32-bit copy of the weights that training keeps (master weights), and knows which operations stay in 32 bits. Interviewers use this topic to separate people who have debugged a diverging run from people who have only turned on a flag.

What each one is

FP16

FP16 is the half-precision floating-point format of the Institute of Electrical and Electronics Engineers (IEEE) 754 standard, a 16-bit number with 1 sign bit, 5 exponent bits and 10 mantissa bits, that stores values from about 6.1e-5 up to 65,504 with roughly three decimal digits of precision.

BF16

BF16, or bfloat16, is a 16-bit floating-point format with 1 sign bit, 8 exponent bits and 7 mantissa bits, designed to match the range of FP32 at the cost of precision.

Loss scaling

Loss scaling is the technique of multiplying the loss by a constant before backpropagation, so that small gradients stay representable in FP16, and dividing the gradients by the same constant before the weight update.

Mixed precision

Mixed precision is a training method that runs most arithmetic in a 16-bit format while keeping a few sensitive steps and a master copy of the weights in FP32.

How a floating-point number is stored

A floating-point number is stored like scientific notation: a sign, an exponent that sets the size (the power of two) and a mantissa that sets the digits. The exponent decides how big or small a number can be, its range. The mantissa decides how finely numbers near each other are told apart, its precision. With a fixed 16 bits, every bit given to one is taken from the other.

FP32, the usual training format, has 8 exponent bits and 23 mantissa bits, which is why it is both wide in range and fine in precision, and why it costs twice the memory of either 16-bit format. The IEEE 754 standard is the specification of floating-point arithmetic that defines the FP16 and FP32 layouts. The diagram shows the three layouts; BF16 is literally the top 16 bits of an FP32 number, which makes converting between them cheap.

Text description of the diagram

• FP32 has 1 sign bit, 8 exponent bits and 23 mantissa bits.
• BF16 has 1 sign bit, 8 exponent bits and 7 mantissa bits, the top 16 bits of an FP32 number.
• FP16 has 1 sign bit, 5 exponent bits and 10 mantissa bits.

FP16 in plain words

FP16 is a ruler with fine tick marks that is only 65,504 units long. Values near one are measured to about three decimal digits, but anything above 65,504 overflows to infinity, and anything below about 6.1e-5, the smallest normal number, becomes subnormal. A subnormal number is a value below the smallest normal number, stored with fewer significant bits so that it can still be told apart from zero; in FP16 those bits shrink until, below about 3e-8, the value rounds to zero.²

The analogy breaks here: floating-point gaps grow with the size of the number, so a value near 60,000 is stored far more coarsely than one near one. NVIDIA's Train With Mixed Precision guide (read Oct 2026) gives the same FP16 limits. Counting in powers of two, which is what dynamic range (the span from a format's smallest to its largest value) measures: FP16 spans 40 with subnormals and 30 for normal numbers, FP32 spans 254 for normal numbers and 277 with subnormals, and BF16's normal numbers span the same 254 as FP32's.²

BF16 in plain words

BF16 is a very long ruler with coarse ticks. It copies FP32's 8 exponent bits, so its largest value is about 3.4e38 and its smallest normal number about 1.2e-38, and it keeps only 7 mantissa bits, about two to three decimal digits. Google Cloud's bfloat16 post, Aug 2019 says the team found neural networks "far more sensitive to the size of the exponent than that of the mantissa", and that a multiplier's physical size scales with the square of the mantissa width. Fewer mantissa bits also mean smaller, cheaper multipliers. The price is that two numbers closer together than about one part in 128 look identical, which is why BF16 sums need an FP32 accumulator.

Worked example

Run the same values through both formats. Every row below is computed by a script in torch, not typed by hand.²

Four demonstrations show what the table means.

First, the number 70,000 overflows to infinity in FP16 and is stored as 70,144 in BF16. Second, a gradient of 1e-8 becomes exactly zero in FP16 and stays about 1.0e-8 in BF16. Third, 1.001 is stored as 1.0009765625 in FP16 and rounds to 1.0 in BF16, so BF16 cannot see a 0.1% change. Fourth, summing 0.1 ten thousand times should give 1,000: FP32 gives 999.90, FP16 stalls at 256 and BF16 at 32, because the increment falls below half the gap between neighbouring numbers. Once the running total reaches 256 in FP16 or 32 in BF16, adding 0.1 changes nothing, so a sum kept in 16 bits stalls and should be accumulated in FP32.²

Text description of the diagram

• Adding 0.1 ten thousand times gives 999.90 in FP32, 256 in FP16 and 32 in BF16 against a true 1,000.
• FP16 stalls at 256 and BF16 stalls at 32, so sums should be accumulated in FP32.

Now the gradients, and a warning: this part is illustrative. Take 100,000 synthetic gradient magnitudes spread evenly on a log scale from 1e-10 to 1e-2. Cast to FP16, 31.2% become zero; cast to BF16, none do. That percentage is set by the range we chose, not by FP16: moving only the lower bound to 1e-8, 1e-9, 1e-10 or 1e-11 gives 8.0%, 21.2%, 31.2% or 38.9%.²

Multiplying by a loss scale of 1,024 first brings the FP16 zero fraction in this sample down to 0%, although the smallest survivors are subnormal numbers with few bits left. To lift a 1e-8 gradient to FP16's smallest normal value takes a scale of about 6,104; real loss scales are powers of two, so the next one up is 8,192.²

The measured cost comes from a real network. Micikevicius et al. (2018) trained the Single Shot MultiBox Detector (SSD), a network that finds and labels objects in images, with FP16 mixed precision: without loss scaling it failed to train, and with scaling it matched FP32, 77.1% mean average precision (a score for how well detected objects match the true ones) against 76.9% (their section 4.2, Table 2).¹ The arithmetic is in their section 3.2: a scale of 8 is 2 to the power 3, so it raises every gradient's exponent by 3.

A gradient at 2 to the minus 27 (about 7.5e-9) is below FP16's smallest subnormal and would be lost; multiplied by 8 it becomes 2 to the minus 24 (about 6e-8), exactly that smallest subnormal. The paper found that values in the 2-to-the-minus-27 to minus-24 range mattered for this model, which is why a scale of 8 was enough. Separately, about 5% of the weight gradients they measured had exponents below minus 24 (section 3.1), which is why they also keep FP32 master weights.¹

The reading: FP16 needs loss scaling because range is its weak side, and BF16 needs care in summation because precision is its weak side.

Trade-offs and edge cases

The recipe that makes both work is mixed precision, introduced for FP16 by Micikevicius et al. (2018): keep a master copy of the weights in FP32 that accumulates updates, run forward and backward in 16 bits, and use loss scaling for FP16. The paper reports tests on convolutional and recurrent networks and generative adversarial networks, with nearly 2x memory savings. NVIDIA, Train With Mixed Precision, read Oct 2026 describes dynamic scaling: start with a large scale, skip the update and reduce the scale if gradients contain infinity or NaN, and raise it after a run of clean iterations (its FAQ cites 2,000).

Kalamkar et al. (2019) report that BF16 training matched FP32 results across image, speech, language, generative and recommendation models with unchanged hyperparameters, avoiding the tuning FP16 often needs. PyTorch, torch.amp page, Oct 2026, the documentation for automatic mixed precision (AMP), describes two tools. Autocast is a context that picks the 16-bit or 32-bit format for each operation, and GradScaler is the helper that applies loss scaling to avoid float16 underflow; its central processing unit (CPU) bfloat16 training example uses autocast alone.

Keep sensitive operations in FP32. PyTorch autocast runs matrix multiplications and convolutions in float16 but runs exp, log, sum, softmax, layer norm and cross entropy in float32. NVIDIA's guide, Oct 2026 likewise keeps large reductions such as softmax in FP32, where a reduction is a sum or other operation that combines many values into one. It also lets Tensor Cores accumulate FP16 products into FP32; a Tensor Core is a unit inside an NVIDIA graphics processing unit (GPU) that multiplies small blocks of numbers very fast, usually in 16-bit or smaller formats.

Text description of the diagram

• Use BF16 with autocast where the hardware supports it, otherwise FP16 with loss scaling.
• Keep sums, softmax and normalization in FP32 under either format.
• If the loss is still NaN, check the data, the learning rate and the first layer that outputs NaN.

The flow reads top to bottom. If the hardware supports BF16, use it with autocast, which usually needs no loss scaling; if not, use FP16 with loss scaling and FP32 master weights. Both branches keep sums, softmax and normalization in FP32. If the loss is still NaN after that, check the data, the learning rate and the first layer whose output is NaN, rather than changing formats again.

Hardware decides. NVIDIA's H100 page, Oct 2026 lists BF16 and FP16 Tensor Core throughput at the same figure on the H100 SXM board (1,979 teraFLOPS, a measure of trillions of operations per second). The figure is quoted with sparsity, and sparsity is a technique that skips zero-valued weights, so on that GPU the choice is about numerics, not speed. Older hardware may support only FP16 Tensor Cores; check yours, and see GPU fundamentals.

BF16 has a cost. Its two to three digits can swallow small updates, which is why master weights and FP32 accumulation stay.

Below 16 bits: 8-bit floating point (FP8). The same range-versus-precision trade returns at 8 bits. The H100 page lists FP8 at 3,958 teraFLOPS, twice the BF16 figure, as of October 2026. NVIDIA Technical Blog, Floating-Point 8, Jun 2025 describes two formats: E4M3 (4 exponent bits, 3 mantissa bits, maximum about 448), which favours precision and is used for the forward pass, and E5M2 (5 exponent bits, 2 mantissa bits, maximum 57,344), which trades mantissa for range and is used for the backward pass.

With so few bits, FP8 needs scaling factors, either one per tensor or one per small block of values, so that each tensor fits the format's range. The post describes mixed precision as keeping FP32 "where needed", and links one team that kept BF16 for sensitive layers to stabilise FP8 training. Our recommendation, not a quoted rule, is to carry the 16-bit recipe down: higher-precision accumulation and master weights with FP8 tensors. The formats themselves are defined in Micikevicius et al., FP8 Formats for Deep Learning, Sep 2022. Quantization, storing numbers in fewer bits than they were trained in, follows the same logic.

How to answer it in an interview

The spoken structure is: bit split, the failure each causes, the fix, then the pick.

Common mistakes

"BF16 is more precise than FP16." (wrong) It has fewer mantissa bits, 7 against 10. (right) "BF16 has more range and less precision."

"FP16 cannot train a model." (wrong) Mixed precision with loss scaling and FP32 master weights trains FP16 models, as the 2018 paper shows. (right) "FP16 needs loss scaling and care."

"BF16 always needs loss scaling too." (wrong) Its FP32-sized range makes scaling normally unnecessary. (right) "BF16 usually trains with autocast alone."

"Cast everything to 16 bits." (wrong) Sums, softmax and normalization lose accuracy. (right) "Keep reductions and the loss in FP32."

Follow-up questions

Your loss is NaN after a few epochs. What can be the reason, and how would you debug it?

Common causes are overflow to infinity in FP16 (for example in the attention logits, which are the raw scores that attention feeds to softmax), log of zero, division by zero, a learning rate that is too high and bad data. Find the first layer whose output is NaN, check gradient norms, and watch whether the loss scale is collapsing; see activation functions and cross-entropy loss for the stable forms.

Why does mixed-precision training work without destroying accuracy?

The FP32 master weights collect small updates that a 16-bit weight would swallow, loss scaling keeps small gradients alive, and sensitive operations stay in FP32.

What are the trade-offs between FP16, INT8 and FP4 quantization?

Quantization stores weights in fewer bits: FP16 takes 2 bytes per weight, 8-bit integer (INT8) takes 1 and 4-bit floating point (FP4) takes half a byte, a 2x and 4x memory cut. The cost is more error, so each step down needs calibration (measuring typical value ranges to choose scales) or retraining, and the latency gain depends on hardware support. For how this plays out when serving a large language model (LLM), see LLM inference.

Can you train in BF16 without master weights?

You can, but updates smaller than roughly 0.2% to 0.4% of a weight's size round away in BF16, since its epsilon is 0.0078 and an update under half a gap is lost. Many setups keep FP32 master weights or use stochastic rounding, which is rounding up or down at random with probability set by the remainder, so small updates survive on average.

When would you train in FP8?

Train in FP8 when memory or throughput limits you and the hardware supports it, as on the H100 page's FP8 line, which lists twice the BF16 throughput. Use E4M3 for the forward pass and E5M2 for gradients, add scaling factors, and keep master weights and accumulation in higher precision as the safe default; fall back to BF16 for layers that become unstable.

What if the loss scale keeps shrinking?

Repeated overflow means the gradients are genuinely huge, usually from a bad learning rate or data, or that the forward pass itself overflows FP16 (activations above 65,504), which no loss scale can fix. Run the forward pass in FP32 to find the operation that overflows, and fix that cause rather than lowering the minimum loss scale, the value the scale is not allowed to fall below.

Sources

1. Mixed Precision Training — arXiv (Micikevicius et al.), v3, 15 Feb 2018
2. A Study of BFLOAT16 for Deep Learning Training — arXiv (Kalamkar et al.), v3, 13 Jun 2019
3. BFloat16: The secret to high performance on Cloud TPUs — Google Cloud, 24 Aug 2019
4. Train With Mixed Precision — NVIDIA Deep Learning Performance documentation, 24 Oct 2026
5. Automatic Mixed Precision package (torch.amp) — PyTorch documentation, 24 Oct 2026
6. NVIDIA H100 Tensor Core GPU — NVIDIA, 24 Oct 2026
7. FP8 Formats for Deep Learning — arXiv (Micikevicius et al.), v2, 29 Sep 2022
8. Floating-Point 8: An Introduction to Efficient, Lower-Precision AI Training — NVIDIA Technical Blog, 4 Jun 2025