FP8 training: what it is and how it works

In this article

1. Before you start
2. Why interviewers ask this
3. FP8 training from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A training run that is 8-bit and still converges saves memory and arithmetic on a very large bill, and a run that is 8-bit and quietly wrong wastes weeks. That is why FP8 training appears when the interviewer wants to know whether you understand numbers inside a GPU and not only model code. The usual probes are "why can't you just cast everything to 8 bits?", "what is the scaling factor for?" and "what stays in higher precision?".

A weak answer says "FP8 is faster because it is smaller". A strong answer names the two formats and their ranges, explains that every tensor is rescaled to fit, says that sums are kept in higher precision, and states that the master copy of the weights is not FP8. It is reported in interviews at AI labs and large technology companies that work on large-model training.

FP8 training from first principles

Scaling factor

A scaling factor is a number that a tensor is multiplied by before it is rounded to FP8, chosen so that the tensor's largest value lands at the largest value the format can hold, and divided out again afterwards.

Amax

Amax, short for absolute maximum, is the largest absolute value in a tensor or in a small group of its values, and it is the number a scaling factor is computed from.

Fine-grained quantization

Fine-grained quantization is the practice of giving each small group of values, such as 128 in a row, its own scaling factor instead of giving the whole tensor one, so that a single outlier cannot ruin the rest.

Subnormal number

A subnormal number is a value smaller than a format's smallest normal number; the format stores it on a fixed, evenly spaced grid of steps, so it keeps far fewer correct digits than a normal value does.

In plain words

FP8 training is like measuring with a ruler that has far fewer markings. A ruler with sixteen marks can still measure a table if you pick the right unit. Training works the same way. Each tensor gets a unit of its own, the scaling factor. Its values then land where the ruler has marks. The multiplications run on these coarse numbers, which is cheap.

The analogy breaks in one place. A ruler never changes its marks. An FP8 format spaces its marks unevenly, with more near zero and fewer far away. In one precise sentence: FP8 training rounds the inputs of each matrix multiplication to 8-bit floats under a per-tensor or per-tile scale, sums in a wider format and updates the weights in full precision.

Step 1: Choose the 8-bit format

FP8 comes in two formats, and the choice is between precision and range. One format has 4 exponent bits and 3 mantissa bits (E4M3), and it has finer steps: its largest value is 448 and the gap between 1 and the next number is 0.125. The other has 5 exponent bits and 2 mantissa bits (E5M2), and it has a longer reach: its largest value is 57,344 and the gap after 1 is 0.25.¹

Micikevicius et al. recommend E4M3 for weights and activations and E5M2 for gradients, because gradients span a wider range of sizes (Micikevicius et al., 2022, section 3). A mixed use of both is called the hybrid format. The first step therefore decides, for each tensor, whether it needs fine steps or a long reach. What you now know: 8 bits give either a short reach with fine steps or a long reach with coarse ones, never both.

Step 2: Scale the tensor into range

A scaling factor moves a tensor into the range the 8-bit format can represent, and it is computed from the tensor's amax. NVIDIA's Transformer Engine documentation gives the rule as scaling factor equals the format's largest value divided by amax (NVIDIA Transformer Engine, read 2026-10-24). If a tensor has amax 3.2, the E4M3 scale is 448 divided by 3.2, which is 140; a value of 0.01 becomes 1.4, rounds to 1.375, and returns as 0.00982, an error of 1.79%.¹

The danger is the outlier, a value far larger than the rest: it forces a tiny scale. Take 4,096 illustrative values with typical size 1 and one outlier of 100,000. The scale is 448 divided by 100,000, which is 0.00448. E4M3's smallest nonzero value is 0.001953, so any value below about 0.22 rounds to zero, and that is about one value in six (16.24%).¹ Nearly all the rest fall below the smallest normal number, 0.015625. What you now know: one extreme value decides how every other value is stored.

Step 3: Multiply in FP8 and add up in higher precision

The matrix multiplication reads FP8 inputs but keeps its running sum, called accumulation, in a wider format. DeepSeek-V3's report runs all three matrix multiplications of a linear layer in FP8: the forward pass, the gradient with respect to the layer's input and the gradient with respect to its weights, with outputs in bfloat16 (BF16, a 16-bit format) or 32-bit floating point (FP32) (DeepSeek-AI, 2025, section 3.3.1). The report found that the Tensor Cores, the GPU units that do the matrix multiplications, keep only about 14 bits of accumulation precision on the H800 GPU.

For two random matrices with an inner dimension of 4,096, the number of products added for each output value, it measured a maximum relative error of nearly 2%. Its fix copies partial sums to full FP32 registers every 128 elements. The same report stores the activations kept for the backward pass in FP8 to save memory, except a few sensitive ones (section 3.3.3). What you now know: low-precision inputs are acceptable only when the sum is not low-precision too.

Step 4: Keep master weights and updates in higher precision

The weights that the optimizer updates stay in higher precision, because an update can be much smaller than an 8-bit gap. The 2018 paper on mixed precision introduced the idea for 16-bit training: keep a single-precision copy of the weights that accumulates the gradients after each step, and round it down for the forward pass (Micikevicius et al., 2018). FP8 training keeps the same design.

DeepSeek-V3 stores master weights, weight gradients and optimizer states in higher precision. It tracks the optimizer's two moment estimates, the running averages of the gradient and its square, in BF16 instead of FP32 with no observable loss of quality. That full-precision copy is called the master weights, and the 8-bit copies are cast from it at every step. What you now know: the copy the optimizer updates is never 8-bit.

1 · Choose format2 · Scale and cast3 · Multiply4 · UpdateReplay

Step-by-step description

1. E4M3 for fine steps, E5M2 for long reach.
2. Scale by maximum over amax, then round.
3. FP8 inputs, wider running sum.
4. Master weights stay in higher precision.

• E4M3 has a maximum of 448 and E5M2 has a maximum of 57,344.
• The scaling factor is the format maximum divided by amax.
• Sums inside the multiplication are kept in a wider format.
• Master weights are kept in higher precision.

Worked example

Two small computations show what an 8-bit format can and cannot hold, and then what one outlier does to it. Every number below comes from a short numpy script whose output is saved with the article.¹ The format limits follow Table 1 of the FP8 paper; the second table is an experiment on illustrative data, not a measurement of any real model.

The experiment takes 4,096 illustrative numbers drawn from a standard normal distribution (random seed 0) and plants one outlier of a chosen size in the first of 32 tiles of 128 values each.¹ It rounds the numbers to E4M3 twice, once with a single scale for the whole tensor and once with a separate scale for each 128-value tile, and reports the root-mean-square error on all the other values as a percentage of their size.

Two readings matter. First, the floor of about 2.6% is the cost of three mantissa bits,¹ and it does not grow until the outlier is huge. Values keep their three-bit precision only in E4M3's normal range, from 0.015625 to 448, about 4.46 orders of magnitude, so moderate outliers are absorbed. Second, at 100,000 the scale is 0.00448, so a typical value of 1 lands at 0.00448, below the smallest normal number.

In the experiment 99.95% of the other values became subnormal and sit on a coarse fixed grid, and any value below about 0.22 rounds to zero, about one in six (16.24%).¹ Most of the 12.78% error is therefore lost precision across nearly all values, not zeroing alone. Per-tile scaling confines the damage to the single tile that holds the outlier, which is why the tile error is 3.41%.

Text description of the diagram

• At an outlier of 100,000 the error is 12.78% with one scale per tensor and 3.41% with one scale per 128 values.
• Below an outlier of 10,000 both scaling methods stay near 2.6%.

Trade-offs and edge cases

FP8 training saves memory and arithmetic but adds bookkeeping, and the sensitive parts of a model stay in higher precision.

DeepSeek-V3 keeps the embedding module, the output head, the mixture-of-experts gating modules (the routers that pick which expert sub-network handles each token), the normalization operators and the attention operators (see FlashAttention) in BF16 or FP32, and runs the matrix multiplications in FP8 (DeepSeek-AI, 2025, section 3.3.1). Its report states that FP8 multiplications "theoretically" double the speed of the BF16 method and that the relative loss error against a BF16 baseline stayed below 0.25% over about one trillion tokens at two model scales. The first is a design target, not a measured speed-up, and the second holds for that recipe on that model.

Practice differs on the recipe, and no single answer is settled. One scale per tensor is a standard practice, and the FP8 paper matched 16-bit training with it for models up to 175 billion parameters (Micikevicius et al., 2022, sections 4.1 and 4.3); it is only outlier-sensitive.

NVIDIA's delayed scaling picks one scale per tensor from the amax values of previous iterations, which is fast but needs a stored history, and its current scaling computes one scale from the tensor in hand. DeepSeek-V3 (section 3.3.2) computes amax online for each 1 by 128 activation tile and each 128 by 128 weight block, and uses E4M3 for every tensor instead of the hybrid format, arguing that small groups share their exponent range. Newer hardware moves the tiles into the chip.

NVIDIA's microscaling FP8 (MXFP8) recipe gives every block of 32 adjacent values its own scale, and the scales are powers of two (NVIDIA Transformer Engine, MXFP8, read 2026-10-24). DeepSeek links its own tiles to these microscaling formats (section 3.3.2). Hardware matters too. The delayed-scaling recipe requires an Ada-generation GPU (streaming multiprocessor (SM) version 89; compute capability 8.9, the GPU hardware version number) or later (NVIDIA Transformer Engine, read 2026-10-24), while MXFP8 lists only versions 10.0 and 10.3, which are Blackwell-class GPUs.

Text description of the diagram

• Matrix multiplications run in FP8 and cached activations are stored in FP8.
• Embeddings, the output head, gating, normalization and attention stay in BF16 or FP32.
• Master weights and weight gradients stay in FP32.

Two edge cases deserve a sentence each. An outlier-heavy layer can still hurt if its tile is too wide, so tile size is a tuning knob. And FP8 for inference is a different problem: it quantizes a finished model, with no gradients and no master weights.

How to answer it in an interview

Say it in this order: the format, the scale, the wide sum, the master copy, then the outlier risk.

Common mistakes

"FP8 training stores everything in 8 bits."

(Wrong) "In FP8 training the whole model is 8-bit." The weights that the optimizer updates, the gradients it accumulates and the optimizer states stay in higher precision, and normalization, attention and embeddings often stay in BF16 or FP32. (Right) "The matrix multiplications run in FP8; the master weights and the sensitive operators do not."

"One scale for the whole tensor is enough."

(Wrong) "Scale the tensor by its maximum and you never need to look at outliers." Per-tensor scaling is standard, but in the illustrative worked example one outlier 100,000 times the typical value rounded about 16% of the other values to zero. (Right) "Per-tensor scaling is fast but outlier-sensitive; small tiles confine the damage."

"E5M2 is just a worse E4M3."

(Wrong) "E4M3 has more precision, so use it everywhere." E5M2 reaches 57,344 against 448, which is why gradients, with their wide spread, are often kept in it. (Right) "The two formats trade precision for range, and the hybrid recipe uses each where it fits."

"FP8 doubles training speed."

(Wrong) "FP8 makes training twice as fast." DeepSeek's report says the multiplications "theoretically" double the speed; scaling, casting and the higher-precision parts take some of it back. (Right) "The multiplication throughput can double; end-to-end speed-up is smaller and has to be measured."

Follow-up questions

Why does FP8 training need a scaling factor?

Because 8 bits cannot cover the range of a real tensor. The scaling factor moves the tensor so that its largest value sits at the format's maximum, 448 for E4M3, and the factor is divided out after the multiplication.

Why use E5M2 for gradients?

Gradients span many orders of magnitude, and E5M2 reaches 57,344 against 448 for E4M3 with a smallest nonzero value near 0.000015. The cost is a coarser gap of 0.25 after 1 instead of 0.125.

Which parts of the model stay in higher precision?

In DeepSeek-V3's recipe, the embedding module, the output head, the mixture-of-experts gating, the normalization operators and the attention operators, plus the master weights, weight gradients and optimizer states.

Does FP8 training make a model train twice as fast?

No source says so. The multiplication throughput can double in theory, but casting and scaling add work, and several operators stay in higher precision, so the speed-up must be measured on the real recipe.

What changes if the outlier gets ten times larger?

With a single per-tensor scale, the error grows quickly: in the worked example it went from 1.64% of values lost to zero at 10,000 to 16.24% at 100,000. With per-tile scales only the one tile that holds the outlier gets worse.

Sources

1. FP8 Formats for Deep Learning — arXiv (Micikevicius et al.), v2, 29 Sep 2022
2. DeepSeek-V3 Technical Report — arXiv (DeepSeek-AI), v2, 18 Feb 2025
3. Mixed Precision Training — arXiv (Micikevicius et al.), v3, 15 Feb 2018
4. MXFP8 — NVIDIA Transformer Engine documentation (date = read date), 24 Oct 2026
5. FP8 Delayed Scaling — NVIDIA Transformer Engine documentation (date = read date), 24 Oct 2026