Deep Learning Fundamentals

Why does softmax need numerical stabilization?

By Interview Clozo · Published 27 October 2026 · Last verified 26 October 2026 · 10 min read

Why does softmax need numerical stabilization? Because it exponentiates every score, and floating-point numbers overflow to infinity or underflow to zero long before the math is in trouble. Subtracting the largest score first fixes both and leaves every probability unchanged. The error that costs the point is thinking only huge scores break it: very negative scores underflow to zero and give NaN too, as the first step shows.

In this article

Before you start
Why interviewers ask this
Softmax stabilization from first principles
The Staff-level answer
The rubric
Three variants of this question
Common mistakes
Follow-up questions
Sources

Before you start

This page belongs to the deep learning path and uses ideas from probability and statistics.

Why interviewers ask this

A model whose loss turns into garbage after a few epochs can waste a training run, and a naive softmax is a known cause. The DeepStability study (2022) reports that weights, biases and loss all became NaN after a couple of training epochs with a numerically unstable softmax in the last layer. NaN (not a number) is the floating-point value that marks an undefined result, such as infinity divided by infinity.

The question tests whether a candidate sees that correct math can still fail on a real machine. A weak answer says "to avoid big numbers" and stops. A strong answer names overflow and underflow, shows why subtracting the max leaves the result unchanged, and says what goes wrong with the log. A Staff-level answer adds that the fix is stable but not exact, and that the loss should be computed from logits.

Softmax stabilization from first principles

Numerical stabilization: Numerical stabilization is a rewrite of a formula that gives the same answer in exact arithmetic but keeps every intermediate value inside the range a floating-point format can hold, so the computation does not produce infinity or NaN.
Underflow: Underflow is the error that occurs when a result is closer to zero than the smallest number a floating-point format can store, so the computer rounds it to exactly zero.

In plain words

Picture a scoreboard with a counter that stops at a fixed maximum. If a score is too big for the counter, it shows "error". Softmax turns scores into shares of a total. You can write each score as its gap behind the leader, so the numbers stay small.

The analogy breaks in one place. Ordinary shares change when you subtract a constant from every score. Softmax shares do not, because exp turns subtraction into a common divisor that cancels.

In one precise sentence: softmax computes exp of each score divided by the sum of those exps, and stabilization rewrites it so no intermediate value leaves the range the number format can hold.

softmax (x)_{i} = \frac{e^{x_{i}}}{\sum_{j} e^{x_{j}}}

A logit is the raw output score of a network before softmax is applied.

Step 1: Naive softmax overflows and underflows

Naive softmax fails because exp of a large score exceeds the largest number the format holds, and exp of a very negative score rounds to zero. In 32-bit floating point (FP32) the largest finite value is about 3.4028e38, so exp(88) fits and exp(89) becomes infinity.¹ The lecture slides of Goodfellow (2017) give the same threshold. For scores [10, 100, 1000], exp(10) is 22,026.5, while exp(100) and exp(1000) overflow, so the sum is infinity and the outputs are 0, NaN, NaN.¹

The low end fails too. For scores [-1000, -10000, -1000000] every exp rounds to zero, the sum is zero, and the division is zero over zero, which is NaN.¹ Equal scores [1000, 1000, 1000] should give one third each, but the naive version returns NaN (Goodfellow et al., 2016, constant-input case).¹

Input: scores. Output: NaN, though the true answer is fine. You now know the failure happens at both ends of the range.

Why half precision breaks sooner

The 16-bit half-precision format (FP16) overflows exp at a much smaller score than FP32 does, while brain floating point 16 (BF16) keeps the FP32 range.

Half precision overflows exp at an argument near 11, so softmax breaks there on scores that FP32 handles.CC BY 4.0 — source: Why does softmax need numerical stabilization?

Text description of the diagram

The largest exp argument that fits is 11.09 in FP16.
The largest exp argument that fits is 88.72 in BF16 and 88.72 in FP32.

The largest exp argument that fits is 11.09 in FP16, 88.72 in BF16 and 88.72 in FP32.¹ A score of 12 is therefore harmless in FP32 and fatal in FP16.

Step 2: Subtract the max, and the result does not change

Softmax is shift-invariant, meaning adding the same constant to every score leaves the output unchanged. Choosing the constant as minus the largest score makes every exponent at most zero:

\frac{e^{x_{i} - c}}{\sum_{j} e^{x_{j} - c}} = \frac{e^{x_{i}} e^{- c}}{e^{- c} \sum_{j} e^{x_{j}}} = \frac{e^{x_{i}}}{\sum_{j} e^{x_{j}}}

With c set to the maximum, every numerator is at most one, and the largest term is exactly one. The denominator then lies between one and the number of classes, so overflow cannot happen (Zhang et al., section 4.5). A practical softmax relies on this shift (2025 error analysis).

Check on scores [2, 1, 0] (illustrative): the shifted scores are [0, -1, -2], the denominator is 1.5032, and the outputs are 0.6652, 0.2447 and 0.0900, identical to the unshifted result.¹ The failing cases now work: [10, 100, 1000] gives [0, 0, 1], [1000, 1000, 1000] gives one third each, and the very negative scores give [1, 0, 0].¹

Input: scores minus their max. Output: valid probabilities. You now know the probabilities are safe; the logarithm is not yet.

Step 3: Compute the log from the logits, not from the probabilities

Cross-entropy needs the log of the softmax, and taking a log of a stabilized softmax output can still hit log(0), because a tiny probability may round to zero. Zhang et al. note that trouble when computing log of a zero; the cure is to combine the two steps. With m as the maximum score:

\log {\hat{y}}_{j} = x_{j} - m - \log \sum_{k} e^{x_{k} - m}

The sum inside the log is at least one, so the log is always finite. This is log-sum-exp, and the lecture slides advise computing the loss from the logits, using a fused cross-entropy function.

Take logits [1000, 0]. Naive log of softmax gives NaN for class one and minus infinity for class two. The stable formula gives 0 and -1000, both finite.¹

Input: logits. Output: log-probabilities that never hit log(0). That is the follow-up trap: a stable softmax is not enough if the loss takes a log of its output.

Subtract the maximum before exp, and compute any log from the logits instead of from the probabilities.CC BY 4.0 — source: Why does softmax need numerical stabilization?

Step-by-step description

Large scores overflow, very negative ones underflow.
Every exponent is at most zero.
The log is computed from the logits.

Naive softmax gives infinity or zero, then NaN.
Subtracting the max makes every exponent at most zero.
Log-sum-exp computes the log from the logits.

Naive softmax gives infinity or zero, then NaN. Subtracting the max makes every exponent at most zero. Log-sum-exp computes the log from the logits.

Knowing it is half the job. Saying it is the other half.

Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.