What are exploding gradients, and how do you fix them?

In this article

1. Before you start
2. Why interviewers ask this
3. Exploding gradients from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step on the deep learning path.

Why interviewers ask this

Exploding gradients are a common test of whether a candidate understands what backpropagation really does, because the failure follows directly from the arithmetic. Questions come in several forms: why does a deep or recurrent network blow up, how do you detect it, what does clipping do, and which fix would you reach for first. A weak answer says "the gradients get too big" and stops. A strong answer names the repeated multiplication, says that depth or sequence length is the exponent, and separates the quick safety net (clipping) from the causes (initialisation, normalisation, architecture). The order to practise is: the multiplication, the symptom, the cap, the cause.

Exploding gradients from first principles

Gradient norm

The gradient norm is a single number, the length of the gradient treated as one long list, that says how large a step plain gradient descent is about to take.

Gradient clipping

Gradient clipping is a safeguard that rescales the gradient whenever its norm is above a chosen threshold, so one bad batch cannot throw the weights far away.

Weight initialisation

Weight initialisation is the rule that sets the random starting values of a network's weights, and it fixes how much a signal grows or shrinks per layer before any training.

In plain words

Imagine a message passed down a line of one hundred people. Each person repeats it a little louder than they heard it. If each is ten percent louder, the last person shouts. If each is ten percent quieter, the last person whispers. The analogy breaks in one place. In a network the message travels in reverse during training, from the loss back to the first layers. What gets amplified is the correction signal, not the data.

An exploding gradient is a gradient whose size grows exponentially with depth. In one precise sentence: during backpropagation the gradient reaching an early layer is a product. It has one factor per layer above it. If those factors are typically above one, the product explodes. Pascanu, Mikolov and Bengio (2013) analyse this for recurrent networks and propose clipping the gradient norm for the exploding case.

Step 1: A deep network is a chain of layers

A neural network computes its output by passing numbers through layers in a chain, and the loss depends on the first layer only through every layer after it. Each layer takes the previous layer's output, multiplies by its weights and applies an activation function. Because the layers are chained, a small change in an early weight changes every later output.

A recurrent network is the same chain with a twist: the same layer is reused at every time step, so a sequence of 100 steps behaves like a network 100 layers deep. That is why recurrent models meet this failure first and most often. Long short-term memory (LSTM) networks add a gated, additive memory path that mainly targets vanishing gradients; LSTMs are still clipped against exploding ones.

1 · A chain of layers2 · One factor per layer3 · Factors compound4 · Spot the spike5 · Cap and fixReplay

Step-by-step description

1. The output depends on the input through every layer.
2. Going backward, each layer multiplies the gradient.
3. A factor above one grows with depth like an exponent.
4. The gradient norm jumps and the loss turns to NaN.
5. Clip the norm now, and repair the factors for good.

• Backpropagation multiplies one factor per layer, so depth becomes the exponent.
• A factor of 1.1 over 100 layers gives about 13,780, and a factor of 0.9 gives about 0.00003.
• Gradient clipping rescales the whole gradient so its norm never exceeds a threshold, and a weight initialisation that starts the factors near one removes the cause.

Step 2: Backpropagation multiplies one factor per layer

Going backward, each layer multiplies the incoming gradient by a factor that depends on its weights and on its activation function's slope. This is the chain rule from calculus: the effect of an early weight on the loss is the product of the effects of every layer in between. The result is a long product of numbers, one per layer.

The simplest case is a single repeated weight $w$ over $T$ steps, where the gradient factor is $w^T$. The exponent is the depth or the sequence length, which is what makes the product so sensitive. Real layers have many weights, so the factor is a matrix, and what matters is how much it stretches or shrinks the gradient on average; the single-weight case keeps the arithmetic visible.

Step 3: Factors above one compound exponentially

A factor above one grows with depth like an exponent, and a factor below one shrinks the same way. With $T=100$, a factor of 1.1 gives about 13,780, a factor of 1.5 gives about 4 × 10¹⁷, and a factor of 0.9 gives about 0.00003.¹ Ten percent per step is nothing, yet a hundred steps turn it into a five-digit multiplier.

The same arithmetic produces the opposite failure when the factor is below one: the early layers receive almost no signal and stop learning. Glorot and Bengio (2010) study how activations and gradients vary across layers, and suggest training is harder when the layer Jacobians' singular values are far from one. The same mechanism, two directions.

Step 4: Spot the spike before the loss turns to NaN

You detect an exploding gradient by logging the gradient norm every step: it jumps by orders of magnitude, and a step or two later the loss becomes NaN or infinity. The warning signs come in order. The global gradient norm spikes, one update throws the weights far from where they were, the loss shoots up, and the next forward pass overflows.

In our scalar recurrent run, the very first gradient is about 14,700 against a smallest value of about 5 × 10⁻¹³ once the weight has converged, and without a cap the weight overflows within three steps.¹ In PyTorch the function clip_grad_norm_ returns the total gradient norm as a tensor, so logging its return value gives you the monitor for free (PyTorch 2.14 documentation). Plot that number, not only the loss.

Step 5: Cap the update with clipping, then fix the cause

Gradient clipping rescales the whole gradient so its norm never exceeds a threshold, which caps the size of one update without changing its direction. The rule in Pascanu et al. is: if the norm is at or above the threshold, scale the gradient down to the threshold. The authors note that it is simple and cheap, adds one hyper-parameter, the threshold, and that training is not very sensitive to it; they suggest looking at the average norm over many updates to pick one.

Clipping treats the symptom. The causes are removed elsewhere, as the table in the next section shows: a weight initialisation that starts the factors near one, normalisation layers, and residual connections paired with normalisation or a scaled initialisation. A lower learning rate does not change the factors; like clipping, it only limits the damage of each step. In our run, at the same learning rate, the task that overflowed without a cap converged when the norm was clipped to 5, and clipping fired on only the first 76 of 300 steps.¹

Worked example

A fifty-layer network shows how sensitive the product is to the weight scale. We built a 50-layer ReLU network of width 256 with random weights, sent a unit-length gradient back from the output, and measured its length at layer 1. Weights with standard deviation $\sqrt{2/n}$, where $n$ is the layer width, are the rule He et al. (2015) derive for rectifier networks; we then scaled that by 0.7 and by 1.4.¹

Text description of the diagram

• The standard scale keeps the gradient steady over 50 layers.
• A weight scale 40% too large makes the layer-1 gradient about 15 million times the output gradient.

The standard scale is stable because the factor per layer is close to one, so fifty of them multiply to about 0.81. A 40% change in scale is a factor of 1.4 per layer in length (1.96 in variance): 1.4 to the 50th is about 2 × 10⁷ (7.3 orders against 7.17 measured), and 0.7 to the 50th is about 2 × 10⁻⁸ (against −7.99 measured).¹ The numbers are for one random draw (seed 0) and are illustrative of the effect, not a benchmark.

The clipping step itself is one line:

total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# total_norm is the norm measured BEFORE clipping; log it every step

The scalar recurrent run makes the benefit concrete. Training $w$ so that $w^{30}$ reaches 2 from a start of 1.1 gives a first gradient near 14,700. At the same learning rate of 0.0002 in both arms, with no clip the weight overflows by step 3; with the norm clipped to 5 it converges to 1.02337, the target, with a loss near 10⁻²⁹ after 300 steps, and clipping fires only on the first 76 steps.¹

Trade-offs and edge cases

Every remedy for exploding gradients trades something, so the choice depends on where the instability comes from. The table summarises the four common ones.

Text description of the diagram

• Gradient clipping rescales the whole gradient so its norm never exceeds a threshold.
• A weight initialisation that starts the factors near one removes the cause.
• Clipping treats the symptom; residual skips stop vanishing but are stable only with normalisation or a scaled-down branch.

Clipping hides a cause that is still there. If the norm is clipped on most steps, the threshold is acting as a learning-rate cap and the model is telling you something else is wrong. Log how often clipping fires.

Normalisation and skips change the factors. Batch normalisation reports allowing much higher learning rates and less sensitivity to initialisation (Ioffe and Szegedy, 2015), and residual connections let networks of 152 layers and more train (He et al., 2015), with batch normalisation inside each block. A deep residual stack without normalisation can still explode at standard initialisation: in a 50-block run with He initialisation and no normalisation the gradient grew by about 10¹¹·⁷, and scaling each branch by 1/sqrt(50) brought it to about 10⁰·⁴.¹

That is the motivation for Fixup initialisation (Zhang, Dauphin and Ma, January 2019), which rescales the initial weights so residual networks train without normalisation. The identity path stops vanishing; stability needs normalisation or a scaled-down branch. Normalisation does not remove the problem in every setting: in a deep network without skips, batch normalisation can itself make gradients explode at initialisation (Yang et al., February 2019). See batch versus layer normalisation for which one a given model uses.

Clipping by norm versus by value. Pascanu et al. mention an earlier element-wise variant that clips each entry past a threshold; clipping the whole norm preserves the direction of the update, which clipping entries does not.

Reported practice differs on the threshold. Pascanu et al. pick the threshold from the average gradient norm over many updates. Recent large language model recipes fix it at 1.0: Brown et al. (May 2020) clip the global norm at 1.0 when training the third generative pre-trained transformer model (appendix B), Touvron et al. (July 2023) train Llama 2 with AdamW and gradient clipping of 1.0 (section 2.2), and DeepSeek-AI (December 2024) set the gradient clipping norm to 1.0 for DeepSeek-V3 (section 4.2). Treat the threshold as a hyper-parameter and log how often it fires.

How to answer it in an interview

The spoken structure is: the repeated multiplication, the symptom you would see, the clip, then the causes.

Common mistakes

"Exploding gradients only happen in recurrent networks." (wrong) Any deep network can multiply factors above one; recurrent ones are the worst case because the same weights repeat. (right) "They happen wherever many factors multiply, and recurrent networks are especially exposed."

"Clipping fixes the problem." (wrong) It caps one update and leaves the factors above one; if it fires on most steps something else is wrong. (right) "Clipping is a safety net; initialisation and architecture are the cure."

"Clipping changes the direction of the gradient." (wrong) Clipping by norm rescales the whole vector, so the direction is the same and only the length shrinks. (right) "Norm clipping keeps the direction and caps the size."

"A lower learning rate is the same as clipping." (wrong) A lower rate scales every update, good or bad, while clipping only touches the oversized ones. (right) "Clipping is conditional; the learning rate is not."

Follow-up questions

Does the vanishing gradient problem appear closer to the beginning or end of a neural network?

The beginning. The gradient is multiplied by one factor per layer on its way back, so the early layers receive the product of the most factors; if those are below one, the early layers get almost no signal and barely learn, while layers near the output train normally.

How do RNN vanishing gradients arise and how do LSTMs solve them?

A recurrent neural network (RNN) applies the same weights once per time step, so the gradient from a distant step is a long product of factors, and below one it vanishes. An LSTM adds a gated, additive cell-state path, so the gradient can flow along it multiplied only by the forget gate, which the network can learn to keep near one. That targets vanishing; exploding gradients remain possible, so LSTMs are still clipped.

How do you detect exploding gradients?

Log the global gradient norm each step and watch for jumps of several orders of magnitude, followed by an infinite or NaN loss. The return value of clip_grad_norm_ is that norm, measured before clipping.

What threshold should gradient clipping use?

Pascanu et al. suggest using statistics of the average gradient norm over many updates, and report that training behaves well even for rather small thresholds. Start from the typical norm of a healthy run, not a magic number.

Does Adam fix exploding gradients?

Not fully. Kingma and Ba (section 2.1) show that Adam's effective step is bounded by about the learning rate, and by at most about 3.2 times it at the default settings, so one spike cannot throw the weights far; the damage is that the spike inflates the running second-moment estimate and stalls later steps, and an inf or NaN gradient poisons the moments for good. The factors above one are also still there, which is why Adam runs still clip the global norm. See optimizers and gradient descent, and the cross-entropy loss page for a loss that can itself produce NaN.

What if the sequence is 1,000 steps long instead of 100?

The exponent grows tenfold, so the same 10% factor gives 1.1 to the 1000th, about 10⁴¹, instead of about 13,780. That is beyond the largest float32 number (about 3.4 × 10³⁸), so the product overflows to infinity; with the 10% factor, float32 overflows after about 931 steps and float16 (largest 65,504) after about 117.¹ Clipping acts after the backward pass, so it cannot rescue a gradient that has already overflowed; long sequences need the factors kept near one, with a gated architecture or a shorter backward window, and clipping as the safety net.

Sources

1. On the difficulty of training Recurrent Neural Networks — arXiv (Pascanu, Mikolov, Bengio), v2; ICML 2013, 16 Feb 2013
2. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — arXiv (He, Zhang, Ren, Sun), 6 Feb 2015
3. Understanding the difficulty of training deep feedforward neural networks — AISTATS 2010, PMLR vol. 9 (Glorot, Bengio), 31 Mar 2010
4. Deep Residual Learning for Image Recognition — arXiv (He, Zhang, Ren, Sun), 10 Dec 2015
5. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — arXiv (Ioffe, Szegedy), v3, 2 Mar 2015
6. Fixup Initialization: Residual Learning Without Normalization — arXiv (Zhang, Dauphin, Ma), v1, 27 Jan 2019
7. Language Models are Few-Shot Learners (GPT-3), appendix B — arXiv (Brown et al.), v1, 28 May 2020
8. Llama 2 technical report, section 2.2 — arXiv (Touvron et al.), 18 Jul 2023
9. A Mean Field Theory of Batch Normalization — arXiv (Yang, Pennington, Rao, Sohl-Dickstein, Schoenholz), v1 (revised v2, 2019-03-05), 21 Feb 2019
10. DeepSeek-V3 Technical Report, section 4.2 — arXiv (DeepSeek-AI), v1, 27 Dec 2024
11. Adam: A Method for Stochastic Optimization, section 2.1 — arXiv (Kingma, Ba), v9; ICLR 2015, 30 Jan 2017
12. torch.nn.utils.clip_grad_norm_ — PyTorch 2.14 documentation (date = read date), 24 Oct 2026