Residual connections: what they are and why deep networks need them

In this article

1. Before you start
2. Why interviewers ask this
3. Residual connections from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

Residual connections are asked about because they are a single line of code that explains why modern networks can be hundreds of layers deep. A weak answer says "they help with vanishing gradients" and stops. A strong answer states the formula, separates the optimisation problem the original authors observed from the vanishing-gradient story, shows what the gradient looks like with a shortcut, and says where the shortcut sits in a transformer.

The question appears in deep learning and machine learning (ML) design rounds, often as "why does ResNet work" or "why does a transformer have residual connections". The deep learning guide lists the neighbouring questions.

Residual connections from first principles

Residual connection

A residual connection is a shortcut that adds the input of a group of layers to that group's output, so the group computes x plus F of x instead of F of x alone.

Residual block

A residual block is the group of layers F together with the shortcut that adds its input back, and it is the unit that residual networks repeat many times.

Identity mapping

An identity mapping is a function that returns its input unchanged, so a block that has learned the identity mapping outputs exactly what it was given.

In plain words

Imagine editing a document. Writing a new version from a blank page is hard. Marking corrections on the existing version is easier, because doing nothing is already a valid answer. A residual block works like the second approach: the shortcut carries the existing version, and the layers only write the corrections. The analogy breaks here: an editor changes a few words, while the layers output a correction for every number in x, and "change nothing" happens only when training pushes their weights toward zero.

Stated precisely, a residual network rewrites each block as y = x + F(x). He et al. say they "explicitly reformulate the layers as learning residual functions with reference to the layer inputs" (He et al., 2015). If the best block is the identity mapping (output equals input), the layers need only push F toward zero.

Step 1: See the problem that plain deep stacks have

The first step is the observation that made residual connections necessary. The input is a plain network of 18 layers and a deeper one of 34; the output is their error, measured as top-1 error (the share of held-out images where the model's single best guess is wrong) on ImageNet (a benchmark of images in 1,000 classes). He et al. found that "adding more layers to a suitably deep model leads to higher training error", which they call the degradation problem, and that it is "not caused by overfitting" (He et al., 2015).

A deeper model could in principle copy the shallow one and set the extra layers to the identity, yet the optimiser does not find that solution. The paper's Table 2 reports validation error: the plain 34-layer network had 28.54 percent top-1 error against 27.94 for 18 layers, and the authors report the same ordering for training error in their Figure 4.¹ What you now know: depth alone made training worse.

Step 2: Add the shortcut around the layers

The second step adds the shortcut. The input is x; the output is y = F(x) + x, where F is two or three layers. The addition is element by element, so it adds no parameters and costs almost no computation. He et al. describe the shortcut as performing "identity mapping" and note that it adds "neither extra parameter nor computational complexity" apart from the addition (He et al., 2015).

If the best block is the identity mapping, the layers can drive F toward zero, which is easier than building an identity out of nonlinear layers. The idea has a precursor: highway networks gate the information flow across layers with learned units (Srivastava et al., 2015). What you now know: the block outputs its input plus a correction.

Step 3: Let the gradient take the shortcut

The third step follows the gradient back through the shortcut. The input is the loss gradient at the block's output; the output is the gradient at its input. Differentiating y = x + F(x) gives:

He et al. showed that direct propagation holds "when using identity mappings as the skip connections and after-addition activation" (He et al., 2016). The original ResNet puts a ReLU (rectified linear unit, an activation that zeroes negatives) after the add, so it meets only the first condition; the pre-activation ResNet meets both by moving normalisation and ReLU inside the branch, so nothing follows the add.

The 1 is the shortcut term: a copy of the gradient arrives through it unchanged, whatever the branch does. A plain stack has no 1, only a product of branch gradients.¹ The factor is above 1 whenever the branch gradient is positive, so n blocks multiply n such factors and the signal can grow. What you now know: no layer can shrink the shortcut term, and the same factor can make signals grow.

Step 4: Match shapes and place the normalisation

The fourth step handles the practical details. The input to the add must have the same shape as F(x). When it does not, for example when the number of channels (the depth of the feature maps a convolutional layer produces) changes, He et al. use a linear projection on the shortcut, written $W_{s}x$, and use it "only when matching dimensions" (He et al., 2015).

The alternative is to pad the new channels with zeros (option A, used in the Table 2 comparison, so no parameters are added). The second detail is where normalisation sits. The original transformer applies layer normalization after the add, as LayerNorm(x + Sublayer(x)) (Vaswani et al., 2017); the later placement inside the branch is called pre-norm. The placement is a design choice with training consequences, covered under trade-offs below. What you now know: shapes must match, and normalisation placement matters.

1 · Plain stacks degrade2 · Add the shortcut3 · Gradient takes the shortcut4 · Match shapesReplay

Step-by-step description

1. Layers F map x, and deeper plain stacks reach higher training error.
2. The input x is carried around F and added: y = x + F(x).
3. The shortcut passes the gradient back unchanged: the 1 in 1 + dF/dx.
4. A projection on the shortcut fixes a shape mismatch.

• A residual block computes x plus F of x.
• The gradient through a block is one plus the branch gradient.

Worked example

A scalar model shows the arithmetic. Suppose every layer has a local gradient of 0.9 (illustrative: not a measurement). In a plain stack the gradient reaching the start is 0.9 multiplied once per layer: 0.349 after 10 layers, 0.122 after 20, 0.00515 after 50 and 0.0000266 after 100. In a residual stack each block multiplies the gradient by 1 plus its branch gradient. If the branch gradient is 0, the factor is exactly 1 at any depth, so the gradient arrives intact.¹

This is not the same network as the plain stack: the comparison holds when branches start near zero, which is a deliberate choice. Goyal et al. initialise the scale of the last batch normalization (a layer that rescales activations using the statistics of a training batch) in each residual block to 0, which makes the signal initially propagate through the identity shortcut (Goyal et al., 2017).

The same arithmetic shows the other side. If the branch gradient is 0.1, each block multiplies by 1.1, and the product is 2.59 after 10 blocks, 6.73 after 20, 117 after 50 and 13,800 after 100.¹ If the same 0.9-gradient layers sat inside the branches, the factor would be 1.9 per block: 613 after 10 blocks, about 3.76 times 10 to the power 5 after 20 and about 8.7 times 10 to the power 13 after 50.¹

That case is why branches are scaled down at the start; Fixup initialisation rescales the branch weights so that residual networks train stably without normalisation, even at 10,000 layers (Zhang et al., 2019). A shortcut stops the signal from vanishing, but it does not stop it from growing, which is why residual networks also use normalisation, careful initialisation or small branch scales.

Veit et al. offer a second view: a residual network with n blocks can be unrolled into 2 to the power n paths of different lengths, which they describe as behaving "like ensembles of relatively shallow networks" (Veit et al., 2016). With 16 blocks that is 65,536 paths whose average length is 8 blocks; with 54 blocks, about 1.8 times 10 to the power 16 paths with an average of 27 blocks.¹ The authors report that in a 110-layer network most gradient comes from paths 10 to 34 layers deep, which is 5 to 17 blocks at two layers per block.

The paper's own result confirms the direction. On ImageNet, the plain 34-layer network had 28.54 percent top-1 error and the residual one 25.03, with the same depth, width and parameter count; at 18 layers the two were close at 27.94 and 27.88 (He et al., 2015, Table 2).¹

Text description of the diagram

• The plain 34-layer network had 28.54 percent top-1 error.
• The residual 34-layer network had 25.03 percent top-1 error.

Trade-offs and edge cases

Residual connections are not a cure for everything. First, the claim that they fix vanishing gradients is incomplete. He et al. argue the plain-network difficulty "is unlikely to be caused by vanishing gradients", because their plain networks used batch normalisation, which kept gradient norms healthy; they conjecture exponentially low convergence rates instead (He et al., 2015). Reported explanations differ, so describe it as an optimisation benefit, with the gradient path as one cause.

Second, very deep is not always better. On the Canadian Institute for Advanced Research (CIFAR) 10 image benchmark the 110-layer residual network reached 6.43 percent test error, and the 1,202-layer one 7.93 percent, worse than the shallower one (He et al., 2015, Table 6). Third, placement matters in transformers.

Xiong et al. show that post-norm transformers have large expected gradients near the output at initialisation and need a warm-up stage (a period of very small learning rates at the start of training), while pre-norm keeps gradients well behaved and can train without it; Xiong et al. also note that the pre-norm transformer uses a final layer normalization right before the prediction (Xiong et al., 2020). The placement is drawn below. Fourth, long short-term memory networks have a related idea: their cell state is updated additively and the update is gated.

Text description of the diagram

• Post-norm computes LayerNorm(x + Sublayer(x)).
• Pre-norm computes x + Sublayer(LayerNorm(x)).

For the normalisation choice, see batch norm versus layer norm; the nonlinearity inside F is covered in activation functions.

How to answer it in an interview

The spoken structure is: say the formula, say the problem it solved, then give the gradient argument and one caveat.

Common mistakes

1. Saying it only fixes vanishing gradients. (wrong) "Residual connections exist because gradients vanish." (right) "They fix a degradation in optimisation, and the direct gradient path is one reason."
2. Concatenating instead of adding. (wrong) "The shortcut stacks the input next to the output." (right) "In a ResNet the two are added element by element; concatenation is a different design."
3. Ignoring shapes. (wrong) "Just add x to F(x)." (right) "The shapes must match, so a projection is used when channels change."
4. Thinking a shortcut means no growth. (wrong) "Signals are always stable with residuals." (right) "The path stops shrinking, but repeated adds can grow the signal without normalisation."
5. Assuming every transformer is post-norm, or putting LayerNorm on the shortcut. (wrong) "A transformer always normalises after the add, so the shortcut passes through LayerNorm." (right) "The original design is post-norm, LayerNorm of x plus Sublayer of x; pre-norm computes x plus Sublayer of LayerNorm of x, so normalisation sits inside the branch and the shortcut carries x untouched."

Follow-up questions

Why do skip connections work?

Skip connections work because they give the optimiser and the gradient a direct route around layers. In a residual (additive) skip, a block also only has to learn a correction, and the gradient is not forced through every layer's product. A skip connection is any link that bypasses layers, and a residual connection is the specific case where the bypassed input is added to the output of the layers it skips. Concatenation, as in some segmentation networks, is a skip connection but not a residual one.

What is the purpose of residual layers?

Residual layers let a network be much deeper without training getting worse: each block learns a correction to its input, and the shortcut keeps a direct path for the signal and the gradient.

Why do residual connections help with vanishing gradients?

The derivative of x plus F of x is one plus the derivative of F, so the shortcut term delivers a copy of the gradient to the input however small the branch gradient is. He et al. show direct propagation "when using identity mappings as the skip connections and after-addition activation", which the pre-activation ResNet (normalisation and ReLU moved inside the branch, nothing after the add) satisfies and the original ResNet, with its ReLU (rectified linear unit) after the add, does not (He et al., 2016). The backpropagation page explains the product of factors this avoids.

Where does the residual connection go in a transformer block?

In a transformer, the residual connection wraps each sub-layer: in the encoder, the attention layer and the feed-forward layer, and in the original decoder, a third attention sub-layer over the encoder output as well. The original paper writes the output as LayerNorm(x + Sublayer(x)) (Vaswani et al., 2017). The transformers and self-attention pages show the sub-layers.

What happens if the input and output shapes differ?

The add fails, so a linear projection of the shortcut, or zero-padding of the new channels, matches the dimensions. In ResNet this happens when the number of channels increases (He et al., 2015).

Can residual networks have exploding signals?

Yes. With a branch gradient of 0.1 per block the product is 117 after 50 blocks in the scalar model above, so normalisation, small initial branch weights or scaled branches are used. The exploding gradients page covers the related failure in the backward pass.

Sources

1. Deep Residual Learning for Image Recognition — arXiv (He, Zhang, Ren and Sun), 10 Dec 2015
2. Identity Mappings in Deep Residual Networks — arXiv (He et al.), v3 revised 2016-07-25, 16 Mar 2016
3. Residual Networks Behave Like Ensembles of Relatively Shallow Networks — arXiv (Veit, Wilber and Belongie), v2 revised 2016-10-27, 20 May 2016
4. Highway Networks — arXiv (Srivastava, Greff and Schmidhuber), v2 revised 2015-11-03, 3 May 2015
5. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — arXiv (Goyal et al.), 8 Jun 2017
6. Fixup Initialization: Residual Learning Without Normalization — arXiv (Zhang, Dauphin and Ma), ICLR 2019, 27 Jan 2019
7. On Layer Normalization in the Transformer Architecture — arXiv (Xiong et al.), ICML 2020, v2 revised 2020-06-29, 12 Feb 2020
8. Attention Is All You Need — arXiv (Vaswani et al.), 12 Jun 2017