What is the chain rule in neural networks, and how does backpropagation use it?

In this article

1. Before you start
2. Why interviewers ask this
3. Chain rule in neural networks from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page builds on how neural networks work.

Why interviewers ask this

"Backprop is just the chain rule" is half an answer, and the follow-up goes after the other half. An interviewer who asks you to explain or derive backpropagation is checking whether you can separate the mathematical identity from the algorithm, whether you know what is stored during the forward pass, and whether you can say what goes wrong with depth.

A weak answer recites "propagate the error backward" with no formula. A strong answer writes the gradient of the loss with respect to one weight as a product of local derivatives and says what happens when a value feeds two later operations. A Staff-level answer adds the cost: the forward pass must keep its intermediate values, so memory is the price of speed, and the product of many factors is why very deep networks need care.

Chain rule in neural networks from first principles

Chain rule

The chain rule is a calculus rule that gives the derivative of a function built by composing other functions as the product of the derivatives of its parts, each taken at the value the previous part produced.

Local derivative

A local derivative is the derivative of one operation's output with respect to its own input, evaluated at the values the forward pass produced, so it needs no knowledge of the rest of the network.

In plain words

Picture three gears in a row. Turn the first gear once. The last gear turns by the product of the three gear ratios. Each ratio is local: it only describes one pair of gears. That is the chain rule.

The analogy breaks in one place. A real gear ratio is fixed. A local derivative in a network changes with the current values, so it must be measured at the numbers the forward pass actually produced.

In one precise sentence, if y = f(u) and u = g(x), then the derivative of y with respect to x is the derivative of y with respect to u times the derivative of u with respect to x, as Zhang et al. (2021) state it:

A gradient is the list of partial derivatives of the loss with respect to every weight and bias; Nielsen states the goal of backpropagation as computing exactly these.

Step 1: A forward pass builds a chain of functions

A neural network is a composition of simple operations. Take the smallest useful case, with illustrative values: one input x = 2.0, one weight w = 0.5, one bias b = 0.1 and a target y = 1.0. The network computes a weighted sum z = w·x + b, applies the sigmoid, a function that squashes any number into the range between 0 and 1, to get an activation a, and then scores the miss with a loss L = ½(a − y)². A real layer does the same with a weight matrix and many numbers, as in a feed forward network.

The forward pass computes the weighted sum, then the activation, then the loss, and stores each value.¹ PyTorch does this by recording each operation and its gradient function in a computational graph as the forward pass runs. You now know the shape of the problem: a chain from w to L.

Step 2: Local derivatives multiply along a path

To learn how L changes when w changes, differentiate each link on its own. The weighted sum changes with w at the rate x, the sigmoid changes at the rate a(1 − a), and the loss changes with a at the rate a − y. Each is a local derivative. The chain rule multiplies them:

Each local derivative describes one link of the chain: 2.0 for the weight, 0.1874 for the sigmoid and negative 0.2497 for the loss.¹ Along a single path you multiply; you never add or average. Input: the stored forward values. Output: three small numbers, none of which needed to know about the others.

Step 3: When a value feeds several paths, the path products add

A value in a real network often feeds more than one later operation. The chain rule then sums over every route. Goodfellow, Bengio and Courville (2016) write this in tensor form as a sum over the intermediate variables, and Nielsen's proof of the error term sums over the next layer's neurons.

Try it with an illustrative function: u = x², v = x + 1 and out = u·v at x = 3. The route through u contributes 24 and the route through v contributes 9, so the derivative is 33.¹ When one variable feeds several paths, the path products are added. A gradient is complete only after every route has been counted.

Step 4: Backpropagation is the chain rule in a cached order

Backpropagation is an algorithm that computes the chain rule in an order that reuses work: it starts at the loss, moves toward the inputs, and keeps each shared product so no factor is computed twice. Goodfellow, Bengio and Courville (2016, §6.5.2) say it directly: back-propagation "is an algorithm that computes the chain rule, with a specific order of operations that is highly efficient." Their lecture slides call it dynamic programming that avoids recomputing repeated subexpressions, at the cost of memory; the book's sections 6.5.3 to 6.5.7 cover repeated subexpressions and the memory trade in detail.

In PyTorch, calling .backward() on the root of the graph walks it in reverse and, using the chain rule, passes gradients all the way to the leaf tensors, which hold them in .grad; the pass is seeded with a gradient of 1.0 at the loss, and PyTorch's autograd blog explains why it multiplies vectors by Jacobians instead of building the full matrix. On our example, multiplying the local derivatives from the output backward gives a weight gradient of negative 0.0936.¹ The chain rule is the identity; backpropagation is the schedule.

The same idea written for whole layers is the form interviewers expect when they ask you to derive backpropagation. Let δ of layer l be the error vector of that layer, the derivative of the loss with respect to the layer's weighted sums. Two of Nielsen's four fundamental equations say that the error of one layer comes from the error of the next, and that the weight gradient is that error times the previous layer's activations:

Here the raised T marks the transpose, which swaps rows and columns, and σ′ is the sigmoid's derivative. Reverse mode suits a network because there is one scalar loss and millions of weights, and Schmidhuber (2014) notes that "the costs of forward activation spreading essentially equal the costs of backward derivative calculation", so the backward pass costs about as much as the forward pass.

1 · Forward pass2 · Local derivatives3 · Paths add4 · Backward productReplay

Step-by-step description

1. Compute and store each value.
2. One small derivative per link.
3. Several routes are summed.
4. Multiply from the loss back.

• The forward pass computes the weighted sum, then the activation, then the loss, and stores each value.
• Each local derivative describes one link of the chain: 2.0 for the weight, 0.1874 for the sigmoid and negative 0.2497 for the loss.
• When one variable feeds several paths, the path products are added.
• Multiplying the local derivatives from the output backward gives a weight gradient of negative 0.0936.

Worked example

The values below use the illustrative network from Step 1: x = 2.0, w = 0.5, b = 0.1 and y = 1.0. The weighted sum is 1.1, the activation is 0.7503 and the loss is 0.0312.¹ Because the target is 1.0, the activation falls short, so the loss derivative with respect to a is negative 0.2497, and the sigmoid's local derivative is 0.7503 × 0.2497 = 0.1874.¹ Their product is the gradient of the loss with respect to the weighted sum, negative 0.0468. That one number is reused: multiplying by x = 2.0 gives the weight gradient, and multiplying by 1.0 gives the bias gradient.¹

The finite-difference estimate nudges w up and down by a tiny amount and divides the change in loss by the gap; it agrees with the chain-rule result to six decimal places.¹ That is the standard way to test a hand-derived gradient. One vendor explainer describes backpropagation as the step that finds how each weight contributes to the loss, with gradient descent as the separate step that adjusts the weights. With a learning rate of 0.5 (illustrative), the weight moves from 0.5 to 0.5468, against the sign of the gradient.¹

The nudging idea does not scale. With one million weights (illustrative), nudging each weight twice needs 2,000,000 forward passes for one gradient, while backpropagation needs one forward pass and one backward pass.¹

Trade-offs and edge cases

Backpropagation trades memory for speed. The backward pass needs the values the forward pass produced, so a deep network keeps its intermediate activations alive until the gradients are done. Gradient checkpointing, described by Chen et al. (2016), keeps only some activations and recomputes the rest during the backward pass; that is a speed-for-memory trade in the other direction.

Text description of the diagram

• The chain rule is a calculus identity, while backpropagation is an algorithm that applies it.
• Backpropagation reuses shared factors and returns the derivative for every weight in one backward sweep.

The chain rule is a calculus identity, while backpropagation is an algorithm that applies it. Backpropagation reuses shared factors and returns the derivative for every weight in one backward sweep. The idea is also older than neural-network training: Schmidhuber (2014) notes that a derivation using only the chain rule goes back to Dreyfus in 1962 and that efficient error backpropagation in general networks "apparently" appeared in a 1970 thesis. The 1986 Nature paper by Rumelhart, Hinton and Williams showed that repeatedly adjusting weights this way lets networks create useful new features, which set it apart from the perceptron-convergence procedure.

Depth multiplies factors. The sigmoid's own derivative never exceeds 0.25, so the five sigmoid factors of a five-layer stack multiply to at most about 0.001.¹ Each layer also multiplies by its weights, so the gradient shrinks unless the weights are large (|w| above 4 for a single path); weights that large push the product the other way, which is the story of exploding gradients. Remedies include other activation functions, residual connections that add a shortcut path, and gated cells such as the long short-term memory (LSTM) cell.

Text description of the diagram

• The sigmoid derivative never exceeds 0.25.
• The five sigmoid factors of a five-layer stack multiply to at most about 0.001.

How to answer it in an interview

The spoken structure: state the rule, say a network is a composition, name the product and the sum, then say what backpropagation adds and what it costs.

Common mistakes

• "Backprop is the chain rule, full stop." (wrong) That drops the order and the reuse. (right) "Backprop is an algorithm that computes the chain rule in an efficient, cached order."
• "Backpropagation updates the weights." (wrong) It only produces gradients. (right) "Backprop gives gradients; an optimizer such as gradient descent applies the update."
• "Along a path you add the derivatives." (wrong) Along one path you multiply; you add only across different paths. (right) "Multiply along a path, sum over paths."
• "Calling backward twice replaces the gradients." (wrong) PyTorch accumulates them in .grad. (right) "Gradients accumulate, so training loops clear them before the next backward pass."

Follow-up questions

Is backpropagation the same as the chain rule?

No. The chain rule is the calculus identity, and backpropagation is an algorithm that applies it from the output backward while reusing shared products. Goodfellow et al. also warn that the calculus chain rule is not the chain rule of probability.

Does backpropagation update the weights?

No. It computes the gradients of the loss with respect to every weight and bias. A separate optimizer step then changes the weights, as in the learning-rate step in the worked example.

What happens when one value feeds two later nodes?

Each route gives its own product of local derivatives, and the routes are added. In the illustrative example the two routes give 24 and 9, so the derivative is 33.¹

Why do gradients vanish when many layers are stacked?

The gradient is a product of one weight-times-sigmoid-derivative factor per layer. The sigmoid part is at most 0.25, so with ordinary weights the product shrinks quickly with depth; see exploding gradients for the opposite case.

Why not estimate gradients by nudging each weight?

Each nudge needs forward passes, and there is one weight per nudge. For a network with millions of weights that is millions of passes per step, while backpropagation needs one forward and one backward pass.

Is the chain rule of calculus the same as the chain rule of probability?

No. They share a name only. The probability version factorizes a joint distribution; the probability and statistics guide covers it, and more practice questions are in the deep learning interview questions.

Sources

1. How the backpropagation algorithm works (Neural Networks and Deep Learning, Ch. 2; page last updated 26 Dec 2019) — Michael Nielsen, accessed 2026-10-26, 26 Dec 2019
2. Deep Learning, Ch. 6, section 6.5 Back-Propagation and Other Differentiation Algorithms — Goodfellow, Bengio, Courville (MIT Press), accessed 2026-10-26, 1 Jan 2016
3. Deep Learning, Ch. 6 lecture slides — Goodfellow, Bengio, Courville, accessed 2026-10-26, 1 Jan 2016
4. Learning representations by back-propagating errors — Nature 323, 533-536 (Rumelhart, Hinton, Williams), accessed 2026-10-26, 9 Oct 1986
5. A Gentle Introduction to torch.autograd — PyTorch documentation, accessed 2026-10-26, 26 Oct 2026
6. Overview of PyTorch Autograd Engine — PyTorch blog (published June 2021), accessed 2026-10-26, 1 Jun 2021
7. What is Backpropagation? — IBM Think, accessed 2026-10-26, 26 Oct 2026
8. Dive into Deep Learning, section 2.4.4 Chain Rule (arXiv 2106.11342) — arXiv (Zhang, Lipton, Li, Smola), accessed 2026-10-26, 1 Jun 2021
9. Deep Learning in Neural Networks: An Overview (arXiv 1404.7828) — arXiv (Schmidhuber), accessed 2026-10-26, 1 Apr 2014
10. Training Deep Nets with Sublinear Memory Cost (arXiv 1604.06174) — arXiv (Chen et al.), accessed 2026-10-26, 21 Apr 2016