What is backpropagation, and how does it work?

In this article

1. Before you start
2. Why interviewers ask this
3. Backpropagation from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning path: how neural networks work. Loss functions such as cross-entropy come from probability, covered in the statistics interview guide for machine learning roles.

Why interviewers ask this

Backpropagation is the mechanism behind every training run, so interviewers use it to separate candidates who can name it from candidates who can run it by hand. It appears in deep learning rounds, often as "explain it to someone with no background", then "derive it on the whiteboard", then "what goes wrong in deep networks".

A weak answer says "the error flows backward and the weights get fixed", which blends gradient computation with the weight update. A strong answer says what is computed (one gradient per parameter), how (the chain rule, one layer at a time, using saved forward values) and what it costs. A Staff-level answer adds the trade-offs: saved activations dominate training memory, and repeated multiplication of small factors makes early layers learn slowly. The deep learning interview questions page shows where this sits among neighbouring questions.

Backpropagation from first principles

Backpropagation

Backpropagation is an algorithm that computes the gradient of a network's loss with respect to every weight and bias in one forward pass and one backward pass, by applying the chain rule layer by layer from the output toward the input and reusing values saved in the forward pass. It computes gradients only; it does not change the weights.

Local gradient

A local gradient is the derivative of one operation's output with respect to one of its own inputs, computed from that operation's own values alone. Backpropagation multiplies it by the gradient arriving from later operations, called the upstream gradient, to get the gradient for that input.

In plain words

Picture an assembly line that ships a flawed product. To fix it you need to know how much each station contributed to the flaw. You ask the last station first. It reports how sensitive the product is to its work, and passes that to the station before it. That station multiplies in its own sensitivity and passes the result on, back to the start. The analogy breaks in one place: stations do not decide anything on the way back. The blame is computed exactly by multiplication, using measurements taken while the product was being made.

In one precise sentence, backpropagation applies the chain rule, the calculus rule that the derivative of a composition is the product of the derivatives of its parts, from the loss back to each parameter.

Michael Nielsen describes the aim as an expression for the derivative of the cost with respect to any weight or bias, obtained with one forward pass and one backward pass. The CS231n notes call the process local: every gate computes its output and its local gradient from its own inputs.

Step 1: The forward pass saves every intermediate value

The forward pass runs the network on one example and stores each layer's weighted sum and activation. Input: the example and the current weights. Output: a prediction, a loss, and a cache of intermediate values.

The cache is not optional. The local gradient of a sigmoid neuron is its output times one minus its output, and the local gradient of a weight is the input that weight multiplied. Both are forward-pass values. Without the cache the backward pass would have to recompute them. This is also why training needs more memory than prediction: the saved values stay alive until the backward pass uses them. In the CS231n circuit example the add gate received two inputs and produced an output, and that stored output is what later gates use. You now know what the backward pass will consume.

Step 2: The loss gives the first error signal at the output

The backward pass starts at the loss. Input: the prediction and the target. Output: the error signal of the output layer, which is how much the loss changes per unit change in the output neuron's weighted sum.

For a squared-error loss and a sigmoid output, that signal is the loss slope, prediction minus target, times the output neuron's local gradient. For a softmax output with cross-entropy loss it simplifies to predicted probabilities minus the one-hot target (derived in the follow-ups). Nielsen's chapter calls this layer quantity the error and writes it with the Greek letter delta, and it assumes the cost is a function of the network's outputs. In symbols, write ${\delta }^l$ for the error of layer $l$, $C$ for the cost, $a^L$ for the output activations, $z^L$ for the output weighted sums and ${\sigma }^{\prime }$ for the activation's slope:

Here the dot is an elementwise product: for each output neuron, the cost slope is multiplied by that neuron's own activation slope. You now have one number per output neuron that says which way and how hard to push.

Step 3: The error travels backward through each layer

To move the error one layer back, multiply it by the transposed weights of the layer in front, then by the local gradient of the earlier layer's activation. Input: the error of layer l+1. Output: the error of layer l.

In symbols, for one layer:

The dot is again the elementwise (Hadamard) product used in Nielsen's equations: the transposed weights carry the next layer's error back, and each entry is then scaled by the activation slope at layer $l$'s saved weighted sums. Each layer needs only its neighbour's error and its own cached values, which is the locality CS231n stresses. The step repeats once per layer, so the backward pass has the same order of cost as the forward pass. Nielsen puts the whole of backpropagation at roughly two forward passes, and in large dense networks the backward pass does about twice the forward pass's arithmetic (Kaplan et al., 2020, §2.1); estimates differ with the layer type. You now hold an error for every layer.

Step 4: Each weight's gradient is an error times an input

The gradient of a weight is the error of the neuron it feeds times the input it carried. The gradient of a bias is that neuron's error. Input: the layer errors and the cached inputs. Output: the full gradient. In symbols, with $j$ the neuron the weight feeds and $k$ the neuron it comes from:

As Nielsen presents it, backprop computes the gradient for one example; implementations do the same in matrix form over a mini-batch. Nielsen notes the method assumes the cost is an average over per-example costs, so per-example gradients are averaged over a batch. Then a separate step, gradient descent or a variant, subtracts a small multiple of the gradient from each weight. That is the learning; backpropagation supplied only the direction. Keeping the two separate is the point most often confused, and the optimizer states page covers what optimizers add on top.

1 · Forward and save2 · Output error3 · Error goes back4 · Weight gradientsReplay

Step-by-step description

1. Run forward and keep every value.
2. The loss gives the first error.
3. Weights and local slopes carry it back.
4. Error times saved input per weight.

• The forward pass saves hidden activations and gives an output of 0.5365 and a loss of 0.10743.
• The output error is -0.11527 and the hidden errors are -0.0137 and 0.01662.
• Each weight gradient is an error times a saved input.

Worked example

A tiny network shows every number. It has two inputs, 0.5 and 1.0, two sigmoid hidden neurons and one sigmoid output, a target of 1, and squared-error loss of half the squared difference. The weights are illustrative, and the network has nine parameters.¹

Forward pass: the hidden weighted sums are 0.45 and 0.4, the hidden activations are 0.6106 and 0.5987, the output weighted sum is 0.1461, the output is 0.5365 and the loss is 0.10743.¹ Backward pass: the output error is -0.11527, the hidden errors are -0.0137 and 0.01662, the gradients of the two output weights are -0.07039 and -0.06901, and the first hidden neuron's weight gradients are -0.00685 and -0.0137.¹

Notice that the hidden errors are about seven to eight times smaller than the output error (0.11527 against 0.01662 and 0.0137).¹ Each path multiplied the signal by a weight of 0.5 or 0.6 and by a sigmoid slope near 0.24, so the signal shrank. That is a miniature of the vanishing-gradient problem discussed below.

The smallest possible case is the CS231n circuit f = (x + y)z with x = -2, y = 5 and z = -4, where q = x + y is 3 and f is -12. Backward, the gradient with respect to q is z, -4, and with respect to z is q, 3; the add gate passes -4 on unchanged to both x and y.¹

Text description of the diagram

• In the circuit q is 3 and f is -12.
• The gradient with respect to q is -4 and with respect to z is 3.
• The add gate passes -4 to both x and y.

The core of the script that produced the gradients is short:

d_out = (a - y) * a * (1 - a)
dw2 = [d_out * h[i] for i in range(2)]
d_hid = [w2[i] * d_out * h[i] * (1 - h[i]) for i in range(2)]
dW1 = [[d_hid[i] * x[j] for j in range(2)] for i in range(2)]

The first line is Step 2, the third is Step 3 and the second and fourth are Step 4. Every term except w2 and x is a value saved in the forward pass.

Trade-offs and edge cases

Backpropagation trades memory for speed. Saving activations keeps the backward pass to the same order of cost as the forward pass: Nielsen puts all of backpropagation at roughly two forward passes, and dense layers do about twice the forward arithmetic going backward (Kaplan et al., 2020, §2.1). The saved values occupy memory for the whole pass. Practitioners cut that memory with lower-precision numbers (see 16-bit floating point (FP16) versus brain floating point 16 (BF16)) or with gradient checkpointing, which Chen et al. (2016) applied to deep networks, recomputing activations for about one extra forward pass instead of storing them; the trade is memory against extra compute. Both techniques are common ways to fit larger models in memory.

It also has a numerical weakness. Because errors are products of per-layer factors, many factors below one shrink the signal toward zero (vanishing gradients) and many above one blow it up, covered in exploding gradients. The sigmoid's slope never exceeds one quarter, which is one reason other activation functions are preferred in deep networks, and residual connections give the error a shortcut. Karpathy (2016) calls backpropagation a leaky abstraction: saturated sigmoids, dead ReLUs and exploding recurrent gradients are only diagnosable if you know the mechanism.

Edge cases: the same method serves recurrent networks, applied to the network unrolled over time (see long short-term memory (LSTM)); and a network of shared weights, as in convolutional networks, adds up the gradient contributions from every place the weight is used.

Text description of the diagram

• Backpropagation is exact and needs one forward and one backward pass.
• A central-difference check needs 18 forward passes for nine parameters and is used to test the code.

How to answer it in an interview

The spoken structure: say what it computes, say forward-then-backward with saved values, give the layer rule, then name the memory cost and that the optimizer is separate.

Common mistakes

• "Backpropagation updates the weights." (wrong) It only computes gradients. (right) "Backpropagation gives the gradient; gradient descent uses it to update the weights."
• "It nudges each weight and re-runs the network." (wrong) That is a numerical gradient, which needs two forward passes per weight. (right) "Backprop is exact and gets every gradient from one backward pass."
• "It was invented in 1986." (wrong) Nielsen says the algorithm dates to the 1970s and its importance was appreciated later. (right) "The 1986 Nature paper popularised it by showing it learns useful internal features."
• "Only neural networks use it." (wrong) It is a general way to differentiate a composition of simple operations. (right) "It is reverse-mode differentiation applied to a network's computation graph."

Follow-up questions

How would you explain backpropagation to someone with no background?

A network guesses, the guess is scored by a loss, and backpropagation works out how much each weight contributed to the score by passing blame backward through the layers. A separate step then changes each weight a little to reduce the loss.

How do you derive backpropagation by hand?

Write the loss as nested functions of the weights, then apply the chain rule from the outside in: the gradient of a weight is the loss slope times each local gradient on the path to that weight. The first-principles Steps 2 to 4 above are exactly that derivation.

What is the mathematical formulation of backpropagation in a simple neural network?

Four equations define it: the output error ${\delta }^L$ is the cost slope times ${\sigma }^{\prime }(z^L)$; each earlier error is $(W^{l+1}{)}^{\mathrm{T}}{\delta }^{l+1}$ times ${\sigma }^{\prime }(z^l)$ elementwise; a weight's gradient is $a_k^{l-1}{\delta }_j^l$; and a bias's gradient is ${\delta }_j^l$. Steps 2 to 4 above derive each one.

How do you derive the softmax gradient by hand?

The softmax turns logits $z_i$ into probabilities:

$$p_i=\frac{e^{z_i}}{\sum _j{e}^{z_j}}$$

The loss is minus the log of the true class's probability. That log equals $z_y$ minus the log of the sum, so the derivative of the log probability with respect to $z_i$ is one when $i$ is the true class and zero otherwise, minus $p_i$. Negating it gives, with $y_i$ the one-hot target:

$$\frac{\partial L}{\partial z_i}=p_i-y_i$$

For logits 2, 1 and 0 with class 0 as the target, that gives -0.3348, 0.2447 and 0.09, matching a numerical check.¹

How do neural networks learn with backpropagation?

A network learns by repeating two separate jobs: backpropagation computes the gradient of the loss for every weight, and gradient descent, or an optimizer such as those in optimizer states, turns that gradient into a small weight update. The two are not the same thing, and in deep networks the gradient can shrink or blow up because it is a product of one factor per layer.

How do you check that a backpropagation implementation is correct?

Compare each gradient with a central-difference estimate on a tiny network, as the worked example does, where the largest gap is 2.26e-12.¹ The CS231n gradient-check notes advise comparing by relative error rather than absolute difference, because a large relative error points to a bug.

Sources

1. How the backpropagation algorithm works (Neural Networks and Deep Learning, chapter 2) — Michael Nielsen (online book, year only: 2015), 1 Jan 2015
2. Backpropagation (CS231n optimization notes) — Stanford CS231n (accessed 2026-10-26), 26 Oct 2026
3. Learning representations by back-propagating errors — Nature 323, 533-536 (Rumelhart, Hinton, Williams), 9 Oct 1986
4. Yes you should understand backprop — Andrej Karpathy (Medium, December 2016), 19 Dec 2016
5. Scaling Laws for Neural Language Models (arXiv 2001.08361) — arXiv (Kaplan et al.), 23 Jan 2020
6. Training Deep Nets with Sublinear Memory Cost (arXiv 1604.06174) — arXiv (Chen et al.), 21 Apr 2016
7. Neural Networks Part 3: Learning and Evaluation (CS231n gradient checks) — Stanford CS231n (year only: 2016), 1 Jan 2016