In this article
Before you start
This page belongs to the machine-learning coding round guide, where coding questions on networks, attention and training loops sit together.
Why interviewers ask this
Candidates report being asked to write a small network, or a custom layer with its own forward and backward, without a framework. The consequence of not knowing the backward pass is that you cannot debug a training run that does not learn, because you cannot say which line of a gradient is wrong.
The question separates three levels. A weak answer recites "backprop uses the chain rule." A strong answer writes the forward pass with its cached values, gives the gradient of each layer with correct shapes, and says where the activation derivative enters. A Staff-level answer adds how to verify the result numerically, what changes for a batch, and why a bad start (all weights equal) stalls learning. The activation used here is the rectified linear unit (ReLU), which a later step defines.
MLP forward and backward from first principles
- Multilayer perceptron
- A multilayer perceptron is a neural network made of stacked layers, each of which multiplies its input by a weight matrix, adds a bias vector and applies a nonlinear activation, so the output of one layer is the input of the next.
- Forward pass
- A forward pass is the computation that carries an input through every layer of a network to produce the output, and in training it also caches each layer's intermediate values so the backward pass can reuse them.
In plain words
Picture a row of workers passing a part down an assembly line. An inspector at the end measures how wrong the finished part is. The inspector tells the last worker how much each of that worker's settings added to the error. That worker tells the one before. The analogy breaks in one place. Real workers pass blame in words. Backpropagation passes exact numbers, found by multiplying local gradients.
The upstream gradient is the gradient of the loss with respect to an operation's output. In one precise sentence: the backward pass multiplies the upstream gradient by that operation's local gradient to obtain the gradient with respect to its input. Rumelhart, Hinton and Williams (1986) published the backpropagation procedure in Nature. The Stanford course notes on backpropagation describe it as the chain rule applied recursively backwards through the circuit. They note that an addition gate has a local gradient of one for each input. So it passes the gradient through unchanged.
Step 1: The forward pass computes and caches every value
The forward pass is the computation that turns an input into a prediction and a loss while saving every intermediate value. For a layer with weight matrix W, bias b and activation f, the pair of equations is z = W x + b and a = f(z). The shape rule is that W has one row per output unit and one column per input.
The example network has one hidden layer with ReLU, which is the function max(0, z): it passes positive inputs and outputs zero otherwise. The output layer uses a sigmoid, p = 1 / (1 + exp(-z)), so the prediction p is a probability, and the loss is binary cross-entropy, L = -(y ln p + (1 - y) ln(1 - p)), where y is the label zero or one.
Input: x. Output: z1, a1, z2, p and the loss. Cache z1, a1 and p, because the backward pass needs them. You now know every value the backward pass will reuse.
Step 2: The output gradient is the prediction minus the label
The output gradient is the gradient of the loss with respect to the output layer's input z2, and for a sigmoid with binary cross-entropy it is the prediction minus the label: dz2 = p - y. The derivation is two local gradients multiplied. The loss changes with p at -y/p + (1-y)/(1-p), the sigmoid changes with z2 at p(1-p), and their product simplifies to p - y. A softmax with cross-entropy gives the same form; see the numerically stable softmax.
Three more lines finish the layer. The weight gradient is dW2 = dz2 a1ᵀ, the bias gradient is db2 = dz2, and the gradient passed to the hidden layer's output is da1 = W2ᵀ dz2. The transpose appears because the forward pass used W2; backward uses its transpose, as Nielsen's chapter on backpropagation explains for general layers.
Input: p, y, a1, W2. Output: dW2, db2, da1.
Step 3: The ReLU gate blocks the gradient of an inactive unit
The ReLU gate is the local gradient of ReLU: one where the unit's input z1 was positive, zero where it was not. The gradient entering the hidden layer is dz1 = da1 * (z1 > 0), an elementwise product, which the Nielsen chapter calls the Hadamard product.
This line is where hidden weights are cut off. A hidden unit whose input was negative on this example passes no gradient, so every weight feeding it gets a zero gradient for this example, however large da1 was. The remaining lines mirror step 2: dW1 = dz1 xᵀ and db1 = dz1. With more layers you would continue with da0 = W1ᵀ dz1 and repeat.
Input: da1, z1, x. Output: dz1, dW1, db1. The recipe is the same for every layer: gate, then weight gradient, then pass back through the transpose.
Step 4: The update is a separate step that uses the gradients
Backpropagation stops once every gradient is computed. Stochastic gradient descent (SGD) is the update rule that moves each weight against its gradient, w = w - lr * grad, where lr is the learning rate, a small positive number chosen by the practitioner. The same rule applies to every weight and bias; gradient descent from scratch covers schedules and variants.
Keeping the two jobs apart matters in an interview. Backprop gives gradients; the optimizer turns them into new weights. A custom autograd function follows the same split: its forward saves what the backward needs and its backward returns the gradients with respect to each input.
Step-by-step description
- Compute and cache every value.
- Prediction minus label.
- The ReLU gate blocks inactive units.
- Weights move against their gradients.
- The forward pass caches the hidden values and the prediction.
- The output gradient is the prediction minus the label.
- The ReLU gate zeroes the gradient of a hidden unit whose input was negative.
- The update subtracts the learning rate times the gradient from each weight.
The forward pass caches the hidden values and the prediction. The output gradient is the prediction minus the label. The ReLU gate zeroes the gradient of a hidden unit whose input was negative. The update subtracts the learning rate times the gradient from each weight.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
The network has two inputs, two hidden ReLU units and one sigmoid output (illustrative). The input is x = (1, 2) with label y = 1. The hidden weights are rows (0.5, −0.2) and (0.3, 0.4) with biases (0.1, −2.0); the output weights are (0.6, −0.4) with bias 0.2; the learning rate is 0.1 (all illustrative).1
Forward: the hidden inputs are 0.2 and −0.9, so the hidden outputs are 0.2 and 0. The output logit is 0.32, the prediction is 0.5793 and the loss is 0.5459.1 Backward: the output gradient is 0.5793 − 1 = −0.4207. The hidden gradient before the gate is (−0.2524, 0.1683); the gate zeroes the second entry because the second hidden input was negative.1
| Quantity | Value |
|---|---|
| Hidden inputs z1 | 0.2 and −0.9 |
| Hidden outputs a1 | 0.2 and 0 |
| Output logit z2 | 0.32 |
| Prediction p | 0.5793 |
| Loss | 0.5459 |
| Output gradient dz2 | −0.4207 |
| Gradient of output weights dW2 | −0.0841 and 0 |
| Gradient da1 | −0.2524 and 0.1683 |
| Gate applied, dz1 | −0.2524 and 0 |
| Gradient of hidden weights, row 1 | −0.2524 and −0.5048 |
| Gradient of hidden weights, row 2 | 0 and 0 |
| Loss after one update | 0.4910 |
After the update the first output weight becomes 0.6084, the output bias 0.2421 and the first hidden row (0.5252, −0.1495); the second hidden row and its bias do not move.1 The first hidden unit's second weight has the largest gradient in magnitude (−0.5048), because its input feature is 2.
Text description of the diagram
- Both weights of the inactive hidden unit have zero gradient.
- The first hidden unit's second weight has the largest gradient in magnitude, 0.5048.
Both weights of the inactive hidden unit have zero gradient. To check the hand result, a finite difference moves one weight by a tiny amount ε in each direction and measures the loss change, (L(w+ε) - L(w-ε)) / (2ε). For every weight the largest gap between that estimate and the hand gradient is below 0.000000001.1
The same step in vectorised code
The code below is the whole step for a batch, in NumPy, using the rule that every array has one row per example. It runs the 2-2-1 network of the table on the single example (illustrative values).
import numpy as np
W1 = np.array([[0.5, -0.2], [0.3, 0.4]]); b1 = np.array([0.1, -2.0])
W2 = np.array([[0.6, -0.4]]); b2 = np.array([0.2])
X = np.array([[1.0, 2.0]]); y = np.array([[1.0]]); lr = 0.1
def forward(X, y, W1, b1, W2, b2):
z1 = X @ W1.T + b1 # one row per example
a1 = np.maximum(0.0, z1) # ReLU
z2 = a1 @ W2.T + b2
p = 1 / (1 + np.exp(-z2)) # sigmoid
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
return z1, a1, p, loss # cache z1, a1, p
z1, a1, p, loss = forward(X, y, W1, b1, W2, b2)
n = X.shape[0]
dz2 = (p - y) / n # mean loss -> divide by n
dW2 = dz2.T @ a1
db2 = dz2.sum(axis=0) # biases: sum over the batch
da1 = dz2 @ W2 # multiply by W2 (rows of dz2 are examples)
dz1 = da1 * (z1 > 0) # the ReLU gate
dW1 = dz1.T @ X
db1 = dz1.sum(axis=0)
W1 -= lr * dW1; b1 -= lr * db1 # SGD update
W2 -= lr * dW2; b2 -= lr * db2Run as assets/data/mlp-backprop-numpy.py, it gives dz2 = −0.4207, dW2 = (−0.0841, 0), dW1 row 1 = (−0.2524, −0.5048), loss 0.5459 before the update and 0.4910 after, the same values as the table.1 The line da1 = dz2 @ W2 is the transpose rule in batch form: with examples as rows, the transpose is already absorbed. On the two-example batch of the next section the same code gives a mean gradient of 0.1695 for the first hidden weight.1
Trade-offs and edge cases
Batches. With several examples the loss is the mean of the per-example losses, so the gradient is the mean of the per-example gradients. For a second example (illustrative, x = (1.5, 0.5), label 0) the first hidden weight's gradient is −0.2524 for the first example and 0.5913 for the second; their sum is 0.3389 and their mean is 0.1695.1 The rule is to match the gradient's reduction to the loss's: a mean loss needs an average, and summing is right only when the loss itself is a sum. Summing under a mean loss multiplies the effective learning rate by the batch size, a common bug.
Equal starting weights. If all weights in a layer and its outgoing weights start at the same value, its units compute the same thing and receive the same gradient. In the computed check, with every hidden weight 0.3 and both output weights 0.5 (illustrative), both hidden rows get identical gradients, so they stay identical after every update.1 Random starting weights break the tie.
Dead and saturated units. A ReLU unit whose input is negative for every example never receives a gradient, so it stops learning. A sigmoid's local gradient is at most 0.25, at an input of zero, so chains of sigmoids tend to shrink gradients as they go back.1 The ReLU derivative at exactly zero is a convention; the example avoids it.
Text description of the diagram
- Hand backprop needs one backward pass and is exact up to rounding.
- Finite differences need two forward passes per weight and are approximate.
Hand backprop needs one backward pass and is exact up to rounding. Finite differences need two forward passes per weight and are approximate. Autograd, a framework's automatic differentiation, builds the same chain-rule computation for you. As of October 2026, frameworks build this graph automatically, so training code rarely writes a backward pass; the hand derivation is what you use to test a custom layer and to debug a run that does not learn. The same pattern of cached values and reversed products underlies building a transformer language model from scratch and scaled dot-product attention from scratch.
How to answer it in an interview
Walk me through a forward and backward pass of a small MLP by hand.
In the forward pass I compute z equals W x plus b, then the activation, layer by layer, and I cache z and a. For a sigmoid output with cross-entropy, the gradient at the logit is prediction minus label. Each weight gradient is that layer's gradient at z, its delta, times the previous activation transposed. I pass delta back through W transposed and multiply by the activation's derivative, which for ReLU is a zero-or-one gate. Backprop only gives gradients; SGD does the update. I verify with finite differences and average over the batch.
The spoken structure: forward with caching, the output delta, the transpose-and-gate rule for every earlier layer, the update as a separate step, then the check.
Common mistakes
- "Backprop updates the weights." (wrong) It only computes gradients. (right) "Backprop gives gradients; the optimizer applies them."
- "I multiply by W in the backward pass." (wrong) The shapes fail or the math is wrong. (right) "I multiply by the transpose of W, because the forward pass used W."
- "I forget the activation derivative." (wrong) Without the gate, the dead unit's weights would move. (right) "I multiply the gradient by the ReLU gate before computing the weight gradient."
- "I sum gradients over the batch." (wrong) With a mean loss the step size then grows with the batch. (right) "I match the reduction to the loss: mean loss, average gradient."
- "I start all weights at zero." (wrong) Units stay identical. (right) "I start with small random values."
Follow-up questions
How do you write a custom autograd function from scratch, with its forward and backward passes?
In PyTorch you subclass torch.autograd.Function and write two static methods, as the PyTorch notes on extending autograd describe. forward(ctx, ...) computes the output and calls ctx.save_for_backward(...) for the tensors the backward needs, such as the input or the activation. backward(ctx, grad_output) receives the upstream gradient and returns one gradient per forward input, multiplying by the operation's local gradient, with None for inputs that are not tensors. You call the function through its apply method. Verify it with torch.autograd.gradcheck on double-precision inputs, which compares your backward with finite differences.
How do you implement a basic MLP forward pass without a framework?
Store each layer's weights as an array with one row per output unit, compute x @ W.T + b, apply the activation and repeat, as in the vectorised code above. Keep every z and a in a list so the backward pass can reuse them. The same layout in a single PyTorch layer is torch.nn.Linear, which holds a weight of shape output by input.
How do you check that your hand-written gradients are correct?
Perturb each weight by a tiny amount in both directions, measure the loss change and divide by twice the step, then compare with your gradient. In the example above the gap is below 0.000000001.1
What changes in the backward pass when you use a batch?
Every array gains a batch dimension, the weight gradient becomes a sum of per-example outer products, and you divide by the batch size so it matches the mean loss.
What if the hidden activation is tanh instead of ReLU?
Only the gate changes: replace the zero-or-one gate with the tanh derivative, one minus the output squared. Everything else, including the transposes, stays the same. The same layer-by-layer logic applies to multi-head attention and to a two-dimensional convolution from scratch.
Why not start every weight at zero?
Equal weights give equal gradients, so units in a layer never differ. Random starting values break the symmetry.
- The forward pass caches every layer's z and a; the backward pass reuses them.
- For a sigmoid output with cross-entropy, the output gradient is the prediction minus the label.
- Pass gradients back through the transposed weights, then through the activation's local gradient.
- ReLU blocks the gradient of any hidden unit whose input was negative.
- Backprop computes gradients; SGD updates; finite differences check.
Sources
- Learning representations by back-propagating errors — Nature (Rumelhart, Hinton, Williams), 9 Oct 1986
- Backpropagation, Intuitions (course notes) — Stanford University (accessed 2026-10-26; page undated), 26 Oct 2026
- How the backpropagation algorithm works (Neural Networks and Deep Learning, ch. 2) — Michael Nielsen (accessed 2026-10-26; page undated), 26 Oct 2026
- Extending PyTorch: custom autograd Functions — PyTorch documentation (read 2026-10-26; page undated), 26 Oct 2026
