Backpropagation from scratch in NumPy: the backward pass and a gradient check

In this article

1. Before you start
2. Why interviewers ask this
3. Backpropagation in NumPy from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning learning path. The general idea is in what backpropagation is and how it works; this page builds the working code.

Why interviewers ask this

A question that asks you to write backpropagation, or to derive it on a whiteboard, separates people who use a framework from people who can debug one. The Stanford course notes on backpropagation argue that understanding the process is what lets you develop, design and debug neural networks, even though frameworks compute the gradients automatically.

A weak answer says "the error flows backward through the network" and stops. A strong answer names what is cached in the forward pass, writes the error term of each layer, gets the transposes and shapes right, separates gradient computation from the weight update, and says how to test the result. A Staff-level answer adds why the backward pass is cheap compared with numerical differentiation, where vanishing gradients come from, and how a framework's automatic differentiation generalizes the same idea.

Backpropagation in NumPy from first principles

Error term

An error term (written δ, delta) is a per-neuron number, the derivative of the loss with respect to that neuron's pre-activation input, that says how much a small change in that input would change the loss, and from which every weight gradient of the neuron's layer is built.

Upstream gradient

An upstream gradient is the derivative of the final loss with respect to the output of a node, handed backward from the node after it, that the node multiplies by its own local gradient to obtain the derivative of the loss with respect to its input.

Gradient check

A gradient check is a test that compares the gradient from backpropagation with a numerical estimate made by nudging each parameter up and down by a tiny amount, and flags a bug when the two differ by more than a small relative error.

In plain words

Imagine a cake that tastes too sweet. You trace the problem backward through the recipe. Each step gets blame in proportion to how much changing that step would have changed the taste. The blame at one step is the blame of the next step times how strongly this step feeds it. The analogy breaks in one place. A network's blame is a precise derivative, valid for tiny changes, and every weight gets its number in one sweep.

In one precise sentence, backpropagation multiplies local derivatives along every path from a weight to the loss, and reuses shared pieces so that all gradients come from one backward sweep. The course notes put it as "a way of computing gradients of expressions through recursive application of chain rule".

The network below has two inputs, two hidden neurons and one output, with the sigmoid on every neuron. The sigmoid is a function, σ(z) = 1 / (1 + e^(−z)), that squeezes any number into the range between 0 and 1, and its derivative is σ(z)(1 − σ(z)).

Step 1: The forward pass computes the loss and caches every value

The forward pass multiplies the input by the hidden weights, adds the bias and applies the sigmoid: z_hid = W_hid x + b_hid, then a_hid = σ(z_hid). The output layer repeats the same two lines on a_hid and gives z_out and a_out. The loss is the quadratic cost, C = ½ (a_out − y)².

The forward pass caches every pre-activation and activation. The backward pass needs them, and the course notes give the rule in three words: "Cache forward pass variables." Nielsen's derivation assumes the cost is an average of per-example costs and a function of the output activations (Nielsen, 2015 book). Input: x and the weights. Output: the loss and the cached z and a values. You now hold everything the backward pass reads.

Step 2: The output error starts the backward pass

The output error term is the loss gradient times the activation derivative: δ_out = (a_out − y) ⊙ σ′(z_out). The symbol ⊙ is the elementwise product, which multiplies two lists entry by entry. For the quadratic cost the loss gradient is a_out − y, as Nielsen's chapter states.

The two factors have a plain reading. The first says how far the output is from the target. The second says how much the output neuron responds to a nudge of its input. The sigmoid's derivative is a(1 − a), so the cached a_out gives it for free; the course notes show the same pattern, an input of 1.0 giving an output near 0.73 and a local gradient near 0.2.

Input: a_out, y, z_out. Output: δ_out, the first upstream gradient of the backward sweep. Every later number is built from it.

Step 3: The backward pass multiplies by the activation derivative

The hidden error term takes the output error term back through the transposed output weights and then through the hidden activation derivative: δ_hid = (W_outᵀ δ_out) ⊙ σ′(z_hid). The backward pass multiplies each error term by the transposed weights and the activation derivative.

The transpose is the shape fix. W_out maps hidden values to outputs, so its transpose maps output errors back to hidden neurons. In this network the shapes catch a missing transpose, but with square matrices a missing transpose runs silently and gives wrong numbers.

The second factor is the one the lead warned about. Because the sigmoid derivative is at most 0.25 (computed, see below), leaving it out makes the hidden gradients at least four times too large; in the worked example they come out about four times too large.¹ Each node does one local multiplication. As the course notes say, backpropagation is "a beautifully local process".

Step 4: Every gradient is an error term times an input

Once the error terms exist, the gradients are short: ∂C/∂b = δ and ∂C/∂W = δ a_prevᵀ, where a_prev is the activation feeding the layer. These two lines and the two error-term lines of Steps 2 and 3 are Nielsen's four fundamental equations.

One consequence follows from the second line: a weight fed by a neuron with a low activation gets a small gradient and learns slowly. Backpropagation stops here. It has produced gradients and changed no weight.

The update is a separate rule. Gradient descent writes it as θ -= α ∇θ J in the Stanford section slides. For a mini-batch of m examples, Nielsen's rule is W → W − (η/m) Σ δ a_prevᵀ, summed over the batch, with learning rate η. Input: the error terms and cached activations. Output: one gradient per parameter.

Step 5: A gradient check proves the backward pass right

A gradient check compares the analytic gradient with a numerical estimate, and it is how you know the other four steps are correct. For each parameter, nudge it by h in each direction and difference the loss. The course notes recommend the centered formula (f(x+h) − f(x−h)) / (2h), with h around 1e-5. They mark the one-sided version "bad, do not use", because its error shrinks in proportion to h, against h squared for the centered one.

Compare with the relative error |a − n| / max(|a|, |n|). The same notes say a relative error above 1e-2 usually means the gradient is probably wrong, and 1e-7 or less is good. They also report errors falling from 1e-2 to 1e-8 on switching to double precision.

The cost is two loss evaluations per parameter, which is 18 for the 9 parameters here.¹ That cost is why the check is a test, never the training method.

1 · Forward pass2 · Output error3 · Backward pass4 · Gradients and update5 · Gradient checkReplay

Step-by-step description

1. Compute the loss and cache every value.
2. Loss gradient times activation derivative.
3. Transposed weights, then activation derivative.
4. Error term times input, then a step.
5. Compare with a numerical estimate.

• The forward pass caches every pre-activation and activation.
• The backward pass multiplies each error term by the transposed weights and the activation derivative.
• A gradient check compares the analytic gradient with a numerical estimate.

Worked example

Start with a circuit small enough to do in your head, the example from the Stanford notes: f = (x + y) z with x = −2, y = 5 and z = −4. The forward pass gives q = 3 and f = −12. The backward pass gives a gradient of 3 for z and −4 for q, x and y. The multiplication node swaps its inputs, so the gradient for z is q and the gradient for q is z; the addition node passes −4 to both of its inputs.

Text description of the diagram

• The forward pass gives q = 3 and f = -12.
• The backward pass gives a gradient of 3 for z and -4 for q, x and y.

Now the 2-2-1 sigmoid network of this page, with illustrative values: hidden weights 0.15, 0.20, 0.25, 0.30, output weights 0.40 and 0.45, all biases 0, input (0.05, 0.10), target 0.01 and learning rate 0.5.¹ Every value below is computed from them.

The same arithmetic in NumPy, vectorized, follows. Serious libraries use this matrix form or a variant (Nielsen).

import json
import numpy as np

x = np.array([0.05, 0.10]); y = np.array([0.01])
W1 = np.array([[0.15, 0.20], [0.25, 0.30]]); b1 = np.zeros(2)
W2 = np.array([[0.40, 0.45]]);               b2 = np.zeros(1)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def forward(x):
    z1 = W1 @ x + b1;  a1 = sigmoid(z1)
    z2 = W2 @ a1 + b2; a2 = sigmoid(z2)
    return z1, a1, z2, a2

def loss():
    return 0.5 * np.sum((forward(x)[3] - y) ** 2)

def backward(x, y):
    z1, a1, z2, a2 = forward(x)
    d2 = (a2 - y) * a2 * (1 - a2)         # output error term
    d1 = (W2.T @ d2) * a1 * (1 - a1)      # transpose, then sigma-prime
    return {"W2": np.outer(d2, a1), "b2": d2,
            "W1": np.outer(d1, x),  "b1": d1}

def num_grad(A, h=1e-5):                  # centered difference
    g = np.zeros_like(A)
    for i in np.ndindex(A.shape):
        old = A[i]
        A[i] = old + h; up = loss()
        A[i] = old - h; down = loss()
        A[i] = old
        g[i] = (up - down) / (2 * h)
    return g

grads = backward(x, y)
errors = {}
for name, A in [("W1", W1), ("b1", b1), ("W2", W2), ("b2", b2)]:
    n = num_grad(A)
    errors[name] = float(np.max(np.abs(grads[name] - n)
                         / np.maximum(np.abs(grads[name]), np.abs(n))))
print(json.dumps(errors))

After the check passes, the update is W2 -= lr * grads["W2"], with the illustrative learning rate 0.5.¹ The same line updates the other three arrays. The check ran in double precision because NumPy floats default to it. This block is saved unchanged as assets/data/backprop-numpy.py; running it prints one JavaScript Object Notation (JSON) object with the largest relative error for each of the four arrays, and each is below 1e-7 (the saved values are in assets/data/backprop-numpy.json).¹

Training the network on a mini-batch

Training repeats the sweep over a mini-batch, a small group of m examples whose gradients are averaged. Stack the examples as rows, so the input X has shape (m, n), and store each weight matrix as (inputs, outputs). The layer gradient is then A_prevᵀ δ / m, and the bias gradient is δ summed over the batch axis and divided by m.

With a sigmoid output and the cross-entropy loss, the output error term simplifies to a_out − y, because the σ′ factor cancels (Nielsen, chapter 3). The script below trains on the exclusive-or (XOR) function, using all four input pairs as one batch, with four hidden neurons, learning rate 2.0 and 5,000 epochs (all illustrative). It runs a batched gradient check first and records the loss every 1,000 epochs.¹

import json
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
Y = np.array([[0], [1], [1], [0]], dtype=float)
rng = np.random.default_rng(0)
W1 = rng.normal(0, 1, (2, 4)); b1 = np.zeros(4)
W2 = rng.normal(0, 1, (4, 1)); b2 = np.zeros(1)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def forward(X):
    A1 = sigmoid(X @ W1 + b1)
    return A1, sigmoid(A1 @ W2 + b2)

def loss():                                # cross-entropy, averaged
    A2 = forward(X)[1]
    return float(-np.mean(Y * np.log(A2) + (1 - Y) * np.log(1 - A2)))

def backward():
    A1, A2 = forward(X)
    m = len(X)
    d2 = A2 - Y                            # sigma-prime cancels
    d1 = (d2 @ W2.T) * A1 * (1 - A1)
    return {"W1": X.T @ d1 / m, "b1": d1.sum(axis=0) / m,
            "W2": A1.T @ d2 / m, "b2": d2.sum(axis=0) / m}

def num_grad(A, h=1e-5):
    g = np.zeros_like(A)
    for i in np.ndindex(A.shape):
        old = A[i]
        A[i] = old + h; up = loss()
        A[i] = old - h; down = loss()
        A[i] = old
        g[i] = (up - down) / (2 * h)
    return g

params = {"W1": W1, "b1": b1, "W2": W2, "b2": b2}
g = backward()
check = max(float(np.max(np.abs(g[k] - num_grad(P))
            / np.maximum(np.maximum(np.abs(g[k]), np.abs(num_grad(P))), 1e-12)))
            for k, P in params.items())

losses = []
for epoch in range(5001):
    if epoch % 1000 == 0:
        losses.append(round(loss(), 4))
    g = backward()
    for k, P in params.items():
        P -= 2.0 * g[k]                    # in-place update

print(json.dumps({"max_relative_error": check,
                  "loss_every_1000_epochs": losses,
                  "final_outputs": [round(float(v), 3) for v in forward(X)[1].ravel()]}))

This block is saved unchanged as assets/data/backprop-batch.py, and its printed JSON is saved in assets/data/backprop-batch.json. Read the recorded losses there rather than trusting a claim on this page.

Trade-offs and edge cases

Backpropagation wins because it is cheap. Nielsen explains that a gradient by finite differences needs one forward pass per weight, so a network with a million weights needs a million and one forward passes per training example, while backpropagation uses one forward and one backward pass, with a backward pass of roughly the cost of the forward one.

The survey by Baydin et al. describes reverse mode as generalizing backpropagation, and says forward mode needs n evaluations for a function from n inputs to one output.

Edge cases. First, gradients add up at forks: if a value feeds two nodes, its gradient is the sum, so code must use += and not = (course notes). Second, kinks such as the rectified linear unit can cause false failures in a gradient check, and dropout must be off or seeded during it (notes). Third, the equations hold for any activation function, because the derivation uses no special property of the sigmoid (Nielsen). Fourth, the sigmoid derivative never exceeds 0.25. Five sigmoid layers shrink the product of derivative factors by at least 1,024 times.¹ Weights can offset that, which leads to exploding gradients, and residual connections are one remedy for the shrinking case.

Text description of the diagram

• The sigmoid derivative never exceeds 0.25.
• Five sigmoid layers shrink the product of derivative factors by at least 1,024 times.

How to answer it in an interview

The spoken structure: say what is cached, give the output error term, give the backward step with its transpose, separate the gradient from the update, then name the gradient check.

Common mistakes

• "Backpropagation updates the weights." (wrong) It computes gradients only. (right) "Backpropagation gives the gradient and gradient descent applies it."
• "The output error term is enough; the hidden layer needs no derivative." (wrong) That drops the activation derivative and inflates the hidden gradients. (right) "Every layer multiplies by its own activation derivative."
• "I check with (f(x+h) − f(x)) / h." (wrong) The one-sided difference is the version the course notes mark "bad, do not use". (right) "I use the centered difference and the relative error."
• "At a fork I assign the gradient." (wrong) Assigning overwrites the first branch. (right) "Gradients add up at forks, so I accumulate."
• "Backpropagation needs the sigmoid." (wrong) The equations hold for any activation function. (right) "Only the derivative term changes."

Follow-up questions

Why do you cache the forward pass values?

The backward pass reuses them: the pre-activations give the activation derivatives and the activations feed each weight gradient. Without the cache the forward pass would run again for every layer. The cost is memory, which grows with depth and batch size.

How do you check that your backpropagation is correct?

Run a gradient check: compare each analytic gradient with a centered-difference estimate and compute the relative error. A value near 1e-7 or below is good, and above 1e-2 usually means a bug. In between, the notes call the result uncomfortable, and for a function without kinks even 1e-4 is already too high, so do not accept 1e-5 as fine without thinking (course notes).

Why is backpropagation faster than finite differences?

Finite differences need a forward pass per parameter, which is a million and one passes for a million weights. Backpropagation reuses shared pieces and gets all partial derivatives from one forward and one backward pass (Nielsen).

Is backpropagation the same as automatic differentiation?

Backpropagation is reverse-mode automatic differentiation applied to neural networks. The Baydin et al. survey calls automatic differentiation similar to but more general than backpropagation.

Why does the output error term become a minus y with cross-entropy?

With a sigmoid output and the cross-entropy loss, the activation derivative cancels against the loss gradient and leaves δ_out = a_out − y. The error term then depends only on how far the output is from the target, so learning does not stall when the output neuron saturates (Nielsen, chapter 3).

Who invented backpropagation?

Nielsen notes that the algorithm was introduced in the 1970s, and that its importance was not fully appreciated until a 1986 paper. That paper is Rumelhart, Hinton and Williams in Nature, which showed hidden units coming to represent important features of the task. More questions of this kind are in the deep learning interview questions.

Sources

1. How the backpropagation algorithm works (Neural Networks and Deep Learning, chapter 2) — Determination Press (Michael Nielsen, 2015 book), 1 Dec 2019
2. Improving the way neural networks learn (Neural Networks and Deep Learning, chapter 3) — Determination Press (Michael Nielsen, 2015 book), 1 Jan 2015
3. Backpropagation, Intuitions (course notes, undated) — Stanford CS231n, 26 Oct 2026
4. Gradient Checks (Neural Networks Part 3, course notes, undated) — Stanford CS231n, 26 Oct 2026
5. Learning representations by back-propagating errors — Nature 323, 533-536 (Rumelhart, Hinton, Williams), 9 Oct 1986
6. Automatic Differentiation in Machine Learning: a Survey — Journal of Machine Learning Research 18(153) (Baydin et al.), 1 Jan 2018
7. Backpropagation section slides — Stanford CS231n, 14 Apr 2023