How autograd works in PyTorch, step by step

In this article

1. Before you start
2. Why interviewers ask this
3. Autograd from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step of the deep learning path, which explains how neural networks work.

Why interviewers ask this

A candidate who says only that PyTorch computes gradients automatically cannot explain the three things that break real training code: a .grad that is None, an error saying the graph was already freed, and a job that runs out of memory. Autograd questions separate people who use the framework from people who know what it does.

A weak answer calls autograd "numerical derivatives" or "symbolic algebra". A strong answer says it records the forward pass as a graph and replays it in reverse with the chain rule. A Staff-level answer adds the cost: why one sweep is enough, what the sweep stores, and how checkpointing trades extra compute for less memory.

Autograd from first principles

Autograd

Autograd is a reverse-mode automatic differentiation system that records the operations performed on tensors during the forward pass and replays them in reverse to compute gradients, so nobody has to write derivative formulas by hand for each model.

Computational graph

A computational graph is a directed acyclic graph (DAG) that records the operations of a calculation, with each operation as a node and the values flowing along the edges, so the calculation can later be walked backwards to differentiate it.

Leaf tensor

A leaf tensor is a tensor at the start of the graph that no recorded operation produced, such as a weight or an input, and it is the kind of tensor that keeps a .grad value when requires_grad is True.

In plain words

Imagine cooking from a recipe while you write down every step and every amount. Later the dish tastes too salty. You read your notes backwards. At each step you ask how much that step added to the saltiness. Autograd does this with numbers. The forward pass is the cooking. The graph is the notes. The backward pass is the reading in reverse.

The analogy breaks in one place. Autograd also stores a small multiplier at each step, so the blame is calculated exactly and not guessed.

In one precise sentence, autograd applies the chain rule to the operations that actually ran, from the loss back to the parameters, and returns derivative values accurate to machine precision (Baydin et al., 2018).

Step 1: The forward pass records a graph

During the forward pass, PyTorch records each operation on tensors that need gradients as a node in a computational graph. The PyTorch autograd mechanics note describes a directed acyclic graph whose leaves are the input tensors and whose roots are the output tensors. The PyTorch tutorial adds that the forward pass does two things at once: it runs the operation to get the result, and it keeps the operation's gradient function in the graph. Each result tensor points to that function through its .grad_fn attribute.

Only tensors with requires_grad set to True are tracked (PyTorch tutorial). Take y = 3 * x, z = y + 2 and loss = z ** 2 with x equal to 2 (illustrative). The graph has one leaf, x, and a chain of three operations ending at loss. You now know what exists after the forward pass: a record of operations that ends at the loss.

Step 2: Each operation saves what its backward step needs

Each operation saves the tensors its backward step needs. The autograd mechanics note gives the example that squaring a number saves the input, because the derivative of the square is twice the input and cannot be computed without it.

These saved tensors are the source of activation memory. Baydin et al. (2018) note that reverse mode has storage requirements that grow, in the worst case, in proportion to the number of operations in the function. The standard remedy is checkpointing: do not store the full record, and re-run parts of the forward computation when they are needed. The survey cites one example, Gruslys et al. (2016) for recurrent networks, that saved up to 95% of memory at a 33% increase in compute time (Baydin et al., 2018). Storing activations in 16-bit number formats also makes them smaller.

Step 3: One backward sweep gives every gradient

One backward sweep applies the chain rule from the loss to every leaf, and that single sweep produces the gradient for every leaf. An adjoint is the sensitivity of the loss to one intermediate value. The sweep starts with an adjoint of one at the loss, because the derivative of the loss with respect to itself is 1 (Baydin et al., 2018).

At each node, the incoming adjoint is multiplied by the node's local derivative, which comes from the saved tensors, and handed to the node's inputs. When one value feeds two later operations, as with a residual connection, the adjoints from both paths are added.

Training losses are a single number and the parameters are many, so one reverse sweep gives the full gradient. Forward mode is the mirror image: it carries a derivative along with each value and needs one pass per input. Baydin et al. (2018) put the cost of reverse mode at a small constant multiple of one forward evaluation, with the constant below 6 and typically 2 to 3. They call the one extra evaluation per input that numerical differentiation needs the main obstacle when inputs number in the millions or billions.

Step 4: Leaf tensors keep the gradients and the graph is freed

After the sweep, leaf tensors keep the gradients in their .grad attribute, and the graph is freed unless retain_graph=True is passed. The PyTorch tutorial says autograd accumulates the results in the .grad of each tensor, adding to what is already there, and that only leaf tensors with requires_grad set to True expose a gradient. It also says a graph can be used for gradient calculation with backward only once, for performance reasons.

The next forward pass builds a new graph from scratch. This is called define-by-run: the graph changes in every iteration, so ordinary Python if statements and loops work (Baydin et al., 2018). Define-and-run systems build the graph once before execution and keep it fixed.

1 · Forward pass2 · Saved tensors3 · Backward sweep4 · Leaf gradientsReplay

Step-by-step description

1. Each operation is recorded in a graph.
2. Operations keep what backward needs.
3. The chain rule runs from loss to leaves.
4. Leaves keep .grad; the graph is freed.

• The forward pass records each operation in a graph.
• Each operation saves the tensors its backward step needs.
• One backward sweep applies the chain rule from the loss to every leaf.
• Leaf tensors keep the gradients in their .grad attribute, and the graph is freed.

A traced example: two gradients from one sweep

Baydin et al. (2018) trace $f(x_1,x_2)=\ln x_1+x_1{x}_2-\sin x_2$ at x1 = 2 and x2 = 5, and we re-ran that trace in code.¹ The forward value is 11.6521 at x1 = 2 and x2 = 5: ln 2 is 0.6931, the product is 10, and sin 5 is −0.9589.¹

The reverse sweep seeds an adjoint of 1 at the output. One reverse sweep gives 5.5 for x1 and 1.7163 for x2.¹ For x1, the log path gives 1/2 and the product path gives 5. For x2, the product path gives 2 and the sine path gives −0.2837, which sum to 1.7163, matching the hand formula x1 − cos(x2). Finite differences agree to four decimal places but need three evaluations of the function here.¹

Text description of the diagram

• The forward value is 11.6521 at x1 = 2 and x2 = 5.
• One reverse sweep gives 5.5 for x1 and 1.7163 for x2.

The three-line PyTorch example from Step 1 works the same way. With x equal to 2, the loss is 64 and the gradient of x is 48: the adjoint at z is 2·8 = 16, and it passes through y = 3 * x multiplied by 3.¹

import torch

x = torch.tensor(2.0, requires_grad=True)
y = 3 * x
z = y + 2
loss = z ** 2
loss.backward()
print(loss.item(), x.grad.item())  # 64.0 and 48.0, checked by hand and by the tape in the data script

How the cost compares with other methods

Reverse mode costs about two function evaluations however many inputs there are. In Baydin et al. (2018), Table 4, on the Helmholtz free-energy function, gradient time relative to one evaluation of the function is, at 50 inputs, 28.63 times for numerical differentiation, 7.69 times for forward mode and 1.96 times for reverse mode. At 8 inputs the figures are 6.93, 2.66 and 2.16. Reverse mode stays near two times across 8 to 50 inputs, so its cost does not grow with the number of inputs.

Text description of the diagram

• At 50 inputs, numerical differentiation costs 28.63 times one evaluation, forward mode 7.69 times and reverse mode 1.96 times.
• Reverse mode stays near two times across 8 to 50 inputs.

The Staff-level answer

What makes it Staff-level: it names the shape condition (a scalar loss), the memory cost of saved tensors, and the dynamic graph, so each follow-up has an answer ready.

The rubric

Three variants of this question

What is the difference between autograd and backpropagation?

Backpropagation is the special case of reverse-mode automatic differentiation applied to a network's loss as a function of its weights (Baydin et al., 2018). Autograd is the general machinery, which differentiates any tensor code, not only layers.

Why not compute gradients with finite differences?

Finite differences need roughly one extra function evaluation per parameter and carry round-off and truncation errors. For a model with millions of parameters that is far too slow, while one reverse sweep costs a small multiple of the forward pass.

What happens if you call backward twice on the same loss?

By default the second call fails, because the graph was freed after the first sweep; passing retain_graph=True keeps it (PyTorch tutorial). Separately, gradients from repeated sweeps are added into .grad, not overwritten.

Common mistakes

• "Autograd approximates derivatives numerically." (wrong) It applies exact derivative rules to each operation that ran. "Autograd evaluates exact derivatives by the chain rule; it is neither finite differences nor symbolic algebra." (right)
• "PyTorch builds one static graph and reuses it." (wrong) The graph is rebuilt on every forward pass. "The graph is dynamic, so control flow and changing shapes work." (right)
• "Every tensor ends up with a .grad." (wrong) Only leaf tensors with requires_grad set to True expose it. "Gradients land on the leaves; intermediate tensors do not keep one by default." (right)
• "Backward recomputes the forward pass." (wrong) It reuses the saved tensors, which is why activations cost memory. "Backward reads saved tensors; checkpointing is the deliberate exception that recomputes." (right)
• "Reverse mode beats forward mode on every function." (wrong) It wins when outputs are far fewer than inputs. "Training has one scalar loss, so reverse mode wins there; forward mode suits few inputs and many outputs." (right)

Follow-up questions

How does autograd handle functions that are not differentiable, such as ReLU?

The PyTorch note uses the gradient where it exists. Where it does not, it picks a sub-gradient of minimum norm for a convex function or a super-gradient for a concave one, which for a ReLU kink is zero. See activation functions for the functions themselves.

When would you use forward mode instead of reverse mode?

Forward mode fits a function with few inputs and many outputs, and it is efficient for Jacobian-vector products; for a function of one input it needs a single pass (Baydin et al., 2018). In the traced example, forward mode with x1 seeded to 1 gives 5.5 in one pass but needs a second pass for x2.

Why does training use so much more memory than inference?

Training keeps the saved tensors of every recorded operation until the backward pass uses them, while inference can discard each activation after use. In PyTorch you get that saving by running inference under torch.no_grad() (or torch.inference_mode()), which stops autograd from recording the graph and its saved tensors (PyTorch tutorial); without it, a forward pass on parameters with requires_grad set to True still records them. Checkpointing and smaller number formats are the usual ways to reduce it.

Can automatic differentiation return a wrong or unstable gradient?

Yes. The survey warns that it is not immune to floating-point problems, and it differentiates the procedure that was coded, not the ideal function the code approximates. Gradients that grow without bound are a separate training problem, covered under exploding gradients.

What does the optimizer do with the gradients autograd computes?

The optimizer reads each parameter's .grad and updates the parameter, and a training loop clears the gradients between steps with optimizer.zero_grad(), because they accumulate. The extra numbers an optimizer keeps per parameter are the optimizer states. More practice questions are in deep learning interview questions.

Sources

1. Autograd mechanics (PyTorch documentation, developer notes) — PyTorch documentation (live page, undated; accessed 2026-10-26), 26 Oct 2026
2. Automatic Differentiation in Machine Learning: a Survey — Journal of Machine Learning Research 18 (Baydin, Pearlmutter, Radul, Siskind), 1 Apr 2018
3. Automatic Differentiation with torch.autograd (PyTorch Tutorials) — PyTorch Tutorials (undated; accessed 2026-10-26), 26 Oct 2026
4. A Gentle Introduction to torch.autograd (PyTorch Tutorials) — PyTorch Tutorials (undated; accessed 2026-10-26), 26 Oct 2026
5. Overview of PyTorch Autograd Engine — PyTorch blog (Preferred Networks), 8 Jun 2021