What is gradient descent in machine learning?

In this article

1. Before you start
2. Why interviewers ask this
3. Gradient descent from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

Why interviewers ask this

A candidate who can only say "it goes downhill" cannot explain why a training run diverged or stalled, and that is the situation interviewers are probing. Gradient descent comes up in classical machine learning screens as "explain how it works", "what happens if the learning rate is too high", "compare stochastic gradient descent (SGD) and Adam", and "can it reach the global optimum". These questions often arrive as a ladder, each rung asking for the next level of detail.

A weak answer names the algorithm. A strong answer gives the update rule, says what each knob does to the step, and knows what changes when the loss surface is not a clean bowl.

Gradient descent from first principles

In plain words

Gradient

A gradient is a vector that holds the partial derivative of the loss with respect to every weight, and it points in the direction in which the loss rises fastest. Its length says how steep the surface is, so a long gradient means a steep slope and a short one means nearly flat ground.

Gradient descent

Gradient descent is an iterative optimization method that lowers a loss by repeatedly subtracting the gradient, scaled by a small number, from the weights. It needs only first derivatives, which backpropagation supplies cheaply.

Mini-batch

A mini-batch is a small random subset of the training examples, typically tens to thousands, used to estimate the gradient for one update instead of using the whole data set.

Picture a hiker on a foggy hillside who wants the valley floor. She cannot see it, but she can feel which way the ground tilts under her feet. She takes a step downhill, feels again, and repeats. The steeper the slope, the bigger the natural step.

The analogy breaks in one place. A model has millions or billions of weights, so the hillside has that many directions, and the hiker feels all of them at once through the gradient.

In one precise sentence, gradient descent updates every weight by subtracting the learning rate times the loss gradient.

Step 1: The gradient points uphill, so step the other way

The basic update is $w\leftarrow w-\eta \mathrm{\nabla }L(w)$, where $w$ holds the weights, $\eta$ is the learning rate and $\mathrm{\nabla }L$ is the gradient of the loss. The PyTorch 2.14 SGD reference documents exactly this update, with a default learning rate of 0.001 and a default momentum of 0, which reduces the method to plain gradient descent.

Take the one-weight loss $L(w)=(w-3{)}^2$, whose gradient is $g(w)=2(w-3)$ and whose best weight is 3. Starting at $w=0$ with $\eta =0.1$, the gradient is $-6$, so the new weight is 0.6. After ten such steps the weight is 2.678¹, closer to 3 on every step but never quite there.

The input is the current weights and a gradient. The output is new weights, slightly lower on the loss surface. In a neural network the loss is computed through layers of activation functions, and backpropagation supplies the gradient.

Step 2: The learning rate sets how far each step goes

The learning rate decides whether training converges, crawls or blows up, because for a bowl-shaped loss the distance to the answer is multiplied by a fixed factor each step. For the loss above the factor is $(1-2\eta )$. A factor smaller than 1 in size shrinks the error, a factor near 0 is fastest, and a factor beyond 1 in size grows it.

That gives the table in the worked example: 0.1 crawls (factor 0.8), 0.5 lands on the answer in one step (factor 0), 0.9 overshoots and bounces back and forth while shrinking (factor −0.8), 1.0 bounces forever between 0 and 6, and 1.1 runs away to −15.575 after ten steps¹. The same pattern holds in a real model, where the limit on stable learning rates is set by the steepest direction of the loss surface.

The input is the gradient and the learning rate. The output is a step whose size, not only whose direction, now matters.

Step 3: A mini-batch estimates the gradient at a fraction of the cost

Mini-batch gradient descent replaces the exact gradient over all examples with an estimate from a small random subset, which makes each update cheap at the price of some noise. Ruder's overview of gradient descent optimization algorithms (2016, revised 2017) names the three variants: full-batch uses all examples per update, stochastic gradient descent uses one, and mini-batch sits between them.

On a synthetic line-fitting problem with 1,000 points, five passes over the data gave 5 updates with full batches, 160 with batches of 32 and 5,000 with a batch of one. The full-batch run was still far from the answer, with a mean squared error of 2.054, while batches of 32 reached 0.261 against a noise floor of about 0.25¹. More updates of lower quality beat few updates of perfect quality, until the noise itself becomes the limit.

The input is a random subset of the data. The output is a noisy but unbiased gradient estimate.

Step 4: Momentum and Adam fix the narrow valley

Momentum and Adam change the step so that the flat direction of a narrow, tilted valley is not stuck with the small rate that the steep direction forces. Momentum keeps a running velocity, the weighted sum of past gradients, and steps along it, so sideways bouncing cancels while consistent downhill movement builds up: on a steady slope the step grows to the learning rate divided by one minus the momentum factor, ten times the rate at a factor of 0.9. Plain gradient descent has no such memory: each step uses only the current gradient. PyTorch 2.14 documents the velocity as the previous velocity times the momentum factor plus the new gradient.

Adam rescales every weight

Adam adds a per-weight scale. Kingma and Ba (2014, v9 2017) keep a running average of the gradient and of its square, correct both for their start at zero, and update by $\alpha {\hat{m}}_t/(\sqrt{{\hat{v}}_t}+\epsilon )$, with suggested defaults of 0.001, 0.9, 0.999 and $\epsilon ={10}^{-8}$. At the very first step the corrected average equals the gradient and the corrected square equals its square, so the step is about $\alpha$ times the sign of the gradient. Gradients of 0.001, 1 and 1,000 all produce a first step of 0.001¹. Adam's step size does not depend on the gradient's scale.

1 · Step against gradient2 · Set the rate3 · Sample a batch4 · Smooth and rescaleReplay

Step-by-step description

1. The gradient points uphill, so the update subtracts it.
2. The learning rate scales every step and decides convergence.
3. A mini-batch gives a cheap, noisy gradient estimate.
4. Momentum averages steps; Adam scales each weight.

• The update subtracts the learning rate times the gradient from the weights.
• A learning rate of 1.1 sends the weight to 15.575 below zero after ten steps.
• Batches of 32 reach a mean squared error of 0.261 in 160 updates.
• Adam takes a first step of about 0.001 whatever the gradient size.

Worked example

The loss is $L=0.5(100x^2+y^2)$, started at $(1,1)$. Its curvature is 100 times larger along $x$ than along $y$, so its condition number, the ratio of the steepest to the flattest curvature, is 100, and the learning rate is capped by the steep direction while the flat direction crawls. The cap is 0.02. The plain method uses 0.019, safely under it, so the steep direction settles quickly while the flat direction shrinks by only 1.9% a step, and the run needs 343 steps¹.

Pushing the rate to 0.0199, closer to the cap, is slower, not faster: the steep direction then flips sign every step and shrinks by only 1% a step, and the count rises to 882¹.

Text description of the diagram

• Plain gradient descent needs 343 steps, momentum 138 and Adam 82.
• The bowl has a condition number of 100.

Text description of the diagram

• After ten steps the weight is 0.322 from the best value at a learning rate of 0.1 and exactly on it at 0.5.
• A learning rate of 1.1 ends 18.575 away from the best weight after ten steps.

Anyone can redo the learning-rate table with a calculator: each step multiplies the distance from 3 by the error factor, so ten steps multiply the starting distance of 3 by the factor to the tenth power. The saved script is the reference for the rest.

Trade-offs and edge cases

Gradient descent is certain to reach the global minimum only for convex losses, and neural network losses are not convex. A convex function is a function whose graph never rises above the straight line drawn between any two of its points, so it has no separate local valleys. For such a loss a small enough learning rate converges, as the quadratic above shows for any rate below 1. For a neural network there is no such certainty, and the practical question is whether the minimum found generalizes.

The main trade-offs, with the reasons:

• Batch size. Large batches give smoother gradients and use hardware well; small batches add noise. The right size depends on the data set and hardware.
• Adam versus SGD with momentum. Adam needs less learning-rate tuning because its step does not depend on the gradient scale; plain SGD with momentum is simpler and is still common where it is tuned carefully. Reported practice differs, and Ruder's overview describes both.
• Weight decay. In Adam, adding an L2 penalty to the loss is not the same as shrinking the weights directly, because the penalty's gradient gets divided by Adam's scale. Loshchilov and Hutter (2017, v3 2019) show L2 regularization and weight decay are equivalent for SGD but not for adaptive methods, and decoupling the decay, which is AdamW, restores the intended behavior.
• Learning-rate schedules. A large rate early and a smaller one late is common. A warm-up, a short ramp from a tiny rate, is often added for large models such as the transformers that interviewers ask about.

How to answer it in an interview

The spoken structure is: say the update, name the learning rate, explain the batch noise, add momentum and Adam, then state the honest limit on certainty.

Common mistakes

• "Gradient descent always finds the global minimum" (wrong). It does for convex losses with a suitable step; neural network losses are not convex (right: say there is no general certainty).
• "A bigger learning rate just trains faster" (wrong). Past the stability limit the error grows each step, as in the table, where 1.1 ends at −15.575 (right: tune the rate against the steepest direction).
• "SGD means any mini-batch" (wrong). Technically, stochastic gradient descent uses one example, although many libraries use the name for mini-batches (right: say which batch size you mean).
• "Adam is just SGD with momentum" (wrong). Adam also rescales every weight by a running gradient scale (right: name both the momentum average and the squared-gradient average).
• "L2 regularization and weight decay are the same in Adam" (wrong). They are the same only for plain SGD (right: use decoupled decay, AdamW).

Follow-up questions

What is the difference between Adam and AdamW?

Adam with an L2 penalty folds the penalty into the gradient, where Adam's scaling shrinks its effect unevenly; AdamW applies the weight decay directly to the weights, separate from the gradient step. Loshchilov and Hutter report that this decoupling improves Adam's generalization.

What is stochastic gradient descent?

Stochastic gradient descent updates the weights using the gradient from a single random example, so each step is very cheap and very noisy. In practice the term is also used for mini-batch training with a batch size above one, so it is worth stating the size.

What is mini-batch gradient descent?

Mini-batch gradient descent estimates the gradient from a small random subset of examples, typically tens to thousands, and updates after each subset. It balances the noise of single-example updates against the cost of full-batch updates, and it maps well onto hardware that multiplies matrices.

Does gradient descent always reach the global optimum?

Only for convex losses, where every local minimum is global and a small enough step converges. For non-convex losses, such as those of deep learning models, it converges to a point where the gradient is near zero, which may be a poor local minimum or a saddle region.

How should the learning rate change during training?

Usually it starts with a short warm-up, stays high while the loss falls quickly, and then decays so that late updates do not bounce around the minimum. The shape of the schedule is a tuned choice, and the machine learning operations (MLOps) side of a project should log it with every run.

Sources

1. Adam: A Method for Stochastic Optimization (Kingma and Ba), arXiv 1412.6980 v9 — arXiv; ICLR 2015 (v1 submitted 2014-12-22), 30 Jan 2017
2. Decoupled Weight Decay Regularization (Loshchilov and Hutter), arXiv 1711.05101 v3 — arXiv; ICLR 2019 (v1 submitted 2017-11-14), 4 Jan 2019
3. torch.optim.SGD, PyTorch 2.14 documentation — PyTorch (page shows version only; read 2026-10-24), 24 Oct 2026
4. An overview of gradient descent optimization algorithms (Ruder), arXiv 1609.04747 v2 — arXiv (v1 submitted 2016-09-15), 15 Jun 2017