How do we decide which direction to take the step in?

In this article

1. Before you start
2. Why interviewers ask this
3. Which direction to step, from first principles
4. The Staff-level answer
5. The rubric
6. Three variants of this question
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page answers one question about how machine learning algorithms learn. It needs only school-level calculus; probability and statistics is useful background for gradients estimated from samples.

Why interviewers ask this

A candidate who says the step goes along the gradient has the sign wrong, and an interviewer hears that error immediately. The question separates three levels of answer. A weak answer repeats "follow the gradient". A strong answer says the gradient points uphill, so the step goes the other way, and justifies it with the dot-product argument. A Staff-level answer adds that the direction is only locally best, that the step length is a separate decision, and that a different measure of distance gives a different direction. Candidates report this question in machine learning interviews; no public source names a company that asks it, so treat it as a common fundamentals check rather than a known question.

Which direction to step, from first principles

Descent direction

A descent direction is a direction at a point whose dot product with the gradient is negative, so that a small enough move that way lowers the function. Many directions qualify. The negative gradient is only the steepest one under ordinary distance.

Newton step

A Newton step is a move of minus the inverse of the Hessian, the matrix of second derivatives, times the gradient. It minimizes the quadratic approximation of the function at the current point, so it bends the direction using curvature. It costs one linear solve per step.

Zig-zagging

Zig-zagging is a pattern in which successive gradient steps alternate between the two sides of a narrow valley, so the iterates advance along the valley slowly although every step goes downhill. It appears when the function is stretched much more in one direction than in another.

In plain words

Picture a hiker in thick fog who wants the valley floor. The hiker cannot see the valley. They can only feel the slope under their feet. So they turn to face the steepest way up, then walk the opposite way. The analogy breaks in two places. A real function has many inputs, so "downhill" is a direction in a space with one axis per input. And the hiker feels only the slope here, not the shape of the land ahead.

In one precise sentence, at each point we pick a direction whose dot product with the gradient is negative, and the most negative one for ordinary distance is minus the gradient.

Step 1: A direction goes downhill when its dot product with the gradient is negative

A direction Δx is a descent direction at a point when its dot product with the gradient is negative. Boyd and Vandenberghe's lecture slides state this test and split every iteration into two separate decisions: a direction Δx and a step size t, with the update x ← x + tΔx.

One variable shows the test. For f(x) = ½x² the derivative is f'(x) = x, so at a negative x the slope is negative and moving right lowers f; at a positive x moving left does (Goodfellow, Bengio and Courville, 2016, section 4.3). Move opposite the sign of the derivative. The learning rate ε is a positive number that sets the length of the step. From the illustrative start x = 2 with ε = 0.5, the update x − εx gives 1, 0.5 and 0.25, and f falls from 2 to 0.5, 0.125 and 0.03125.¹ At f'(x) = 0 the derivative gives no direction. The squared-error loss in linear regression is a bowl of this kind in many variables.

Step 2: Minus the gradient is the steepest descent direction

The steepest descent direction is minus the gradient. The slope of f along a unit vector u (a direction of length one) is the dot product of u with the gradient, which equals the gradient's length times cos θ, where θ is the angle between u and the gradient. That slope is smallest when θ is 180 degrees, so u points opposite the gradient (Goodfellow et al., 2016). The gradient points straight uphill, its negative straight downhill, and the method that follows it is called steepest descent, or gradient descent, with update x' = x − ε∇f(x), where ∇f, the gradient, is the list of slopes.

Take the illustrative bowl f(x1, x2) = ½(x1² + 10x2²) at the point (10, 1). The gradient there is (10, 10), so the step direction is (−10, −10), and their dot product is −200, below zero as the test requires. The slope along the unit vector of that direction is −14.14, along (−1, 0) it is −10, and along a direction at right angles to the gradient it is 0.¹

Step 3: Steepest means locally steepest, and a stretched bowl makes it zig-zag

Steepest descent picks the best direction at the current point, not the quickest route to the minimum. Boyd and Vandenberghe analyze f(x) = ½(x1² + γx2²) with an exact line search, which picks the step length that lowers f the most along the chosen direction. Starting from (γ, 1), each step shrinks the position by the factor (γ−1)/(γ+1), and they call convergence very slow when γ is much larger or much smaller than one. For γ = 10 the factor is 9/11, about 0.82, and x2 changes sign at every step.¹

The function value falls from 55 to 7.39 in five gradient steps, and it takes 28 steps to push f below 0.001.¹ Every step goes downhill, yet the path crosses the valley back and forth, because the direction uses only the slope at the current point. A Brigham Young University lab makes the same practical remark: the method is theoretically foundational but often slow to converge.

Step 4: A different norm or curvature bends the direction

"Steepest" depends on how distance is measured, so a different norm gives a different direction. A norm is a rule for measuring the length of a vector. Boyd and Vandenberghe show that for ordinary Euclidean length the steepest direction is −∇f, while for a quadratic norm, one whose length is set by a matrix P, it is −P⁻¹∇f. Choosing P as the Hessian gives the Newton step −H⁻¹∇f.

On the bowl above the Hessian is diagonal with entries 1 and 10, so at (10, 1) the Newton step is (−10, −1) and lands at the minimum (0, 0) in one step, where f is 0. It points 39.3 degrees away from the negative gradient yet still has a negative dot product with the gradient, −110.¹

The price is a linear solve per step, that is, solving a system of n linear equations; Boyd and Vandenberghe's slides put the Cholesky solve at (1/3)n³ floating-point operations for a dense n-variable system. In their two-variable example Newton's method converges in only 5 steps. The direction depends on the geometry you assume, which is why the gradient is the default and not the only answer.

1 · Descent test2 · Minus the gradient3 · Zig-zag4 · CurvatureReplay

Step-by-step description

1. A negative dot product with the gradient goes downhill.
2. It has the most negative slope of the three directions.
3. Steepest is local, so x2 flips sign at every step.
4. The Newton step lands on the minimum in one step.

• The dot product with the gradient is -200, below zero.
• The slope along the unit vector of that direction is -14.14, along (-1, 0) it is -10, and at right angles it is 0.
• The x2 coordinate changes sign at every step.
• The Newton step lands at the minimum (0, 0) in one step.

Text description of the diagram

• The function value falls from 55 to 7.39 in five gradient steps.
• One Newton step lands at the minimum, where the function value is 0.

The Staff-level answer

What makes this Staff-level is the descent test, the point that steepest is local and depends on the norm, and the separate step length.

The rubric

Three variants of this question

Why do we move against the gradient and not along it?

The gradient points toward the fastest increase, so to minimize a function we go the other way; to maximize, we go along it, which is the same as minimizing minus the function.

What if the derivative is zero?

At a zero derivative the slope gives no direction (Goodfellow et al., 2016), and the point is a critical point: a local minimum, a local maximum or a saddle point, so a zero gradient alone does not show a minimum, let alone a global one.

Is the negative gradient always the best direction to use?

No. It is the steepest direction for ordinary distance, but a different norm, Newton's curvature-based step, or the conjugate gradient method, which the Brigham Young University lab says converges much faster in some situations, can reach the minimum in fewer steps.

Common mistakes

• "The gradient is the direction to go." (wrong) The gradient points uphill. (right) "I go along the negative gradient, because the gradient is the direction of steepest ascent."
• "Steepest descent is the quickest route to the minimum." (wrong) It is only locally steepest and zig-zags on stretched bowls. (right) "It is the best direction at this point; over many steps curvature-aware methods can be far faster."
• "If the direction is right, the function must go down." (wrong) The step length matters as much as the direction (Meza, Lawrence Berkeley National Laboratory). From x = 2 with an illustrative ε = 2.5 the points are −3 and then 4.5, and f rises from 2 to 4.5 and then 10.125.¹ (right) "A correct direction lowers f only for a small enough step."
• "A zero gradient means I found the minimum." (wrong) It means a critical point. (right) "It may be a local minimum, a local maximum or a saddle point, so I check the curvature (the Hessian's eigenvalues)."
• "The gradient tells me how far to step." (wrong) It gives only the direction. (right) "The length comes from the learning rate, a line search or curvature; at the start of the example the exact step length is 0.1818."¹

Follow-up questions

How do you choose the step size once you have the direction?

Choose a small constant learning rate, or a line search. Boyd and Vandenberghe describe backtracking, which shrinks a trial step until f drops enough. Meza notes that Barzilai and Borwein proposed two new step sizes for the negative gradient direction in 1988.

Why does gradient descent zig-zag, and how do you fix it?

It zig-zags when the function is much more stretched in one direction than another, because the slope at each point points across the valley. Curvature-aware steps such as Newton's, conjugate gradient methods or scaling the inputs all reduce it.

What changes when the gradient is only estimated from a sample of the data?

The direction becomes a noisy version of the true one and is right on average rather than exactly. Losses such as cross-entropy loss are averages over examples, which is why this estimate is natural.

How does a penalty term change the direction of the step?

An L2 penalty adds a term proportional to the weights to the gradient, which pulls each step toward zero. An L1 penalty has no single slope at zero, so subgradient methods or a proximal (soft-thresholding) update are used; see L1 vs L2 regularization.

Sources

1. Deep Learning, Chapter 4: Numerical Computation, section 4.3 Gradient-Based Optimization — MIT Press (Goodfellow, Bengio, Courville), 1 Jan 2016
2. Convex Optimization, Chapter 9: Unconstrained minimization (lecture slides) — Stanford University (Boyd and Vandenberghe), 1 Jan 2004
3. Gradient descent (article) — Khan Academy, 26 Oct 2026
4. Steepest Descent (Juan C. Meza) — WIREs Computational Statistics (2010), Lawrence Berkeley National Laboratory, via eScholarship, 1 Jan 2010
5. Gradient Descent Methods lab — Brigham Young University, 26 Oct 2026