Linear and logistic regression from scratch by gradient descent, step by step

In this article

1. Before you start
2. Why interviewers ask this
3. Logistic and linear regression from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

The machine learning (ML) coding round applies the same write-it-from-scratch skill to other models.

Why interviewers ask this

Writing a regression by hand is a common short coding task. Candidates report being asked to implement logistic regression in NumPy, to build linear regression in core Python or PyTorch, to add stochastic gradient descent (SGD), an L2 penalty and early stopping, and to write ordinary least squares (OLS) in matrix form. These are candidate reports, not a published list from any company.

The task is short but it separates people. A weak answer pastes a library call or forgets the bias term. A strong answer writes the loop, names the loss, checks the shapes and explains why the gradient looks the way it does. A Staff-level answer also says when the loop fails: unscaled features, separable data, a learning rate that is too large, or an unregularized model that is compared with a regularized library default.

Logistic and linear regression from first principles

Linear regression

Linear regression is a model that predicts a continuous number as a weighted sum of input features plus a bias, with weights chosen to make the squared gap between predictions and true values small.

Logistic regression

Logistic regression is a classification model that passes the same weighted sum through the sigmoid function to output the probability that the label is one, with weights chosen to maximize the likelihood of the observed labels.

In plain words

Imagine pricing a flat. You charge so much per square meter, plus a fixed fee. The prices per unit are the weights. The fixed fee is the bias. The quote is the weighted sum. Linear regression picks the prices that bring its quotes closest to real sale prices. The analogy breaks for logistic regression. Its answer is not a price but a probability between zero and one.

In precise terms, both models compute z = Xw, with a column of ones in X for the bias. Linear regression predicts z. Logistic regression predicts the sigmoid of z. Gradient descent adjusts w to reduce the loss.

Step 1: A weighted sum scores every row

Put the features in a matrix X with one row per example. Add a column of ones on the left so the bias becomes just another weight. The score of every row is then one matrix product, z = Xw.

Input: X of shape n by d plus one column, and w of matching length. Output: n scores. Both models start here, and nothing else about the model is decided yet. The Stanford course notes call a problem with a continuous target a regression problem, and one with a small set of discrete values a classification problem. The scikit-learn guide names the weights coef_ and the bias intercept_. You now know the shared body; the next step decides what the score means.

Step 2: The output rule and the loss say what the score means

Linear regression uses the score as the prediction and measures mean squared error (MSE), the average squared gap to the true value. The Stanford notes state that this cost is a convex quadratic with one global optimum. In code we minimize half the MSE, (1/2n)·Σ(z − y)², because the ½ cancels the 2 from differentiating the square; this only rescales the learning rate.

Logistic regression needs a probability, so it applies the sigmoid, a function that maps any number to a value between zero and one by computing one divided by one plus e to the power minus z. Its loss is log loss. The same notes note that fitting it maximizes the log-likelihood, hence gradient ascent with a positive sign; minimizing log loss is the same thing with a minus sign.

Input: scores z. Output: predictions and one loss number. The scikit-learn guide describes logistic regression as a generalized linear model (GLM) with a Bernoulli distribution and a logit link. Squared error on a 0/1 label is the naive route the notes set up and then reject.

Step 3: The gradient has the same shape for both models

The gradient, the list of slopes of the loss with respect to each weight, is g = Xᵀ(pred − y) / n for both models. This shared formula is the gradient of half the MSE for linear regression and of log loss for logistic regression. For linear regression pred is z. For logistic regression pred is the sigmoid of z. That one substitution is the single line that differs between the two training loops. The gradient is the transposed feature matrix times prediction minus label, divided by n.

Input: X, predictions, labels. Output: one slope per weight. The shapes check out: Xᵀ is (d+1) by n, because of the bias column, and the error is a list of n, so g has the same length as w. The log loss and the sigmoid cancel so cleanly that the logistic gradient has no extra factor, which is why the formula is easy to misremember as having a sigmoid derivative inside it. You now have everything needed to update the weights.

Step 4: The update moves the weights against the gradient

The update is w ← w − lr · g, where the learning rate lr is a small positive number that sets how far each step moves. The weights are updated by subtracting the learning rate times the gradient. Repeat steps 1 to 3 and the update until the loss stops falling. The Stanford notes call using every example on every step batch gradient descent, and say that for linear regression it converges to the global minimum if the learning rate is not too large.

Input: weights and gradient. Output: new weights. Here is the whole thing in NumPy, with a flag for the one difference.

import numpy as np

def sigmoid(z):
    z = np.clip(z, -30, 30)          # avoid overflow in exp
    return 1.0 / (1.0 + np.exp(-z))

def fit(X, y, kind, lr, iters, lam=0.0):
    n, d = X.shape
    Xb = np.column_stack([np.ones(n), X])   # bias column
    w = np.zeros(d + 1)
    for _ in range(iters):
        z = Xb @ w
        pred = z if kind == "linear" else sigmoid(z)
        g = Xb.T @ (pred - y) / n
        g[1:] += lam * w[1:]                # L2 on weights, not the bias
        w -= lr * g
    return w

1 · Weighted sum2 · Output rule3 · Gradient4 · UpdateReplay

Step-by-step description

1. Score every row with one matrix product.
2. Keep the score, or apply the sigmoid.
3. Same line for both models.
4. Step against the gradient and repeat.

• The score of every row is the product of the feature matrix and the weights.
• The gradient is the transposed feature matrix times prediction minus label, divided by n.
• The weights are updated by subtracting the learning rate times the gradient.

Worked example

The data are illustrative, not measurements.¹ For linear regression, x takes the values 1 to 5 and y is 2.1, 3.9, 6.2, 7.8 and 10.1. Start at w = (0, 0) with a learning rate of 0.05 and 500 iterations (illustrative choices). All first predictions are zero, so the errors are minus y. The gradient is (−6.02, −22.04), and the first update gives (0.301, 1.102).¹

The normal equation, the closed-form answer found by setting the gradient to zero, gives an intercept of 0.05 and a slope of 1.99. After 500 iterations gradient descent reaches 0.0567 and 1.9882, close but not identical, and the line predicts 11.99 at x = 6.¹

Most of the drop happens in the first five updates. The costs above come from the same run.¹ The cost is 0.0317 after five updates and 0.0107 after 500 updates. The cost keeps creeping down after the first five updates, so the chart below starts at update 5, where the bars are still visible.

Text description of the diagram

• The cost is 0.0317 after five updates and 0.0107 after 500 updates.
• The cost keeps creeping down after the first five updates.

For logistic regression, x takes the values 1 to 6 and the labels are 0, 0, 1, 0, 1, 1 (illustrative; one label breaks any clean split). At zero weights every probability is 0.5, so the loss is ln 2, which is 0.6931. The first gradient is (0, −0.5833) and, with a learning rate of 0.5 (illustrative), the slope becomes 0.2917.¹ The loss then falls to about 0.579 after 10 updates, 0.4294 after 100 and 0.413 after 1,000, where it stops moving. The weights settle at an intercept of −4.2491 and a slope of 1.214. The decision boundary, where the probability is 0.5, sits at x = 3.5, and accuracy on this toy set is 4 of 6.¹

These values repeat the runs above.¹

Why not just fit a line to the labels? The OLS line on the same labels has a value of −0.2 at x = 0 and 1.4 at x = 8, so it outputs "probabilities" outside zero to one.¹

Trade-offs and edge cases

Regularization is a penalty added to the loss that discourages large weights. An L2 penalty adds lam times the weight to the gradient, as in the code above, and leaves the bias alone. On the toy labels, a penalty strength of 0.5 (illustrative) shrinks the slope from 1.214 to 0.5022, a ratio of 0.4137.¹

Text description of the diagram

• Linear regression predicts a number and logistic regression predicts a probability.
• Linear regression has a closed form, so it can be solved in one linear-algebra step.
• Logistic regression has no closed form, so it is fitted by iteration.

Linear regression predicts a number and logistic regression predicts a probability. Linear regression also has a closed form, so it can be solved in one linear-algebra step when XᵀX is invertible; with collinear features XᵀX is singular, so use a least-squares solver or add a penalty. Logistic regression has no closed form, so it is fitted by iteration.

Edge cases that break a from-scratch version:

• Library mismatch. The scikit-learn undefined reference states that regularization is applied by default. Its user guide adds that no regularization amounts to a very high C, so a larger C means weaker regularization. In the 1.7 reference the direct way to turn it off is penalty=None.
• Unscaled features. Features on very different scales make one learning rate too large for one weight and too small for another; standardize them first.
• Perfectly separable classes. The unregularized weights grow without limit, because the loss keeps falling as the weights grow. A penalty or early stopping stops that.
• Overflow. The sigmoid computes exp(-z), which overflows when z is a large negative score, so clip z (as the code does) or use the trick in numerically stable softmax.
• Data leakage. Fit scalers and thresholds on the training split only.

How to answer it in an interview

The spoken structure: name the shared sum, say what differs, give the gradient, then name scaling, regularization and early stopping.

Common mistakes

• "I fit a line to the 0/1 labels and threshold it." (wrong) Predictions leave the range zero to one and outliers drag the line. (right) "I use a sigmoid and log loss so outputs are probabilities."
• "Gradient descent on linear regression can get stuck in a local minimum." (wrong) The cost is convex with one global optimum. (right) "It converges if the learning rate is not too large."
• "Logistic regression minimizes squared error." (wrong) It maximizes the Bernoulli likelihood, which equals minimizing log loss. (right) "I minimize log loss."
• "My weights differ from scikit-learn, so my code is wrong." (wrong) The library regularizes by default. (right) "I match the penalty first, then compare."
• "I forgot the bias column." (wrong) The line is forced through the origin. (right) "I add a column of ones and do not penalize it."

Follow-up questions

How do you implement linear regression from scratch using only core Python or PyTorch?

Use the same four lines on lists or tensors: score, error, gradient, update. In PyTorch you can write the gradient by hand as above or let autograd compute it; check that both give the same weights. The same by-hand style appears in building generative pre-trained transformer (GPT) from scratch.

How do you write a function that trains a logistic regression and evaluates it?

Split the data first, fit scaling on the training part only, run fit, then report log loss and accuracy on the held-out part; evaluation metrics from scratch shows how to compute more of them. Return the weights, the scaler and the metrics together so the caller can reuse them.

How do you implement logistic regression with SGD, L2 regularization and early stopping in NumPy?

Stochastic gradient descent (SGD) updates after one example or a small batch, which the Stanford notes derive for a single example. Shuffle each epoch, apply the L2 term to the weights only, track validation loss after each epoch and keep the best weights when it stops improving.

How do you write ordinary least squares in matrix form?

Set the gradient of the squared error to zero to get the normal equation, w = (XᵀX)⁻¹Xᵀy, and solve it with a linear solver rather than an explicit inverse. The worked example shows it landing within a hair of gradient descent.

How do you train a linear model on very high-dimensional sparse features?

Keep X in a sparse format so the products touch only non-zero entries; the scikit-learn reference says its logistic regression handles dense and sparse input. An L1 penalty drives many weights to zero, and its docs describe sparsify() as a way to store such weights compactly. For storage and training at scale, feature hashing maps many raw features into a fixed-size vector and online SGD updates from one example at a time, so the data need not fit in memory; scikit-learn documents FeatureHasher and SGDClassifier for this.

Why does your from-scratch result differ from scikit-learn?

Almost always because of the penalty and the solver. Turn off regularization on one side or match the objective on the other, then compare. The same checking habit helps in a code review, and the gradient derivation behind the match is worked in multi-layer perceptron (MLP) forward and backward by hand.

Sources

1. CS229 Lecture Notes (Tengyu Ma and Andrew Ng) — Stanford University, 23 Aug 2026
2. CS229 notes 1: linear regression, classification and logistic regression (Andrew Ng) — Stanford Engineering Everywhere, 26 Oct 2026
3. scikit-learn User Guide: 1.1 Linear Models — scikit-learn, 26 Oct 2026
4. sklearn.linear_model.LogisticRegression (v1.7) — scikit-learn, 26 Oct 2026