Logistic regression explained: how it works and how it is trained

In this article

1. Before you start
2. Why interviewers ask this
3. How logistic regression works from first principles
4. Worked example
5. Trade-offs and edge cases
6. How to answer it in an interview
7. Common mistakes
8. Follow-up questions
9. Sources

Before you start

This page is one step in the machine learning algorithms learning path.

Why interviewers ask this

A candidate who names logistic regression is expected to defend every part of it, because it is the smallest model that holds the whole supervised-learning loop: a score, a loss, a gradient and a decision. Jurafsky and Martin (draft chapter, accessed 2026-10-26) list four ingredients, a feature representation, a classification function, an objective function and an optimiser, and note that a neural network can be viewed as logistic regression classifiers stacked on top of each other.

Candidates report questions on why the model uses a sigmoid, why the loss is cross-entropy, how to derive that loss from a Bernoulli model, and what to do when the area under the receiver operating characteristic (ROC) curve (AUC) is high but the probabilities are miscalibrated.

A weak answer says "it fits an S-curve to separate classes." A strong answer states the model in two lines, writes the loss and its gradient, and explains why squared error is the wrong loss. A Staff-level answer adds what the model assumes, how it fails and how to repair each failure.

How logistic regression works from first principles

Log-odds

The log-odds of an event with probability p is the natural logarithm of p divided by one minus p, a number that can take any real value. In logistic regression the weighted sum of the features, called z, is the log-odds of the positive class.

Sigmoid function

The sigmoid function is an S-shaped curve that maps any real number to a value strictly between zero and one and equals one half at zero. It is the inverse of the logit (log-odds) function, so it turns a linear score into a valid probability.

In plain words

Picture a lender scoring a loan. Income adds points. Missed payments take points away. The points add up to one total. A dimmer switch then turns that total into a likelihood of repayment. A high total gives a likelihood near one. A low total gives one near zero. A total of zero gives one half.

The analogy breaks in two places. The points are learned from labelled examples, not set by hand. And the dimmer is curved: one more point changes the likelihood most near zero and barely at all far from zero.

Despite the name, the model outputs a probability between zero and one rather than an unbounded real number, unlike linear regression. In one precise sentence: logistic regression learns weights w and a bias b so that the sigmoid of w·x + b is close to one for positive examples and close to zero for negative ones.

Step 1: A weighted sum scores each example as log-odds

Logistic regression first computes a weighted sum of the features, $z=b+{\sum }_j{w}_j{x}_j$, and this number z is called the log-odds of the positive class. Odds are the probability of an event divided by the probability that it does not happen, so a probability of 0.8 is odds of 4 and a log-odds of 1.3863.¹

The log-odds can be any real number: negative when the event is unlikely, zero when it is a coin flip, positive when it is likely. Input: the features and the current weights. Output: one number z.

The central modelling assumption is that the log-odds are linear in the features; the model also assumes each label is a Bernoulli draw given x and that examples are independent. The probability itself is not assumed linear. Linear regression would stop here and report z; logistic regression does one more step. You now have a score, and Step 2 turns it into a probability.

Step 2: The sigmoid function turns the score into a probability

The sigmoid function maps the log-odds z to a probability p between zero and one, using the formula in Google's crash course:

A score of 0 gives p = 0.5, a score of 2 gives 0.881, a score of -2 gives 0.119 and a score of 4 gives 0.982.¹ The curve is steepest near zero and flattens towards both ends.

The sigmoid is not an arbitrary squashing curve. Solving z = ln(p / (1 - p)) for p gives exactly the formula above, so "the log-odds are linear in the features" and "the probability is the sigmoid of a linear score" are one statement. The curve is also smooth, so its derivative exists everywhere, which gradient descent needs. Input: z. Output: p. Step 3 measures how wrong p is.

Step 3: Cross-entropy loss gives a confident mistake a large gradient

Log loss, the loss logistic regression minimises, scores one prediction as the negative log of the probability the model gave to the true label. This loss is called log loss or binary cross-entropy (BCE); see what cross-entropy loss is.

The loss is the negative log-likelihood of a Bernoulli model, and the derivation is written out below Step 5. Jurafsky and Martin train with this loss.

Now the payoff. Take a confident mistake: label 1, score z = -5, so p = 0.0067 (illustrative). The gradient of cross-entropy with respect to z is p - y = -0.9933, a full-strength correction. Squared error, in the form $(1-p{)}^2$, gives -0.0132, about 75 times smaller, because the flat sigmoid tail multiplies the signal away; with a factor of one half in front of the squared error, the gradient halves and the ratio is roughly 150.¹ Google gives a related reason, precision near zero and one; the convexity argument is in the trade-offs below.

Step 4: The gradient tells each weight which way to move

The gradient of log loss with respect to weight $w_j$ is the prediction error times the feature value.

The chain rule gives this in three factors. The loss changes with p by $\frac{p-y}{p(1-p)}$, p changes with z by $p(1-p)$, and z changes with $w_j$ by $x_j$. The $p(1-p)$ terms cancel. The bias gradient is p - y. The update is $w_j\leftarrow w_j-\eta (p-y)x_j$, where the learning rate is the step size η. A gradient step moves the weights, and training loops back to the score with the new weights.

Input: the weights and a batch of examples. Output: new weights. Linear regression with squared error written with a factor of one half has a gradient of the same shape, (ŷ - y)·x_j, with a raw number in place of p.

A convex loss is a loss with a single basin, so no local minimum can trap gradient descent. The cross-entropy loss of logistic regression is convex, as Jurafsky and Martin note. scikit-learn offers several solvers: lbfgs, liblinear, newton-cg, newton-cholesky, sag and saga.

Step 5: The decision boundary turns the probability into a class

The decision boundary is the set of inputs where the predicted probability equals the threshold. With the usual threshold of one half it is where z = 0, that is $w·x+b=0$, a straight line or flat plane in feature space. With one feature it is the point $x=-b/w$.

A different threshold moves the boundary parallel to itself: a threshold of 0.7 corresponds to z = 0.8473, and 0.3 to z = -0.8473.¹ Choose the threshold from the cost of each kind of mistake, not by habit. Input: p. Output: a class label.

For more than two classes, the sigmoid generalises to softmax, or one binary model is fitted per class; scikit-learn documents that the liblinear solver can only handle binary classification by default, and that its one-vs-rest option fits a binary problem for each label. You now have the full picture: in training, score, sigmoid, loss and gradient step repeat; at prediction, a threshold on p sets the class.

1 · Weighted sum2 · Sigmoid3 · Log loss4 · Gradient step5 · BoundaryReplay

Step-by-step description

1. The features and weights give a score z, the log-odds.
2. The sigmoid turns z into a probability.
3. Log loss measures how wrong the probability is.
4. The gradient moves each weight, and training loops back.
5. At prediction, a threshold on p gives a class.

• A weighted sum of the features gives the log-odds z.
• The sigmoid turns z into a probability p.
• Log loss measures how far p is from the label.
• A gradient step moves the weights, and training loops back.
• At prediction, a threshold on p gives a class.

How do you derive the logistic regression loss from a Bernoulli model?

The logistic regression loss comes from writing the likelihood of the labels under a Bernoulli model and taking the negative log. For one example, the label y is 1 with probability p and 0 with probability 1 − p, where p is the sigmoid of a score that is linear in the features. One formula covers both cases:

For independent examples the likelihood is the product over all of them:

The logarithm is increasing, so the weights that maximise the log-likelihood are the same, and a sum is easier to differentiate than a product:

Negate it and divide by the number of examples n, and the result is the mean BCE loss. Maximising the likelihood and minimising BCE are one problem.

Worked example

Start with one training example and the smallest possible model. The feature is x = 2, the label is y = 1, the weight is w = 1.5, the bias is b = -3 and the learning rate is 0.5, all illustrative.¹ The score is z = 1.5 × 2 - 3 = 0, so p = 0.5 and the loss is -ln 0.5 = 0.6931. The gradient for w is (0.5 - 1) × 2 = -1.0 and for b it is -0.5; a finite-difference check of the loss gives the same -1.0.¹

The update gives w = 2.0 and b = -2.75. The new score is 1.25, so p = 0.7773 and the loss falls to 0.2519.¹

1 · Score2 · Probability3 · Loss4 · Gradient5 · UpdateReplay

Step-by-step description

1. Weighted sum of the feature plus the bias.
2. The sigmoid of zero is one half.
3. Log loss of a true label at probability 0.5.
4. Error times feature for the weight, error for the bias.
5. A step of 0.5 lowers the loss to 0.2519.

• The score is 1.5 times 2 minus 3, which is 0, so the probability is 0.5 and the loss is 0.6931.
• The gradient for the weight is minus 1.0 and the gradient for the bias is minus 0.5.
• The update gives a weight of 2.0 and a bias of minus 2.75, and the loss falls to 0.2519.

Each extra unit of x adds w to the log-odds, so it multiplies the odds by e to the power w. With w = 1.5 (illustrative) that factor is 4.4817: at x = 2 the odds are 1 and p = 0.5, and at x = 3 the odds are 4.4817 and p = 0.8176.¹

Logistic regression from scratch in NumPy

The loop below is the five steps in code, and it is the one the data script runs. The comment marks where each step happens; the optional lam adds an L2 penalty on the weights.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def fit(X, y, lr=0.5, steps=2000, lam=0.0):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(steps):
        p = sigmoid(X @ w + b)                      # steps 1 and 2: score, probability
        w -= lr * (X.T @ (p - y) / n + lam * w)     # step 4: gradient, plus L2 term
        b -= lr * np.mean(p - y)
    return w, b

On six illustrative points (x from 1 to 6, labels 0, 0, 1, 0, 1, 1, so the classes overlap at x = 3 and 4), starting from zero weights, the mean loss falls from 0.6931 to about 0.58 after ten steps, 0.4294 after a hundred and 0.413 after five thousand. The fit ends at w = 1.214 and b = -4.2491, a boundary at x = 3.5 and an odds factor of 3.367 per unit of x. The six probabilities are 0.046, 0.139, 0.353, 0.647, 0.861 and 0.954.¹

Trade-offs and edge cases

Logistic regression trades flexibility for a convex loss and weights a person can read. Five questions show where the textbook answer needs care. As of the scikit-learn 1.6.1 documentation (read 2026-10-26), the liblinear solver can only handle binary classification by default, while the other solvers minimise the full multinomial loss for several classes, so check the solver before a multiclass fit.

Why is MSE non-convex for logistic regression?

Mean squared error (MSE) is a poor loss for logistic regression because, combined with the sigmoid, it is not convex and its gradient fades on confident mistakes. For label 1, with squared error written as $(1-p{)}^2$, the second derivative with respect to z is $-2p(1-p{)}^2(1-3p)$. That is negative whenever p is below one third, that is z below -ln 2, about -0.6931,¹ so the loss curves downward there.

A numeric check agrees: the smallest second difference on a grid is -0.1202 for squared error and never negative for log loss.¹ Because z is linear in the weights, the same shape appears in the weights. Google's stated reason, precision, is compatible with this one.

Text description of the diagram

• At a score of minus five with a true label of 1, the cross-entropy gradient has size 0.9933.
• The squared-error gradient has size 0.0132, about 75 times smaller.

How do logistic regression and linear regression compare?

Logistic and linear regression share a weighted sum and a gradient of the same shape but differ in output and loss.

What happens when the classes are perfectly separable?

Perfect separation is the case where one straight boundary splits the training labels with no mistakes, and then unregularised logistic regression drives its weights ever larger. The loss can always be lowered a little by making the sigmoid steeper.

On four points, x = 1 to 4 with labels 0, 0, 1, 1 (illustrative), the weight is 1.716 after 100 steps, 4.583 after 1,000 and 8.925 after 10,000, while the mean loss keeps falling to 0.0058. With an L2 penalty of 0.1 (illustrative) the weight stays at 1.508 and the loss at 0.2424.¹ scikit-learn supports L1, L2 and elastic-net penalties, and Google calls regularisation crucial for preventing overfitting. See L1 vs L2 regularisation.

Can logistic regression learn a non-linear boundary?

Logistic regression has a boundary that is linear in its input features, but the features can be non-linear transforms of the raw data. Adding a squared term, a product of two features or a set of bins turns a straight boundary in feature space into a curve in the original inputs.

The model is still a weighted sum, so the loss stays convex. The cost is that you must choose the transforms; stacked layers of logistic units, as in a neural network, learn them instead. More transforms lower bias and raise variance, the trade in bias vs variance. Encoding and scaling choices are covered in feature engineering techniques.

Why can AUC be high while the probabilities are miscalibrated?

Area under the ROC curve (AUC) measures only how well a model ranks positives above negatives, so it cannot see whether the probabilities are right. Calibration is the property that, among examples given probability p, about a fraction p are positive. AUC depends only on the order of the scores, so any strictly increasing map of the scores, such as sigmoid (Platt) recalibration, leaves it unchanged; isotonic recalibration can create ties and shift AUC slightly.

The Brier score is the mean squared difference between the predicted probability and the 0 or 1 outcome. An illustrative case: ten scores with AUC 0.96. Stretching their log-odds by a factor of three, a strictly increasing map, keeps AUC at 0.96 but moves log loss from 0.3493 to 0.2707 and the Brier score from 0.1025 to 0.0864.¹ Regularisation, class weights and shifted data can all distort the probabilities (general reasoning, not a measurement). For the ranking side, see the ROC curve in scikit-learn.

How to answer it in an interview

The spoken structure: say the model in two lines, give the loss and the gradient, then name the convexity, the linear boundary and the calibration check.

Common mistakes

• "Logistic regression predicts a number, like linear regression." (wrong) It outputs a probability and is used for classification, unlike linear regression. (right) "It models the log-odds linearly and returns a probability."
• "The probability has to be a straight-line function of the features." (wrong) The central assumption is on the log-odds, and the probability follows an S-curve. (right) "The log-odds are linear in the features."
• "Squared error works just as well as the loss." (wrong) It is not convex with a sigmoid and sends a weak signal on confident mistakes. (right) "I use log loss, from maximum likelihood on a Bernoulli model."
• "A high AUC means the probabilities are good." (wrong) AUC measures ranking only. (right) "I also check a calibration curve, log loss and the Brier score."
• "A larger C means stronger regularisation in scikit-learn." (wrong) C is the inverse of the regularisation strength, so a smaller C is stronger, as the scikit-learn documentation describes it. (right) "I lower C to regularise more."

Follow-up questions

Why do we use the sigmoid function?

The sigmoid maps any real score to a value between zero and one and is the inverse of the logit (log-odds) function, so a linear score becomes a valid probability. It is also smooth, which gives the clean gradient (p - y)·x_j.

Why is cross-entropy commonly used for classification tasks?

Cross-entropy is the negative log-likelihood of the labels under the model's probabilities, so minimising it is maximum likelihood. For logistic regression it is convex and its gradient stays large on confident mistakes, which squared error does not give. The cross-entropy loss page covers the general case.

Your model's AUC is high but it's miscalibrated on the high-velocity segment. What do you do?

Confirm the gap first: plot a calibration curve for that segment and compare its log loss and Brier score with the overall ones, since AUC will not show it. Then fit a recalibration map, sigmoid (Platt) or isotonic, on held-out data from that segment; a per-segment map is a judgement call, not a documented rule. Check whether a shift in that segment's data explains the gap.

When would you use a two-tower model versus logistic regression?

Logistic regression scores one feature vector per example, which suits classification and ranking over a modest candidate set with known features. A two-tower model encodes queries and items separately so item vectors can be precomputed and searched among millions. The two are often stacked: a two-tower model retrieves candidates, then a logistic or neural ranker orders them.

What happens if you move the threshold from 0.5 to 0.7?

The boundary moves to a score of 0.8473 instead of zero.¹ The weights do not change. Fewer examples are called positive, which usually raises precision and lowers recall; choose the value from the cost of each mistake on held-out data.

What are the loss functions and the gradient during backprop for linear and logistic regression?

Linear regression minimises squared error and logistic regression minimises log loss. In backpropagation terms, the gradient at the output has the same form in both: for linear regression, with squared error written with a factor of one half, $\frac{\partial L}{\partial \hat{y}}=\hat{y}-y$, and for logistic regression $\frac{\partial L}{\partial z}=p-y$. The chain rule then multiplies by $x_j$ to reach each weight, and by one for the bias.

Sources

1. Logistic regression: Calculating a probability with the sigmoid function (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
2. Logistic regression: Loss and regularization (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
3. Logistic Regression module overview (Machine Learning Crash Course) — Google for Developers, 26 Oct 2026
4. Speech and Language Processing, 3rd ed. draft: Logistic Regression chapter — Stanford University (Jurafsky and Martin), 26 Oct 2026
5. sklearn.linear_model.LogisticRegression (scikit-learn 1.6.1 documentation) — scikit-learn, 26 Oct 2026