L1 vs L2 regularization

In this article

1. Before you start
2. Why interviewers ask this
3. What each one is
4. L1 and L2 side by side
5. Worked example
6. Trade-offs and edge cases
7. How to answer it in an interview
8. Common mistakes
9. Follow-up questions
10. Sources

Before you start

Why interviewers ask this

The wrong penalty costs you either a model nobody can interpret or one that falls apart when two inputs move together. The question usually arrives as "what is the difference between L1 and L2 regularization?", and the real test is the follow-up: why does one of them produce zeros? A weak answer recites "L1 is lasso, L2 is ridge".

A strong answer states the two penalties, says which one selects features, gives the geometric or gradient reason, and names a case where each is the better choice. Interviewers use it to separate candidates who memorized a table from candidates who can reason about what a penalty does to a fitted model. It is asked in machine learning fundamentals rounds and data science screens; the classical machine learning overview covers the wider family of models.

What each one is

Regularization

Regularization is a technique that limits how closely a model can fit its training data; the most common form adds a penalty on the size of the weights to the training loss, so the fitted model prefers smaller weights and overfits less.

L1 penalty

An L1 penalty is a regularization term equal to a constant times the sum of the absolute values of the weights; a linear model trained with it is called a lasso model.

L2 penalty

An L2 penalty is a regularization term equal to a constant times the sum of the squares of the weights; a linear model trained with it is called a ridge model.

Sparse model

A sparse model is a model in which many weights are exactly zero, so the prediction depends on only a few input features.

In plain words

Regularization is a tax on large weights. Training still tries to fit the data. But every unit of weight now costs something, so a weight grows only if it earns its keep. L1 is a flat fee per unit of weight; L2 is a rate that rises with the size of the weight. The strength of the tax is a constant called alpha in scikit-learn. The analogy breaks in one place. A taxpayer decides each cost on its own, but the optimizer re-chooses all the weights together.

Feature selection is the choice of which input features a model uses. A penalty that zeroes weights makes that choice automatically.

Both objectives are squared error plus a penalty. The scikit-learn linear models page (version 1.9.1, read Oct 2026) gives ridge as ${\min }_w\Vert Xw-y{\Vert }_2^2+\alpha \Vert w{\Vert }_2^2$. It gives lasso as ${\min }_w\frac{1}{2n}\Vert Xw-y{\Vert }_2^2+\alpha \Vert w{\Vert }_1$. The error term is scaled differently. So one alpha value is not the same penalty strength in the two models. Compare shapes, not alpha values.

1 · L1 penalty2 · L2 penalty3 · Elastic netReplay

Step-by-step description

1. Sum of absolute weights: some become zero.
2. Sum of squared weights: all shrink.
3. A mix: sparse and stable.

• The L1 penalty is the sum of absolute weights and gives a sparse model.
• The L2 penalty is the sum of squared weights and shrinks every weight.
• The elastic net is a weighted mix of the L1 and L2 penalties.

Step 1: The L1 penalty (lasso)

The L1 penalty charges the sum of absolute weights, so every weight pays the same price per unit of size, however large or small it is. The sklearn documentation says the lasso "estimates sparse coefficients, i.e., it is able to set coefficients exactly to zero" and tends to prefer solutions with fewer non-zero coefficients. Tibshirani (1996) introduced the lasso as least squares with the sum of absolute coefficients constrained, and reports that the constraint drives some coefficients to exactly zero, giving more interpretable models. Input: a loss and alpha. Output: a model with some weights removed. You now know why L1 is described as a feature selector.

Step 2: The L2 penalty (ridge)

The L2 penalty charges the sum of squared weights, so the pull toward zero is proportional to the weight itself. With one weight, or features that are uncorrelated and on the same scale, the L2 penalty divides every weight by the same number, 1 + lambda, and a fraction of a non-zero number is never zero. With real, correlated features the weights shrink by different amounts and a few can even grow, but the total size of the weights falls and exact zeros still do not appear.

The sklearn documentation says ridge imposes a penalty on the size of the coefficients and that a larger alpha gives more shrinkage and coefficients that are less sensitive to collinearity. Collinearity means two or more input features carry nearly the same information. Input: a loss and alpha. Output: a model whose weights are smaller as a whole. You now know why ridge is the default when all features matter a little.

Step 3: The elastic net combines them

The elastic net is a model trained with a weighted mix of the L1 and L2 penalties, set in scikit-learn by the l1_ratio parameter. Zou and Hastie (2005) report that it often beats the lasso while giving a similarly sparse model, that it tends to select or drop strongly correlated predictors together, and that it helps when there are far more predictors than observations, a setting where the lasso does poorly. scikit-learn describes it as learning a sparse model "while still maintaining the regularization properties of Ridge". Input: both penalties and a mixing ratio. Output: sparse but stable weights. The next section shows the mechanism behind all three.

Why does L1 give exact zeros while L2 does not?

L1 gives zeros because its penalty pulls a weight toward zero with a constant force, and L2 does not because its pull shrinks as the weight shrinks. For one weight $w$ with an unpenalized best value $b$, minimizing $(w-b{)}^2+\lambda \left|w\right|$ gives $w=\mathrm{sign}(b)\max (\left|b\right|-\lambda /2,0)$, while $(w-b{)}^2+\lambda w^2$ gives $w=b/(1+\lambda )$.

L1 subtracts a fixed amount and clips at zero; L2 divides by a constant, which never reaches zero. In words: the L1 penalty takes the same bite out of a weight whether it is small or large, so a small weight is eaten away completely, while the L2 bite is a share of the weight, so a small weight loses only a small amount.¹

What does one weight do under each penalty?

The computed one-weight values make it concrete (penalty strength 1, so L1 subtracts 0.5 and L2 halves). An unpenalized weight of 0.3 becomes 0 under L1 and 0.15 under L2. A weight of 1.0 becomes 0.5 under both. A weight of 2.0 becomes 1.5 under L1 and 1.0 under L2, and 4.0 becomes 3.5 under L1 and 2.0 under L2.¹ At b = 0.5 the L1 line leaves zero and from there runs parallel to the diagonal, while the L2 line is straight from the start. L1 zeroes every weight up to 0.5 and takes the same small amount from big weights; L2 takes the same fraction from every weight and zeroes only a weight that was already zero.

Text description of the diagram

• With penalty strength 1, L1 maps b to max(b - 0.5, 0), so every b up to 0.5 becomes exactly 0.
• With penalty strength 1, L2 maps b to b/2, so a non-zero b never becomes 0.

What do the L1 and L2 graphs look like?

The geometric picture says the same, once two terms are defined. The penalty can be written as a limit on the weights instead of a charge: L1 says the sum of the absolute weights must stay at or below some budget t, and L2 says the sum of squares must. With two weights, the allowed region for L1 is a diamond whose corners sit on the axes, and for L2 a circle with no corners.

An error contour is the set of weight pairs that give the same training error; the unpenalized best fit sits at the center of nested ovals, and the penalized answer is the point where the smallest oval first touches the allowed region. Against a diamond, that first touch usually lands on a corner, and a corner on an axis is a point where one weight is exactly zero. Against a circle, the touch point is almost never on an axis.

L1 and L2 side by side

L1 and L2 differ in what they subtract, whether they produce zeros and which features they suit, as the table shows.

The objectives are from the scikit-learn linear models page (version 1.9.1, read October 2026); the worked-example figures are computed below;¹ the priors follow from maximum a posteriori (MAP) estimation.

Worked example

We built a regression problem with 20 features, of which 4 matter (true weights 3, -2, 1.5 and 1) and 16 are pure noise, with 60 training rows and noise of standard deviation 2. The data is synthetic and illustrative, drawn with seed 0, and a separate 2,000-row set scores each model.¹ The penalty strength of ridge and lasso was each chosen by cross-validation. Cross-validation is fitting on part of the training rows and scoring on the rest, in turn.

Mean squared error is the average of the squared gaps between predictions and true values. The lowest error any model could reach here is 4.0, the noise variance, so both penalties close part of the gap that overfitting opens. Ridge shrank the irrelevant weights a little (4.30 to 3.73) but left all 16 non-zero. The lasso removed 11 and cut their total size to 1.55. Compared with the unpenalized fit, the lasso pulled the two largest real weights toward zero: the first from 3.54 to 3.25 and the second from -1.98 to -1.64.¹

The ordinary fit overshoots on this sample (3.54 against a true 3.0), so the lasso lands closer to the truth on the first weight and further on the second (-1.64 against -2.0). A penalty pulls the weights as a whole toward zero compared with the unpenalized fit (the lasso cuts the sum of absolute weights from 13.54 to 9.51, and ridge cuts the sum of squares from 25.47 to 21.22), which adds bias on average in return for lower variance.¹ A single weight can still move away from zero: ridge makes 4 of the 20 weights larger than the unpenalized fit, and two change sign.¹

Strength matters, as the next chart shows. Using the same alpha values for the lasso, the count of exact zeros among all 20 weights rises from 1 at alpha 0.01 to 6 at 0.1, 13 at 0.3, 16 at 1 and 20 at 3. The ridge model has none at any alpha. The lasso's test error is lowest near alpha 0.3 (4.70) and then climbs to 9.82 at alpha 1 and 21.54 at alpha 3.¹

At alpha 1 the lasso keeps exactly the 4 real features and zeroes all 16 irrelevant ones, so the selection is perfect, yet the error is worse: the penalty over-shrinks the kept weights (the third weight is 0.045 against a true 1.5, and the first is 2.00 against 3.0). Only at alpha 3 does the penalty delete the real features too, leaving a model with no non-zero weight at all.¹

Text description of the diagram

• The lasso has 1 exact zero among 20 weights at penalty 0.01 and 20 at penalty 3.
• Ridge has no exact zeros at any penalty.

One more case separates the two: correlated features. We made two features with correlation 0.999 that each have a true weight of 2. Ridge (alpha 10) returned 2.01 and 2.03, sharing the weight evenly. The ordinary fit gave 1.74 and 2.39. The lasso (alpha 0.5) returned 1.38 and 2.29, splitting it unevenly, and with alpha 2 it kept one and dropped the other (0 and 2.34).¹ Ridge raised the first weight above the ordinary fit (1.74 to 2.01), so a penalty can move a single weight away from zero.

Trade-offs and edge cases

The choice depends on what you believe about the features and what you need from the model.

Feature selection. Use L1 when you suspect most features are irrelevant and want a model that names the few that matter. Ng (2004) proved that for logistic regression with L1 regularization the number of training examples needed grows only logarithmically in the number of irrelevant features, while any rotationally invariant method, including L2-regularized logistic regression, needs at least a linear number in the worst case. That is the theory behind "L1 for many irrelevant features".

Correlated features. Use L2, or the elastic net, when features are strongly correlated. The lasso's pick between correlated features can be arbitrary and can change with a small change in the data, which makes the selected set unstable; ridge shares the weight, and Zou and Hastie's elastic net keeps that grouping behavior with sparsity.

Scale first. Both penalties charge by weight size, so features on different scales are penalized unequally. Standardize features to a common scale (feature scaling and encoding) before fitting. A feature whose values are numerically small (kilometers rather than meters) needs a larger weight for the same effect, so the penalty falls hardest on it and the lasso may zero it, while a feature in large numbers needs only a tiny weight and is barely penalized.

More features than rows. The lasso can select at most as many features as there are rows, a limit Zou and Hastie (2005) describe in their introduction; the elastic net does not have that ceiling, and they present it as the fix for the many-predictors case.

Neural networks. L2 penalties appear as weight decay in deep learning; L1 is used when sparse weights are wanted. The two are equivalent for standard stochastic gradient descent (once rescaled by the learning rate) but not for adaptive optimizers such as Adam: Loshchilov and Hutter (2017, arXiv:1711.05101) show that L2 regularization and weight decay differ there, and propose decoupled weight decay (AdamW). The Bayesian reading, an L2 penalty as a Gaussian prior and an L1 penalty as a Laplace prior, is the maximum a posteriori (MAP) reading of the two penalties; the statistics interview questions cover priors. The flow below is a starting order, not a rule.

Text description of the diagram

• Use L1 to select among many irrelevant features.
• Use L2 when features are correlated or all matter a little.
• Use the elastic net when you want both.

How to answer it in an interview

The spoken structure is: both penalties, the one-line difference in effect, the reason, then when to use each.

Common mistakes

"L2 removes features." (wrong) It shrinks weights but leaves them non-zero. (right) "L2 shrinks all weights; L1 can zero them."

"L1 and L2 are different models." (wrong) They are penalties added to the same loss. (right) "Lasso and ridge are the same linear model with different penalties."

"Use the same alpha for both." (wrong) The scaling differs, as the scikit-learn objectives show. (right) "Tune the penalty strength for each by cross-validation."

"Skip scaling." (wrong) The penalty then falls hardest on features whose values are numerically small, because they need larger weights. (right) "Standardize first."

"L1 always beats L2." (wrong) With correlated features its selection is unstable. (right) "It depends on the features; try elastic net."

Follow-up questions

How does L1 exhibit feature selection while L2 does not?

L1 pulls each weight toward zero with a constant force, so a weak weight is driven all the way to zero. L2's pull is proportional to the weight, so it fades as the weight gets small and never reaches zero.

How do L1 and L2 losses help with overfitting?

Both limit how large the weights can grow, which limits how closely the model can chase noise. In the worked example the lasso cut test error from 7.20 to 5.25 and ridge cut it to 6.35 on the same data.

What if two features are strongly correlated?

Ridge splits the weight between them. The lasso tends to keep one and drop the other, and which one it keeps can change with small data changes. The elastic net is the usual compromise.

Do you remember the graphs for L1 and L2?

Yes. Written as a budget on the weights, the L1 region (sum of absolute weights at most t) is a diamond with corners on the axes and the L2 region is a circle. A contour is the set of weights with equal training error, and the smallest contour that reaches the region usually touches the diamond at a corner, where one weight is zero. The circle has no corners, and the one-weight chart above shows the same effect in one dimension.

What are the different types of regularization?

Beyond L1 and L2 penalties and their mix in the elastic net, common types are early stopping, dropout in neural networks and data augmentation. See bias and variance for how they fit the trade-off. Gradient-boosted trees have their own penalties: the XGBoost parameter documentation (read October 2026) lists an L1 term on weights (alpha, alias reg_alpha) and an L2 term on weights (lambda, alias reg_lambda), and says that raising either makes the model more conservative. By the one-weight logic above, the L1 term can push the weight of a leaf with a weak signal to exactly zero, while the L2 term shrinks every leaf weight.

Do you need to scale features before regularizing?

Yes. The penalty charges by weight size, so features on different scales are penalized unequally (small-valued features, which need big weights, are hit hardest); standardize to a common scale first.

Sources

1. 1.1. Linear Models (Ridge, Lasso, Elastic-Net), scikit-learn 1.9.1 documentation — scikit-learn developers (page shows version only; date = read date), 24 Oct 2026
2. Regression Shrinkage and Selection via the Lasso — Journal of the Royal Statistical Society B, 58(1), 267-288 (Tibshirani; year only), 1 Jan 1996
3. Regularization and Variable Selection via the Elastic Net — Journal of the Royal Statistical Society B, 67(2), 301-320 (Zou and Hastie), 9 Mar 2005
4. Decoupled Weight Decay Regularization (arXiv:1711.05101v3) — Loshchilov and Hutter, arXiv (ICLR 2019), 14 Nov 2017
5. Feature selection, L1 vs. L2 regularization, and rotational invariance — ICML 2004 (Ng; year from the proceedings, day approximate), 4 Jul 2004
6. XGBoost Parameters (tree booster: lambda and alpha), stable documentation — XGBoost developers (page shows no date; date = read date), 26 Oct 2026