What is the main difference between L1 and L2 regularization?

L1 penalizes the absolute values of weights and drives some exactly to zero, producing sparse models that effectively select features. L2 penalizes squared weights and shrinks all of them smoothly toward — but not to — zero.

Why does L1 regularization produce sparse models?

The L1 penalty exerts a constant pull on every weight regardless of its size. Weights that don't earn their keep against that fixed force are pushed exactly to zero. L2's pull weakens as weights shrink, so they approach zero without reaching it.

When should I use ridge instead of lasso?

When most features carry some signal, or when features are strongly correlated — lasso tends to arbitrarily keep one of a correlated group and drop the rest, while ridge distributes weight across them more stably.

What is weight decay in deep learning?

The neural-network form of L2 regularization: each update shrinks weights slightly toward zero in proportion to their size. It's a default in most modern training recipes, often combined with dropout.

GuideJuly 15, 2026Learn AI & Data Science

L1 vs L2 Regularization: Sparsity, Shrinkage, and When to Use Each

How L1 (lasso) and L2 (ridge) penalties differ, why L1 produces sparse models and L2 shrinks weights smoothly, and how to choose between them.

Regularization is the standard cure for overfitting: add a penalty to the loss function so the model pays a price for large weights, trading a little bias for a big cut in variance — the bias–variance tradeoff put to work. The two classic penalties look almost identical on paper and behave very differently in practice.

The two penalties

L1 (lasso) adds λ × Σ|w| — the sum of the *absolute values* of the weights.
L2 (ridge) adds λ × Σw² — the sum of the *squares* of the weights.
In both, λ controls the strength: λ = 0 is no regularization; a huge λ crushes the model toward zero (and into underfitting).

Why L1 zeroes weights and L2 doesn't

The intuition lives in the gradients. The L2 penalty's pull on a weight is proportional to the weight itself — as a weight approaches zero, the pull fades, so weights glide *toward* zero but settle just short of it. The L1 penalty's pull is constant regardless of size — a weight whose usefulness can't outbid that fixed force gets pushed all the way to exactly zero and stays there. Geometrically: the L1 constraint region is a diamond whose sharp corners sit on the axes (where coordinates are zero), and optimal solutions tend to land on those corners; the L2 region is a circle with no corners to catch on.

Choosing between them

Situation	Better fit	Why
Many features, few truly relevant	L1	Zeros out the noise features — a sparse, interpretable model
Most features carry some signal	L2	Keeps everyone with shrunk weights instead of amputating
Correlated features	L2 (or elastic net)	L1 arbitrarily picks one of a correlated group and drops the rest; L2 shares weight across them
You need built-in feature selection	L1	The zeroed weights are the deselected features
Deep learning weight decay	L2	The standard choice, usually alongside dropout

Elastic net: don't choose

Elastic net combines both penalties with a mixing ratio, giving L1's sparsity with L2's stability on correlated features. When in doubt — especially with wide, correlated data — it's a strong default, with λ tuned by cross-validation.

In neural networks the same L2 idea appears as weight decay, typically combined with dropout — see regularization: L1, L2 and dropout and bias, variance and regularization for the drill-down. Always tune λ with cross-validation rather than guessing. **AI Learning** covers regularization across its cheat sheets and ~10,000 practice questions — a favourite interview topic — fully offline, no account.

AI Learning: AI/ML/DS Q&A

Free · Works offline · Regularization is a top interview topic — practice it until it's easy.

Coming soon toGoogle Play

Sources

Tibshirani (1996) — 'Regression Shrinkage and Selection via the Lasso', JRSS B
Hastie, Tibshirani & Friedman — The Elements of Statistical Learning, ch. 3
Zou & Hastie (2005) — 'Regularization and variable selection via the elastic net', JRSS B