Feature Scaling and Normalization: When You Need It and How to Do It Right
Why some models need feature scaling, the difference between standardization and min-max normalization, which algorithms care, and the leakage trap to avoid.
Put a salary feature (say 20,000–200,000) next to an age feature (18–90) and, for many algorithms, salary wins every argument by default — not because it matters more, but because its numbers are bigger. Any model that computes distances or takes gradient steps across features inherits this accidental weighting. Feature scaling removes it by putting every feature on a comparable footing, letting the *data* decide what matters.
The two main techniques
- Standardization (z-score): subtract the mean, divide by the standard deviation — each feature ends up centred at 0 with unit variance. The robust default: no bounded range assumed, moderate outliers tolerated.
- Min-max normalization: map values linearly into [0, 1] using the observed min and max. Intuitive and bounded (nice for images and neural-net inputs), but a single outlier can crush everything else into a sliver of the range.
- Robust scaling: like standardization but using median and interquartile range — the choice when outliers are heavy.
- Log transform: for heavily skewed positive features (income, counts), taking the log first often helps more than any scaler.
Which algorithms care — and which don't
| Scaling matters | Scaling is irrelevant |
|---|---|
| k-nearest neighbours, k-means, SVMs (distance/margin-based) | Decision trees |
| Linear/logistic regression with L1/L2 regularization (penalty compares weight sizes) | Random forests |
| Neural networks (conditioning of gradient descent) | Gradient-boosted trees |
| PCA (variance-based — the largest-scale feature dominates the components) | Naive Bayes |
The pattern: anything built on distances, dot products, penalized weights or gradient descent needs scaling; anything built on split thresholds (trees and their ensembles) doesn't, because asking 'is salary > 50,000?' works identically on any scale. Note that L1/L2 regularization specifically assumes comparable scales — the penalty treats a weight of 5 the same whether its feature is measured in years or in rupees, so unscaled features get unfairly punished or favoured.
The trap: fit the scaler on training data only
The most common preprocessing leak
Computing the mean/std (or min/max) over your entire dataset before splitting leaks test-set statistics into training — quietly inflating your scores. Fit the scaler on the training fold only, then apply it unchanged to validation and test data. Inside cross-validation, that means the scaler lives inside the pipeline, re-fit per fold.
This is the same leakage discipline covered in cross-validation explained. For the reference version, see the feature scaling and normalization topic page and the standardize vs min-max drill. **AI Learning** turns preprocessing questions like these into offline practice — ~10,000 Q&A with explanations, no account, nothing leaves your device.
Free · Works offline · Preprocessing traps are interview staples — drill them in advance.
Sources
Frequently asked questions
More in Learn AI & Data Science
- Coding Interview Strategy: A Framework for Solving Problems Under PressureKnowing algorithms isn't enough — you have to deploy them under pressure. Here's a repeatable framework for solving unseen problems and a study plan that builds durable skill.
- P vs NP and Complexity Classes: What 'Hard' Really Means (Intuition, No Proofs)Why do some problems have fast algorithms and others resist every attempt? P vs NP is the deepest open question in computer science — here's the intuition, minus the proofs.
- Union-Find (Disjoint Set Union): Near-Constant Connectivity with Two OptimizationsUnion-Find answers 'are these two things connected?' and 'connect them' in almost O(1) — thanks to two beautifully simple optimizations that make the trees nearly flat.