Feature scaling and normalization

Feature scaling is the process of transforming numeric features so they share a comparable range or distribution. It prevents features with large numeric ranges from dominating those with small ranges, which helps many algorithms learn fairly and converge faster.

Min-Max normalization uses x' = (x - min) / (max - min). It linearly rescales each value so the minimum becomes 0 and the maximum becomes 1, preserving the shape of the original distribution.

Standardization computes x' = (x - mean) / std, giving the feature a mean of 0 and a standard deviation of 1. It centers the data and expresses each value as the number of standard deviations from the mean.

k-NN computes distances (e.g., Euclidean) between points, so a feature measured in large units would dominate the distance calculation. Scaling puts all features on a comparable footing so each contributes appropriately to the distance.

Tree-based models such as decision trees and random forests split on thresholds of individual features, so any order-preserving (monotonic) scaling does not change the splits. Therefore they generally do not require feature scaling.

Min-Max normalization rescales values into a fixed bounded range like [0, 1], while standardization centers data to mean 0 and unit variance without a fixed bound. Min-Max is sensitive to outliers because it depends on min and max, whereas standardization depends on mean and standard deviation.

Fitting the scaler (computing min/max or mean/std) on the whole dataset leaks information from the test set into training, giving overly optimistic results. You fit on training data and then apply that same transformation to the test data to keep evaluation honest.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)  # transform only

StandardScaler in sklearn.preprocessing standardizes features by removing the mean and scaling to unit variance. It exposes fit, transform, and fit_transform methods following the standard scikit-learn API.

RobustScaler subtracts the median and divides by the interquartile range (IQR), making it far less sensitive to extreme values than Min-Max or Standard scaling. It is a good choice when the data contains outliers.

Normalizer scales each individual sample (row) to have unit norm (e.g., L2 norm = 1), rather than scaling each feature (column). It is useful when the direction of the data vector matters more than its magnitude, such as in text/TF-IDF vectors.

Using x' = (x - min)/(max - min) = (30 - 10)/(50 - 10) = 20/40 = 0.5. The value 30 is the midpoint of the range, so it maps to 0.5.

Z-score = (x - mean)/std = (70 - 50)/10 = 2.0. So 70 lies two standard deviations above the mean.

Because Min-Max uses the minimum and maximum, a single extreme outlier stretches the range and squashes all the other values into a tiny interval. This makes Min-Max sensitive to outliers and can hurt the usefulness of the scaled feature.

When features have very different scales, the loss surface becomes elongated, so gradient descent zig-zags and takes many small steps. Scaling makes the surface more symmetric (closer to circular contours), allowing larger, more direct steps toward the minimum.

No. Standardization is a linear transformation (subtract mean, divide by std), so it shifts and rescales values but does not change the underlying shape of the distribution. A skewed feature remains skewed after standardization.

A log transformation (e.g., log(x) or log1p(x)) compresses large values and expands small ones, reducing right skew and making the distribution more symmetric. It is often applied before standard scaling on heavily skewed positive features.

import numpy as np
x_log = np.log1p(x)  # log(1+x), safe for zeros

Putting the scaler in a Pipeline ensures it is fit only on the training folds during cross-validation and applied consistently at prediction time, preventing data leakage. It also bundles preprocessing and the model into one reusable object.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([('scale', StandardScaler()), ('clf', LogisticRegression())])

The fit step learns the parameters needed for the transformation from the data, for example the min and max for MinMaxScaler or the mean and standard deviation for StandardScaler. These stored parameters are then used by transform to scale any data.

Generally there is no need to scale one-hot columns, since they are already on a small 0/1 scale; scaling them can make them harder to interpret. Continuous features benefit most from scaling, so scaling is usually targeted at them.

The scaled value will exceed 1, because the transform uses the training min and max. This is expected and acceptable; the scaler does not clip by default, so out-of-range test values map outside [0, 1].

MaxAbsScaler divides each feature by its maximum absolute value, mapping data into [-1, 1] without shifting/centering it. Because it does not subtract a mean, it preserves zeros and is well suited to sparse data.

L2 regularization penalizes the size of coefficients, but coefficients depend on feature scale, so a large-range feature naturally gets a small coefficient and is penalized differently. Scaling makes the penalty apply fairly across all features.

StandardScaler stores the mean and the standard deviation of each feature computed on the training data. Transform then subtracts the mean and divides by the standard deviation for each feature.

For Min-Max, max - min = 0 causing division by zero; for Z-score, the standard deviation is 0, also causing division by zero. Such constant features carry no information and are typically dropped before scaling (sklearn sets their scaled output to 0).

Dividing pixel values by 255 maps them to the range [0, 1], which is a form of Min-Max scaling with a known fixed min (0) and max (255). It helps neural networks train more stably on image data.

x_scaled = pixels / 255.0

Yes, scaling (usually standardization) is strongly recommended because PCA maximizes variance, and unscaled features with large variance would dominate the principal components. Standardizing ensures each feature contributes based on correlation structure rather than raw units.

The transform method applies the already-learned parameters to new data, while fit_transform both learns and applies. For test data you should use transform (not fit_transform) so the training parameters are reused.

Unit variance means the feature's variance (and standard deviation) equals 1 after scaling. Combined with a mean of 0, it expresses each value in terms of standard-deviation units from the mean.

The RBF kernel relies on distances between points, so features with larger ranges dominate the kernel computation and bias the decision boundary. Standardizing features ensures each contributes appropriately, often dramatically improving SVM performance.

Because Min-Max scaling is an invertible linear transformation, the scaler's inverse_transform method uses the stored min and max to recover the original values. This works as long as the same fitted scaler is used.

orig = scaler.inverse_transform(X_scaled)

L1 normalization scales each sample so the sum of absolute values equals 1, while L2 normalization scales so the square root of the sum of squares (Euclidean norm) equals 1. They normalize the magnitude of each row but using different distance measures.

Logistic regression has no closed-form solution, so it is fit iteratively with gradient descent, whose step behavior interacts with feature scales: very different scales slow or destabilize convergence and force careful learning-rate tuning. Without regularization, a fully converged solution gives an equivalent model regardless of scale (the coefficients simply absorb the scale factor), but in practice scaling still improves convergence speed, numerical stability, and fair regularization.

Scaling the target is optional and not generally required; models can fit any target scale, though it can sometimes help optimization or interpretation. If you do scale y, you must inverse-transform predictions back to the original units to interpret them correctly.