Normalization (standardize, min-max)

Normalization is the process of rescaling numeric features so they share a comparable range or distribution. It is needed because many algorithms (like gradient descent, KNN, and SVM) are sensitive to the scale of features, and unscaled features with large ranges can dominate the learning. Bringing features to a common scale speeds up training and prevents bias toward high-magnitude features.

Min-Max scaling uses x' = (x - min) / (max - min), where min and max are the minimum and maximum of the feature. This maps the smallest value to 0 and the largest value to 1, with all others in between. It is a linear transformation that preserves the relative spacing of the data.

Standardization uses z = (x - mu) / sigma, where mu is the feature mean and sigma is its standard deviation. The result has a mean of 0 and a standard deviation of 1. It measures how many standard deviations a value lies from the mean.

After standardization the transformed feature has a mean of 0 and a standard deviation of 1. This is why it is called the standard score. The shape of the distribution itself is not changed, only its center and spread.

After Min-Max scaling to [0, 1], the smallest original value becomes 0 and the largest becomes 1, so the minimum is 0 and the maximum is 1. All other values fall between these bounds. The data is squeezed into a fixed, bounded interval.

Using x' = (x - min)/(max - min) = (30 - 10)/(50 - 10) = 20/40 = 0.5. So 30 maps to 0.5, which makes sense because 30 is exactly halfway between 10 and 50.

Using z = (x - mu)/sigma = (30 - 20)/5 = 10/5 = 2. The value 30 lies 2 standard deviations above the mean. A positive z-score means the value is above average.

Min-Max scaling is generally more sensitive to outliers because it depends on the minimum and maximum, and a single extreme value can stretch the range and squash the rest of the data into a tiny interval. Standardization is also affected by outliers (through the mean and standard deviation) but typically less severely. For data with strong outliers, robust scaling (using median and IQR) is often preferred.

No. Standardization only shifts the data (subtracting the mean) and rescales it (dividing by the standard deviation), which are linear operations. A skewed distribution stays skewed after standardization. To change the shape, you need nonlinear transforms like log or Box-Cox.

Fitting only on training data ensures the scaling parameters (like min, max, mean, std) come solely from data the model is allowed to see. If you fit on the full dataset, information from the test set leaks into training, giving overly optimistic results. You call fit_transform on the training set and transform (not fit) on the test set.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

StandardScaler performs z-score standardization (mean 0, standard deviation 1), and MinMaxScaler performs Min-Max scaling (default range [0, 1]). Both are in sklearn.preprocessing and follow the fit/transform API. You choose based on whether you want a bounded range or a centered, unit-variance feature.

from sklearn.preprocessing import StandardScaler, MinMaxScaler

KNN computes distances (often Euclidean) between points, so a feature with a large numeric range contributes much more to the distance than a feature with a small range. Without scaling, large-range features dominate the neighbor selection and small-range features are effectively ignored. Scaling puts features on a comparable footing so each contributes fairly.

When features are on very different scales, the loss surface becomes elongated, causing gradient descent to zig-zag and converge slowly. Scaling features to similar ranges makes the surface more symmetric (more circular contours), so gradient descent takes a more direct path and converges faster. This is why scaling is recommended for linear/logistic regression and neural networks trained with gradient descent.

Tree-based models such as decision trees, random forests, and gradient-boosted trees generally do not require scaling. They split on thresholds of individual features, so monotonic rescaling does not change which splits are possible. Scaling neither helps nor hurts their splits.

First scale to [0, 1] with x01 = (x - min)/(max - min), then map to [a, b] using x' = a + x01 * (b - a). For [-1, 1], this gives x' = -1 + 2 * (x - min)/(max - min). In scikit-learn you simply set MinMaxScaler(feature_range=(-1, 1)).

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
X_scaled = scaler.fit_transform(X)

A negative z-score means the value lies below the mean of the feature. For example, z = -1.5 means the value is 1.5 standard deviations below average. A z-score of 0 means the value equals the mean.

The scaled value will be greater than 1 because the scaler uses the training min and max, and the new value lies outside that range. Min-Max scaling does not clip by default, so out-of-range inputs produce out-of-range outputs. You can enable clipping (e.g. MinMaxScaler(clip=True)) if you need to bound the result.

Yes, Min-Max scaling is linear because it has the form x' = a*x + b, where a = 1/(max - min) and b = -min/(max - min). Both a and b are constants for a given feature. Linear transformations preserve the order and the relative spacing of the data points.

Standardization (StandardScaler) rescales each column (feature) to mean 0 and standard deviation 1. Normalizer instead rescales each row (sample) so that its vector has unit norm (e.g. L2 norm = 1). They operate on different axes and serve different purposes; Normalizer is common in text/TF-IDF and cosine-similarity settings.

Prefer Min-Max scaling when you need values in a bounded, fixed range (such as [0, 1]) and the data has few outliers, for example feeding pixel intensities into a neural network or algorithms that expect bounded inputs. Standardization is usually preferred when data is roughly Gaussian or when the algorithm assumes zero-centered features. The choice depends on the algorithm and the data distribution.

If all values are equal, then max - min = 0, causing a division by zero in the formula. scikit-learn handles this by treating the zero range specially (it does not divide), so the constant feature is mapped to the low end of the range (0 for the default [0, 1]) rather than crashing. A constant feature carries no information and is often dropped anyway.

Scaling parameters must be learned after splitting, by fitting the scaler only on the training set. If you scale before splitting (on the whole dataset), test-set statistics leak into training and inflate performance estimates. The clean approach is to fit on train, then transform both train and test, ideally inside a Pipeline.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
pipe = Pipeline([('scaler', StandardScaler()), ('knn', KNeighborsClassifier())])
pipe.fit(X_train, y_train)

Here min = 2 and max = 8, so x' = (4 - 2)/(8 - 2) = 2/6 = 0.333. So 4 maps to about 0.33. The smallest value 2 would map to 0 and the largest 8 to 1.

Subtracting the mean shifts the data so its average becomes 0, which is mean centering. Dividing by the standard deviation rescales the spread so the variance (and standard deviation) becomes 1, which is unit variance. Together these two steps produce the standardized z-scores.

Usually only the input features (X) are scaled, and the target is left as-is so predictions stay in the original units. However, scaling the target can sometimes help optimization (e.g. for neural nets), in which case you must inverse-transform the predictions back to original units before interpreting them. Whether to scale y depends on the model and goal.

For an approximately normal distribution, about 68% of values fall within one standard deviation of the mean, i.e. between z = -1 and z = +1. About 95% fall within z = -2 to +2. This is the empirical (68-95-99.7) rule.

Yes, because Min-Max scaling is a monotonic increasing linear transformation, it preserves the order: if a < b before scaling then a' < b' after scaling. The relative spacing is also preserved up to a constant factor. Only the numeric range changes, not the ranking.

Compute the mean and standard deviation along the appropriate axis, then subtract the mean and divide by the standard deviation: z = (x - x.mean()) / x.std(). For column-wise scaling of a 2D array, use axis=0. This reproduces what StandardScaler does.

import numpy as np
z = (x - x.mean(axis=0)) / x.std(axis=0)

A single extreme outlier becomes the new max (or min), stretching the denominator (max - min) so that almost all the normal data gets compressed into a very narrow sub-interval near 0 (or 1). This loss of resolution makes the scaled feature less useful. In such cases, standardization or robust scaling is usually better.

RobustScaler scales features using the median and the interquartile range (IQR = Q3 - Q1) instead of the mean/std or min/max. Because the median and IQR are resistant to outliers, it is useful when the data has many outliers that would distort StandardScaler or MinMaxScaler. The formula is x' = (x - median) / IQR.

from sklearn.preprocessing import RobustScaler
X_scaled = RobustScaler().fit_transform(X)

Euclidean distance adds squared differences across features, so a feature with a numerically larger unit/range contributes disproportionately to the total distance. Without scaling, the result depends on arbitrary unit choices rather than true similarity. Scaling makes each feature contribute comparably, so the distance reflects genuine differences.

The mean is 20, so 20 - 20 = 0, and dividing by any nonzero standard deviation still gives 0. Therefore the z-score of the value equal to the mean is always 0. The other two values, 10 and 30, would be symmetric negative and positive z-scores.

In cross-validation, each fold should have its scaler fit only on that fold's training portion; a Pipeline ensures fit happens per-fold automatically, preventing leakage from validation data. Applying scaling manually to the whole dataset before CV leaks information and inflates scores. The Pipeline also makes the code cleaner and ensures the same transform is reused at prediction time.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
model = make_pipeline(StandardScaler(), SVC())
scores = cross_val_score(model, X, y, cv=5)

False. Standardization centers data to mean 0 and unit standard deviation, but the values are unbounded; large outliers can produce z-scores like 4 or -3. If you specifically need bounded values in [0, 1], use Min-Max scaling instead. Standardization controls center and spread, not the range.