Outliers and anomaly detection

An outlier is a data point that differs significantly from the other observations in a dataset. It lies far away from the bulk of the data, either much higher or much lower than typical values. Outliers can arise from measurement errors, data-entry mistakes, or genuinely rare events.

The mean is highly sensitive to outliers because it sums every value, so a large outlier pulls the mean strongly toward itself. The median is resistant (robust) because it only depends on the middle position, so it barely changes. This is why the median is preferred for skewed data with outliers.

A Z-score measures how many standard deviations a data point is away from the mean. It is computed as z = (x - mean) / standard deviation. Points with |z| greater than a threshold (commonly 3) are flagged as outliers.

z = (x - data.mean()) / data.std()
outliers = data[abs(z) > 3]

The 3-sigma rule considers values within three standard deviations of the mean as normal, i.e. between mean - 3*std and mean + 3*std. About 99.7% of data in a normal distribution falls within this range. Values outside it are treated as potential outliers.

The IQR is the difference between the third quartile (Q3) and the first quartile (Q1), covering the middle 50% of the data. Outliers are typically defined as values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR. These cutoffs are called Tukey's fences.

Q1, Q3 = data.quantile(0.25), data.quantile(0.75)
IQR = Q3 - Q1
out = data[(data < Q1-1.5*IQR) | (data > Q3+1.5*IQR)]

In a boxplot, the box spans Q1 to Q3 and the whiskers usually extend up to 1.5*IQR beyond the quartiles. Data points plotted individually beyond the whiskers are shown as dots or stars and are flagged as potential outliers. This makes the boxplot a quick visual tool for spotting extreme values.

The terms overlap and are often used interchangeably. 'Outlier' is a statistical term for any point far from the rest of the data, often treated as something to clean. 'Anomaly' usually implies a point that is interesting or meaningful, such as fraud or a system failure, that we actively want to detect. In practice, an anomaly is typically a meaningful outlier we care about.

The three main types are point anomalies (a single instance far from the rest), contextual anomalies (normal in general but abnormal in a specific context, like high heating in summer), and collective anomalies (a group of points that together are abnormal even if each alone seems normal). Recognizing the type guides the choice of detection method.

A contextual anomaly is a value that is normal in one context but abnormal in another. For example, a temperature of 30 degrees Celsius is normal in summer but anomalous in winter. The same numeric value is only an anomaly because of its surrounding context, often time or location.

In fraud detection, the vast majority of transactions are legitimate and only a tiny fraction are fraudulent, so the classes are heavily imbalanced. A naive model that predicts everything as legitimate can still get very high accuracy while catching no fraud. Therefore metrics like precision, recall, and the F1-score matter more than plain accuracy.

When anomalies are very rare, a model can achieve high accuracy simply by labeling everything as normal, while completely failing to detect the anomalies. Accuracy hides this failure because the normal class dominates. Precision, recall, and F1-score better reflect performance on the rare anomaly class.

Isolation Forest builds random trees by repeatedly splitting data on random features at random values. Anomalies are easier to isolate, so they require fewer splits and end up with shorter average path lengths in the trees. Points with short path lengths are scored as anomalies.

from sklearn.ensemble import IsolationForest
model = IsolationForest(contamination=0.05)
labels = model.fit_predict(X)  # -1 = anomaly

LOF compares the local density of a point to the local densities of its neighbors. A point in a region much sparser than its neighbors gets a high LOF score and is flagged as an outlier. This lets LOF detect local outliers that global methods like Z-score might miss.

from sklearn.neighbors import LocalOutlierFactor
lof = LocalOutlierFactor(n_neighbors=20)
labels = lof.fit_predict(X)  # -1 = outlier

DBSCAN groups together points that are densely packed and labels points in low-density regions as noise. The points marked as noise (not assigned to any cluster) are treated as outliers or anomalies. This works well because anomalies tend to lie in sparse regions away from dense clusters.

from sklearn.cluster import DBSCAN
labels = DBSCAN(eps=0.5, min_samples=5).fit_predict(X)
anomalies = X[labels == -1]

The modified Z-score replaces the mean and standard deviation with the median and the median absolute deviation (MAD), which are robust to outliers. Because the ordinary mean and std are themselves distorted by extreme values, the MAD-based version gives more reliable outlier flags. A common rule flags points with a modified Z-score above about 3.5.

import numpy as np
med = np.median(x)
mad = np.median(np.abs(x - med))
mod_z = 0.6745 * (x - med) / mad

Supervised anomaly detection uses a labeled dataset where normal and anomalous examples are marked, training a classifier to tell them apart. Unsupervised anomaly detection has no labels and instead relies on the assumption that anomalies are rare and different, using density or distance to flag them. Unsupervised methods are common because labeled anomalies are often scarce.

Winsorizing limits extreme values by capping them at a chosen percentile, for example replacing everything above the 95th percentile with the 95th-percentile value and everything below the 5th with the 5th-percentile value. Unlike deletion, it keeps every data point but reduces the influence of extremes. This makes statistics like the mean more robust.

from scipy.stats.mstats import winsorize
clean = winsorize(data, limits=[0.05, 0.05])

No. Some outliers are errors and should be removed or corrected, but others are genuine, important observations such as fraud, equipment failures, or rare events. Removing meaningful outliers can destroy the very signal you are trying to detect. The right action depends on understanding why the outlier occurred.

Least-squares regression minimizes the sum of squared errors, so it heavily penalizes large residuals. A single far-off outlier can therefore pull the fitted line strongly toward itself, distorting the slope and intercept. Robust regression methods or removing/down-weighting the outlier can reduce this effect.

Common causes include data-entry mistakes, sensor or measurement errors, and unit or formatting inconsistencies, as well as genuinely rare but real events. Knowing the cause matters because errors can be corrected or removed, while real rare events should usually be kept. Investigating an outlier's origin is a key part of data cleaning.

Percentile capping sets thresholds at chosen percentiles, for example the 1st and 99th, and treats any value beyond them as an outlier to be capped or removed. It is simple and distribution-agnostic, making no assumption of normality. This is useful when data is skewed and the Z-score is unreliable.

lo, hi = data.quantile(0.01), data.quantile(0.99)
capped = data.clip(lower=lo, upper=hi)

The Z-score assumes the data is roughly symmetric and normal, using the mean and standard deviation that are themselves distorted by skew and extreme values. On skewed data, the mean shifts toward the long tail and the std inflates, so the thresholds become unreliable and can mislabel normal points. Robust methods like the IQR or modified Z-score are better choices.

In outlier detection, the training data itself contains outliers and the goal is to find them within that data. In novelty detection, the model is trained only on clean normal data and then decides whether new, unseen points are normal or novel (anomalous). One-Class SVM is a typical novelty-detection method.

A One-Class SVM learns a boundary that encloses most of the normal training data in feature space. New points falling inside the boundary are classified as normal, while points outside it are flagged as anomalies. It is often used for novelty detection when only normal examples are available for training.

from sklearn.svm import OneClassSVM
model = OneClassSVM(nu=0.05).fit(X_normal)
labels = model.predict(X_new)  # -1 = anomaly

An autoencoder is trained to compress and then reconstruct normal data, learning to rebuild typical patterns with low error. When given an anomalous input it cannot reconstruct it well, so the reconstruction error is large. Points with reconstruction error above a chosen threshold are flagged as anomalies.

A scatter plot displays each observation as a point using two variables as coordinates, making it easy to see the general pattern. Points that lie far from the main cloud of data stand out visually as potential outliers. It can also reveal outliers that are unusual only in the relationship between the two variables, not in either one alone.

Mahalanobis distance measures how far a point is from the center of a distribution while accounting for the correlations and spread of the variables. Unlike plain Euclidean distance, it scales by the covariance, so it correctly identifies multivariate outliers even when features are correlated. Points with a large Mahalanobis distance are flagged as outliers.

A false positive occurs when a normal data point is incorrectly flagged as an anomaly. Too many false positives waste investigators' time and can cause 'alert fatigue,' where real anomalies get ignored. Tuning thresholds balances catching true anomalies (recall) against keeping false positives low (precision).

A false negative is a true anomaly that the system fails to detect, classifying it as normal. In domains like fraud, security, or medical diagnosis, a missed anomaly can be far more costly than a false alarm. This is why recall (the fraction of real anomalies caught) is often emphasized.

If you compute statistics like the mean, std, or quartiles using the whole dataset including the test set, information from the test set leaks into preprocessing. This data leakage gives an overly optimistic and unrealistic estimate of performance. The correct practice is to fit thresholds on the training data and then apply them to the test data.

A collective anomaly is a sequence or group of data points that together are anomalous, even though each individual point may look normal on its own. For example, in an ECG signal a single low reading is fine, but a long flat stretch of low readings together signals a problem. Detecting them requires looking at patterns over the group, not isolated values.

Robust statistics are influenced very little by extreme values, because the median depends only on rank order and the IQR ignores the tails. The mean and standard deviation, by contrast, are computed from every value and get pulled toward outliers. So using robust measures gives a more faithful summary of the typical data when outliers exist.

Min-max scaling maps values to a range using the dataset's minimum and maximum. A single extreme outlier becomes the new maximum (or minimum), squeezing all the normal values into a tiny portion of the 0-to-1 range. This makes the scaled normal data nearly indistinguishable, which is why robust scaling is sometimes preferred.

from sklearn.preprocessing import RobustScaler
X_scaled = RobustScaler().fit_transform(X)  # uses median & IQR

A time-series anomaly is a point or pattern that deviates from the expected temporal behavior, such as a sudden spike, a level shift, or a break in seasonality or trend. Because order and context matter, a value can be anomalous purely due to when it occurs. Methods often model the expected value at each time step and flag large deviations.