Handling imbalanced data

With a 1% fraud prevalence, a model that always predicts 'not fraud' scores 99% accuracy while detecting zero fraud cases. Accuracy is dominated by the majority class and ignores the rare positives that actually matter. Metrics like precision, recall, PR-AUC, or F-beta on the minority class reveal the true (poor) performance.

ROC-AUC uses the false positive rate, whose denominator is the large negative class, so a flood of false positives barely moves the curve and ROC-AUC can look optimistic. Precision-Recall AUC focuses on precision and recall of the positive class, making it far more sensitive to performance on rare positives. PR-AUC therefore better reflects practical utility when positives are scarce.

SMOTE selects a minority-class point, finds its k nearest minority neighbors, and creates new samples by linearly interpolating between the point and a randomly chosen neighbor (x_new = x + lambda*(x_neighbor - x), lambda in [0,1]). This fills in the minority feature space rather than just duplicating points. It reduces overfitting compared to naive random oversampling.

from imblearn.over_sampling import SMOTE
X_res, y_res = SMOTE(k_neighbors=5, random_state=42).fit_resample(X, y)

If you oversample before splitting, synthetic or duplicated minority points leak between train and validation folds, so the validation set contains information derived from training rows. This produces optimistically biased metrics that collapse in production. Use an imblearn Pipeline so resampling is fit only on each training fold.

from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
pipe = Pipeline([('smote', SMOTE()), ('clf', clf)])
# cross_val_score(pipe, X, y, cv=5) -> SMOTE fit per fold

Resampling changes the training data distribution (adding/removing rows), while cost-sensitive learning leaves data intact and modifies the objective by assigning higher misclassification cost (or class weights) to the minority class. Cost-sensitive methods avoid duplicating data and information loss, integrate with the optimizer directly, and are often more memory-efficient. Both aim to shift the decision boundary toward the rare class.

clf = LogisticRegression(class_weight='balanced')
# weight_c = n_samples / (n_classes * n_samples_c)

It sets each class weight to n_samples / (n_classes * n_samples_in_class), which is inversely proportional to class frequency. The rare class receives a larger weight so its errors contribute more to the loss. This effectively rebalances the gradient without changing the data.

w_c = n_samples / (n_classes * count_c)

A Tomek Link is a pair of nearest-neighbor points from opposite classes; removing the majority member (or both) cleans overlapping, borderline, or noisy samples and sharpens the class boundary. It is an under-sampling/cleaning technique, often paired with SMOTE (SMOTE+Tomek) to first oversample then clean. It does not balance ratios on its own but improves separability.

from imblearn.under_sampling import TomekLinks
X_res, y_res = TomekLinks().fit_resample(X, y)

Focal Loss multiplies cross-entropy by (1 - p_t)^gamma, down-weighting easy, well-classified examples so training focuses on hard/rare ones. Larger gamma increases the down-weighting of easy examples; gamma=0 reduces it to standard cross-entropy. An alpha term additionally balances class frequencies.

FL = -alpha * (1 - p_t)**gamma * log(p_t)

The 0.5 threshold assumes balanced classes and equal error costs; with skewed priors a well-calibrated model rarely exceeds 0.5 for the rare class, yielding low recall. Tune the threshold on a validation set to optimize the operating metric (F-beta, recall at fixed precision, or expected cost) using the precision-recall curve. Threshold moving is cheap and decouples ranking quality from the decision policy.

from sklearn.metrics import precision_recall_curve
p, r, t = precision_recall_curve(y_val, proba)
f1 = 2*p*r/(p+r+1e-9)
best_t = t[f1[:-1].argmax()]

Borderline-SMOTE only oversamples minority points that lie near the decision boundary (those with many majority neighbors but not all), since these are the hardest and most informative to classify. Vanilla SMOTE samples uniformly across all minority points. Focusing on the borderline region strengthens the boundary where errors actually occur.

from imblearn.over_sampling import BorderlineSMOTE
X_res, y_res = BorderlineSMOTE(kind='borderline-1').fit_resample(X, y)

ADASYN (Adaptive Synthetic Sampling) generates more synthetic samples for minority points that are harder to learn, i.e., those surrounded by more majority neighbors, by weighting synthesis according to local class density. This adaptively shifts the decision boundary toward difficult regions, whereas SMOTE generates uniformly. The trade-off is that it can amplify outliers/noise in sparse regions.

from imblearn.over_sampling import ADASYN
X_res, y_res = ADASYN().fit_resample(X, y)

SMOTE interpolates in continuous Euclidean space, so linear combinations of one-hot or embedded categorical values produce meaningless fractional categories and distort distances. For mixed/categorical data use SMOTENC (handles categorical columns by majority vote) or SMOTEN for all-categorical, and avoid interpolating raw text features. Applying vanilla SMOTE post one-hot encoding can create invalid synthetic rows.

from imblearn.over_sampling import SMOTENC
sm = SMOTENC(categorical_features=[0,3,5])
X_res, y_res = sm.fit_resample(X, y)

Oversampling/undersampling changes the base rate the model sees, so its output probabilities reflect the resampled prior, not the true population prior, and are typically over-estimated for the rare class. You can correct with a prior-shift adjustment to the log-odds or, more robustly, apply post-hoc calibration (Platt/isotonic) on a held-out set drawn from the original distribution. Class-weighting can have a similar distorting effect.

from sklearn.calibration import CalibratedClassifierCV
cal = CalibratedClassifierCV(base_clf, method='isotonic', cv='prefit')
cal.fit(X_cal_original_dist, y_cal)

scale_pos_weight multiplies the gradient/hessian contribution of the positive class, effectively up-weighting it; a common heuristic is sum(negative)/sum(positive). It is a cost-sensitive alternative to resampling that keeps all data. Note it skews predicted probabilities, so calibrate if you need true probabilities.

spw = (y==0).sum() / (y==1).sum()
XGBClassifier(scale_pos_weight=spw)

Random undersampling discards potentially informative majority examples, increasing variance and risking loss of important patterns, especially when the majority is itself diverse. Mitigations include informed undersampling (NearMiss, Tomek, ENN, ClusterCentroids), or ensemble approaches like EasyEnsemble/BalancedBagging that train many models on different majority subsets to use all data. This recovers information while keeping class balance per learner.

from imblearn.ensemble import BalancedRandomForestClassifier
clf = BalancedRandomForestClassifier(n_estimators=300)

EasyEnsemble trains an ensemble of base learners (often AdaBoost), each on the full minority class plus a different random undersample of the majority class, then aggregates them. Across all subsets the majority data is largely used, so little information is lost, while each learner sees balanced data. It combines bagging-style diversity with balanced training.

from imblearn.ensemble import EasyEnsembleClassifier
clf = EasyEnsembleClassifier(n_estimators=10, random_state=42)

F-beta = (1+beta^2) * P*R / (beta^2 * P + R); beta>1 weights recall more heavily than precision, appropriate when missing a positive (false negative) is costlier than a false alarm, such as disease screening. beta<1 favors precision. Choosing beta encodes the relative cost of FN vs FP.

from sklearn.metrics import fbeta_score
fbeta_score(y, pred, beta=2)

MCC incorporates all four confusion-matrix cells (TP, TN, FP, FN) and is high only when the model does well on both classes, whereas F1 ignores true negatives and focuses on the positive class. MCC ranges from -1 to +1 and is more robust to class imbalance and to swapping which class is 'positive'. It gives a balanced summary even with very skewed priors.

from sklearn.metrics import matthews_corrcoef
matthews_corrcoef(y, pred)

Performance degradation depends not just on the imbalance ratio but on data difficulty factors: class overlap, small disjuncts (rare sub-concepts), noise, and minority-class within-class scarcity. A 1:100 ratio with well-separated classes can be easy, while 1:10 with heavy overlap can be very hard. Diagnosing these factors guides whether to resample, clean, or gather more data.

With absolute scarcity, the bottleneck is too few real positives, so synthetic oversampling overfits the few examples and cost-weighting has high variance. Prioritize collecting/labeling more positives, use stratified k-fold with repeated CV for stable estimates, leverage transfer learning or anomaly-detection framings, and apply strong regularization. The fix is data quantity, not just rebalancing.

When positives are extremely rare, heterogeneous, or ill-defined and you mainly have abundant 'normal' data, one-class methods (Isolation Forest, One-Class SVM, autoencoder reconstruction error) model the majority and flag deviations. This avoids needing many labeled positives and handles novel anomaly types unseen in training. Supervised methods are better when positives are sufficient and share consistent structure.

from sklearn.ensemble import IsolationForest
IsolationForest(contamination=0.01).fit(X_normal)

Random splitting can leave folds or the test set with too few (or zero) minority samples, producing unstable or undefined metrics and biased models. Stratification preserves the class proportion in every split so each fold is representative. For very small minorities, use repeated stratified CV to reduce variance of estimates.

from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

If you trained on a balanced sample but the true positive prior is pi and the sampled prior is pi_s, correct each prediction with p_corrected = (p * (pi/pi_s)) / (p*(pi/pi_s) + (1-p)*((1-pi)/(1-pi_s))). Equivalently, subtract log(pi_s/(1-pi_s)) - log(pi/(1-pi)) from the logit (an intercept shift). This restores probabilities consistent with the real base rate.

logit_corr = logit - (log(pi_s/(1-pi_s)) - log(pi/(1-pi)))

Micro-F1 aggregates TP/FP/FN across all classes, so it is dominated by frequent classes and effectively approximates accuracy. Macro-F1 computes F1 per class then averages with equal weight, giving rare classes equal say and exposing poor minority performance. Choose macro (or weighted) depending on whether all classes matter equally or proportionally.

from sklearn.metrics import f1_score
f1_score(y, pred, average='macro')

Selecting the operating threshold using the test set leaks the test labels into the decision policy, producing optimistic precision/recall that won't generalize. Thresholds (and any resampling/calibration) must be chosen on a separate validation split or via nested CV, with the test set used once for final reporting. This preserves an unbiased estimate of deployed performance.

NearMiss is a family of informed undersamplers that keep majority points based on distances to minority points: NearMiss-1 keeps majority samples with the smallest average distance to their k nearest minority neighbors, NearMiss-2 uses the average distance to the k farthest minority neighbors, and NearMiss-3 keeps majority samples that are among the nearest neighbors of each minority point. The goal is to retain majority examples near the boundary rather than discard them randomly.

from imblearn.under_sampling import NearMiss
X_res, y_res = NearMiss(version=1).fit_resample(X, y)

SMOTE can create synthetic points in overlapping regions or amplify noise; ENN then removes any sample (synthetic or original) whose class disagrees with the majority of its k nearest neighbors, cleaning ambiguous and noisy points from both classes. The result is a more separated, less noisy distribution and typically better generalization than SMOTE alone. It is more aggressive cleaning than SMOTE-Tomek.

from imblearn.combine import SMOTEENN
X_res, y_res = SMOTEENN().fit_resample(X, y)

Cui et al.'s class-balanced loss reweights each class by (1 - beta) / (1 - beta^n_c), where n_c is the class count and beta in [0,1); this models diminishing returns of additional samples (effective number = (1-beta^n)/(1-beta)) rather than simple inverse frequency. It avoids over-weighting extremely rare classes as harshly as 1/n_c. It can wrap cross-entropy, focal, or sigmoid losses.

w_c = (1 - beta) / (1 - beta**n_c)  # beta e.g. 0.999

Tree ensembles like random forests tend to push probabilities toward the majority and are often poorly calibrated under imbalance, while uncalibrated neural nets with class-weighting or focal loss also distort probabilities. Regardless of model, imbalance handling that alters the effective prior requires post-hoc calibration (isotonic/Platt) on original-distribution data if you need reliable probabilities. Ranking metrics may be fine even when probabilities are miscalibrated.

Balanced accuracy is the average of recall (true positive rate) across classes: for binary, (sensitivity + specificity)/2. It corrects ordinary accuracy's majority bias by giving each class equal weight, so a trivial majority predictor scores ~0.5, not 0.99. It is the macro-average of per-class recall.

from sklearn.metrics import balanced_accuracy_score
balanced_accuracy_score(y, pred)

Duplicating or interpolating raw minority text (SMOTE on embeddings) creates unnatural samples and degrades language structure; instead, balance the few-shot exemplars or fine-tuning batch composition so the model sees adequate minority signal without corrupting inputs. You can also use LLM-based data augmentation (paraphrase/generate realistic minority examples) and class-weighted loss during fine-tuning. This preserves linguistic validity while addressing skew.

G-mean = sqrt(sensitivity * specificity) (geometric mean of per-class recalls). Because it is a product, if recall on either class drops toward zero the whole metric collapses to zero, so a model that ignores the minority class is heavily penalized. It rewards balanced performance across both classes.

from imblearn.metrics import geometric_mean_score
geometric_mean_score(y, pred)

Reweighting the loss up-weights positive-class errors, which mainly rescales the predicted scores and relocates where the natural 0.5 boundary falls, while leaving the relative ordering of examples largely unchanged; since AUC is rank-based, it stays roughly constant (it can move modestly because the fit itself changes). Thus class weighting mostly relocates the decision threshold/operating point rather than improving discrimination. If discrimination is the goal, better features/models matter more than reweighting.

Prefer threshold tuning and cost-sensitive learning first since they keep data intact, are cheap, and preserve probability semantics (with calibration); use them when the model already ranks well. Reach for resampling (SMOTE/under-sampling/hybrids) when the learner cannot handle weights, when minority structure is sparse and synthetic interpolation is meaningful, or when class overlap is low. Always validate with PR-AUC/F-beta/MCC under proper CV and consider absolute minority count before adding synthetic data.