Feature selection methods

Filter methods rank features using intrinsic statistical measures (e.g., correlation, mutual information, chi-square) independent of any model, making them fast but blind to the downstream learner. Wrapper methods evaluate feature subsets by training and scoring an actual model (e.g., RFE, forward/backward selection), making them accurate and model-aware but computationally expensive. Embedded methods perform selection during model training as part of the learning objective (e.g., L1/Lasso, tree importances), balancing cost and model-awareness.

Selecting features using the target on the full dataset before CV leaks information from the validation folds into the selection step, producing optimistically biased performance estimates. The correct approach nests selection inside each training fold so the held-out fold never influences which features are chosen. This is especially critical with high-dimensional data (p >> n), where naive selection can inflate measured accuracy dramatically even on pure noise.

L1 adds a penalty proportional to the sum of absolute coefficients, whose diamond-shaped constraint region has corners on the axes, so the loss contour frequently touches a corner, driving some coefficients exactly to zero and thus selecting features. L2 penalizes squared coefficients, whose smooth spherical constraint shrinks them toward but essentially never exactly to zero, so it never removes features. This geometric difference is why Lasso yields sparse, interpretable models.

from sklearn.linear_model import LassoCV
import numpy as np
lasso = LassoCV(cv=5).fit(X, y)
selected = np.where(lasso.coef_ != 0)[0]

Mutual information measures the reduction in uncertainty about the target given a feature, capturing arbitrary (including nonlinear and non-monotonic) statistical dependencies. Pearson correlation only detects linear relationships and can be near zero for strong nonlinear ones (e.g., y = x^2). MI is non-negative and zero only under statistical independence, making it a more general univariate relevance score; its main downsides are estimation difficulty (binning/kNN sensitivity) and that it is still univariate, so it ignores redundancy.

from sklearn.feature_selection import mutual_info_classif
mi = mutual_info_classif(X, y, random_state=0)

RFE fits a model that exposes feature importances or coefficients, removes the least important feature(s), and repeats on the shrinking set until the desired number remains. It is a wrapper-style method that can capture feature interactions through the model. Its drawback is cost: it retrains the estimator many times (roughly O(p) fits, or far more with RFECV which adds an inner CV loop), which is expensive for large p or slow estimators.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
rfe = RFE(LogisticRegression(max_iter=1000), n_features_to_select=10).fit(X, y)

Impurity-based importance is biased toward high-cardinality and continuous features because they offer more candidate split points and can reduce impurity by chance, inflating their apparent importance. It is also computed on training data, so it can reward features that the model overfits. Permutation importance computed on held-out data is the standard, less-biased alternative.

After training a model, permutation importance shuffles a single feature's values in a validation set, breaking its relationship with the target, and measures the resulting drop in model performance. A large drop means the model relied heavily on that feature. It is model-agnostic and computed on held-out data, but it can be misleading when features are strongly correlated because shuffling one is partly compensated by its correlated partners (and shuffling can create unrealistic out-of-distribution inputs).

from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=0)

mRMR (minimum Redundancy Maximum Relevance) selects features that are highly relevant to the target while being minimally redundant with already-selected features, typically using mutual information for both terms. It addresses a key weakness of univariate filters that pick many correlated, redundant top-ranked features. The greedy objective maximizes relevance minus (MID) or divided by (MIQ) the average redundancy at each step.

VIF for a feature is 1/(1 - R^2) where R^2 comes from regressing that feature on all other features; it quantifies how much a coefficient's variance is inflated by multicollinearity. Values above 5 (or more conservatively 10) commonly indicate problematic collinearity that destabilizes linear-model coefficients. VIF is used to iteratively drop redundant predictors, though it does not consider the target.

from statsmodels.stats.outliers_influence import variance_inflation_factor
vifs = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

The chi-square test measures dependence between a categorical feature and a categorical target by comparing observed vs expected co-occurrence counts; higher statistics indicate stronger dependence. scikit-learn's chi2 requires non-negative features (e.g., counts or frequencies) and is intended for categorical/discrete data, so continuous features should be binned. It is a common univariate filter for text/count (e.g., TF or term-frequency) data.

from sklearn.feature_selection import SelectKBest, chi2
Xk = SelectKBest(chi2, k=20).fit_transform(X, y)

Boruta creates randomized copies of all features called shadow features, trains a random forest on the combined set, and compares each real feature's importance against the maximum importance among shadow features. A feature is confirmed relevant if it significantly outperforms the best shadow feature across many iterations (via a binomial test), and rejected if it consistently underperforms. It aims for all-relevant feature selection rather than a minimal-optimal subset.

Forward selection starts with no features and greedily adds the one that most improves the model at each step until no improvement is gained. Backward elimination starts with all features and greedily removes the least useful one each step. Forward is cheaper when the optimal subset is small; backward can better capture features that are only useful jointly but is costly with many features and infeasible when p > n for some models.

Feature selection chooses a subset of the original features, preserving their interpretability and measurement meaning. PCA and similar methods create new features as (linear) combinations of all originals, reducing dimensionality but yielding components that are harder to interpret and that still require measuring all inputs. Choose selection when interpretability or reducing data-collection cost matters; choose projection when you mainly want a compact, decorrelated representation.

Stability selection runs a base selection method (often Lasso) on many bootstrap/subsampled datasets and retains features selected frequently across them, controlling false discoveries. It addresses the instability of methods like Lasso, whose chosen feature set can change drastically with small data perturbations or correlated predictors. The selection-frequency threshold provides finite-sample control over the expected number of false positives.

The ANOVA F-test (f_classif) measures whether the mean of a continuous feature differs significantly across target classes by comparing between-group to within-group variance. A higher F-statistic indicates the feature separates classes well. It is fast and works for numeric features with a categorical target, but it only detects linear/mean-shift differences and assumes roughly normal, equal-variance groups.

from sklearn.feature_selection import SelectKBest, f_classif
sel = SelectKBest(f_classif, k=15).fit(X, y)

VarianceThreshold removes features whose variance falls below a set threshold, eliminating near-constant features that carry little information. It is unsupervised and extremely cheap, useful as a first cleanup step. Its key limitation is that it ignores the target, so a low-variance feature might still be highly predictive, and variance is scale-dependent, so it should follow appropriate scaling for continuous features.

from sklearn.feature_selection import VarianceThreshold
Xr = VarianceThreshold(threshold=0.0).fit_transform(X)

Mean absolute SHAP values aggregate each feature's contribution to model predictions, providing a consistent, model-aware importance ranking that can guide which features to keep. Unlike split-count importance, SHAP is theoretically grounded (Shapley values) and additive. A caveat is that for correlated features credit is split among them, and the interventional vs conditional SHAP variants can attribute importance differently, so a high SHAP ranking does not guarantee causal or non-redundant relevance.

import shap, numpy as np
vals = shap.TreeExplainer(model).shap_values(X)
imp = np.abs(vals).mean(0)

Relief assigns feature weights based on how well a feature distinguishes between instances that are near each other: for each sampled instance it rewards features that differ for the nearest neighbor of a different class (near-miss) and penalizes differences for the nearest same-class neighbor (near-hit). ReliefF extends this to multi-class and noisy data by using k nearest neighbors and averaging. It captures feature interactions and local structure that purely univariate filters miss.

As dimensionality grows, data becomes sparse and distances between points become less discriminative, so models need exponentially more data to generalize. Irrelevant or redundant features add variance and noise, increasing overfitting risk while contributing little signal, which degrades generalization. Feature selection counters this by removing low-value features, improving the signal-to-noise ratio, training speed, and interpretability.

Use Spearman (rank correlation) when the feature-target relationship is monotonic but nonlinear, or when the data has outliers or is ordinal, since it operates on ranks and is robust to monotonic transformations. Pearson assumes a linear relationship and is sensitive to outliers. Spearman will still miss non-monotonic dependencies (e.g., U-shaped), where mutual information is preferable.

Elastic Net combines L1 and L2 penalties, giving sparsity from L1 plus the grouping effect from L2. When features are highly correlated, plain Lasso tends to arbitrarily pick one and drop the rest and can be unstable; Elastic Net tends to select or shrink correlated groups together. It is also better behaved when p > n, where Lasso saturates at selecting at most n features.

from sklearn.linear_model import ElasticNetCV
en = ElasticNetCV(l1_ratio=[.1,.5,.9], cv=5).fit(X, y)

Minimal-optimal selection seeks the smallest feature subset that maximizes predictive performance for a given model, potentially discarding relevant-but-redundant features. All-relevant selection (e.g., Boruta) aims to identify every feature that carries information about the target, even if redundant, which is valuable for scientific understanding and causal exploration. The two goals can yield very different feature sets.

SelectFromModel wraps an estimator that exposes coef_ or feature_importances_ and keeps features whose importance exceeds a threshold. The threshold can be a numeric value, a string like 'mean' or 'median' (optionally scaled, e.g., '1.5*mean'), or defaults sensibly (e.g., 1e-5 for L1 models that already zero coefficients). It is an embedded approach using a single fit, much cheaper than RFE.

from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier
sfm = SelectFromModel(RandomForestClassifier(), threshold='median').fit(X, y)

Univariate filters score each feature independently against the target, so several highly correlated features can all rank near the top, yielding a redundant subset that adds little joint information. Multivariate methods (wrappers, embedded models, mRMR, CFS) evaluate features in the context of others, penalizing redundancy and rewarding complementary information. They also capture interactions where a feature is useful only in combination with another.

The regularization path traces how each coefficient changes as the L1 penalty strength (lambda/alpha) varies from large (all zeros) to small (full model). It reveals the order in which features enter the model and their stability, helping choose a sparsity level. Algorithms like LARS compute the entire piecewise-linear path efficiently, and cross-validation along the path selects an optimal alpha.

from sklearn.linear_model import lars_path
alphas, _, coefs = lars_path(X, y, method='lasso')

The Markov blanket of the target consists of its parents, children, and the children's other parents in the underlying probabilistic graph; conditioned on it, the target is independent of all other variables. It is therefore the minimal set of features that carries all predictive information about the target, making it the theoretical optimum for selection. Algorithms like IAMB and HITON aim to discover it, though reliable estimation is hard in practice.

Genetic algorithms encode a feature subset as a binary chromosome (1 = include), evaluate subsets via a fitness function (typically cross-validated model performance, possibly penalized by subset size), and evolve the population through selection, crossover, and mutation over generations. They explore the combinatorial subset space stochastically and can escape local optima that greedy stepwise methods get stuck in. The cost is many model evaluations and tuning of GA hyperparameters, with no guarantee of global optimality.

Drop-column importance removes one feature, retrains the model from scratch, and measures the performance change relative to the full model; a large drop means high importance. Unlike permutation importance, which only shuffles a feature on an already-trained model, drop-column lets the model fully re-optimize without the feature, giving a cleaner measure but at the cost of retraining p+1 models. Both can be misled by correlated features.

Stratified folds preserve class proportions in each fold so selection scores and validation estimates aren't distorted by uneven class splits, which matters greatly under imbalance. With imbalanced data, evaluating selection by accuracy is misleading because a trivial majority predictor scores high; metrics like ROC-AUC, PR-AUC, F1, or balanced accuracy should guide subset evaluation. Otherwise selection may favor features that only help the majority class.

CFS evaluates a feature subset by a heuristic that rewards high correlation between features and the target while penalizing inter-feature correlation (redundancy). The merit score is roughly (k * mean feature-class correlation) / sqrt(k + k(k-1) * mean feature-feature correlation), favoring subsets of relevant, mutually uncorrelated features. It is a multivariate filter, typically paired with a search such as best-first.

SFS is a wrapper that greedily adds (forward) or removes (backward) features based on cross-validated score improvement, and it works with any estimator even if it has no coef_ or feature_importances_. RFE instead relies on the estimator exposing importances/coefficients to rank and eliminate features and does not require an inner CV per step. SFS is more general but typically more expensive due to repeated CV evaluations.

from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.neighbors import KNeighborsClassifier
sfs = SequentialFeatureSelector(KNeighborsClassifier(), n_features_to_select=8, direction='forward').fit(X, y)

With correlated features, a tree ensemble distributes splits among the correlated group, diluting and underestimating each one's individual importance, so a genuinely predictive feature may appear weak. Conversely, dropping one correlated feature may barely change performance because its partners compensate, masking its true relevance. Remedies include clustering correlated features and keeping a representative, or using conditional/grouped importance measures.

A cheap unsupervised pre-filter (near-zero variance removal, dropping one of each highly correlated pair) shrinks the feature space quickly, reducing the combinatorial burden and runtime of the expensive wrapper/embedded selection that follows. This hybrid pipeline keeps the accuracy benefits of model-aware selection while making it tractable on high-dimensional data. Care is needed so the pre-filter does not discard features that are predictive only jointly.

Lasso can select at most n features (it saturates) and tends to pick one feature arbitrarily from a group of correlated ones, giving unstable selections. Common remedies include Elastic Net (which adds an L2 term to handle p>n and grouped correlated features) and stability selection over bootstraps to obtain robust, reproducible feature sets. Domain-informed grouping (e.g., group Lasso) can also help.