Feature selection methods quiz

Question 1

What are the three broad categories of feature selection methods, and how do they fundamentally differ?

Accepted Answer

Filter, wrapper, and embedded methods

Answer

Linear, nonlinear, and hybrid methods

Answer

Supervised, unsupervised, and reinforcement methods

Answer

Forward, backward, and bidirectional methods

Question 2

Why must feature selection that uses the target variable be performed inside cross-validation folds rather than once on the whole dataset?

Accepted Answer

It avoids data leakage and optimistic bias in performance estimates

Answer

It makes the pipeline run faster

Answer

It is required by scikit-learn's API

Answer

It reduces the number of features selected

Question 3

How does L1 (Lasso) regularization perform feature selection, and why does L2 (Ridge) not?

Accepted Answer

L1 sets some coefficients exactly to zero (sparsity) while L2 only shrinks them

Answer

L1 shrinks coefficients smoothly while L2 sets them to zero

Answer

Both set coefficients to zero but L2 does it faster

Answer

Neither performs feature selection

Question 4

What is mutual information in the context of feature selection, and what advantage does it have over Pearson correlation?

Accepted Answer

It captures arbitrary nonlinear dependencies, unlike Pearson correlation

Answer

It only captures linear relationships, like correlation

Answer

It requires the data to be normally distributed

Answer

It is identical to the chi-square statistic

Question 5

Describe Recursive Feature Elimination (RFE) and its main computational drawback.

Accepted Answer

It iteratively removes the least important features, retraining each round

Answer

It adds features one at a time based on correlation

Answer

It uses only univariate statistics and never trains a model

Answer

It applies PCA before removing components

Question 6

What bias affects impurity-based (Gini) feature importance in random forests, and how can it be mitigated?

Accepted Answer

It is biased toward high-cardinality / continuous features and computed on training data

Answer

It is biased toward low-cardinality categorical features

Answer

It always equals permutation importance

Answer

It cannot be computed for classification trees

Question 7

How does permutation feature importance work?

Accepted Answer

It shuffles a feature's values and measures the performance drop

Answer

It removes a feature and retrains the model from scratch

Answer

It computes the gradient of loss with respect to each feature

Answer

It ranks features by their variance

Question 8

What is the mRMR criterion in feature selection?

Accepted Answer

Maximize relevance to target while minimizing redundancy among features

Answer

Maximize redundancy and minimize relevance

Answer

Minimize both relevance and redundancy

Answer

Select features only by variance

Question 9

What does the VIF (Variance Inflation Factor) measure, and what threshold commonly flags problematic multicollinearity?

Accepted Answer

VIF measures multicollinearity (1/(1-R^2)); >5-10 is problematic

Answer

VIF measures feature relevance to the target; >0.5 is bad

Answer

VIF measures variance of the target; >1 is bad

Answer

VIF measures mutual information; >2 is problematic

Question 10

How is the chi-square statistic used as a feature selection filter, and what are its requirements?

Accepted Answer

It measures dependence between categorical feature and target; needs non-negative values

Answer

It requires negative-valued continuous features

Answer

It only works for regression targets

Answer

It is identical to Pearson correlation

Question 11

What is the Boruta algorithm and how does it decide whether a feature is relevant?

Accepted Answer

It compares real features against randomized shadow features' importance

Answer

It selects the k features with highest correlation

Answer

It uses L1 regularization paths

Answer

It performs PCA on the feature matrix

Question 12

Distinguish forward selection from backward elimination in stepwise feature selection.

Accepted Answer

Forward adds features from empty set; backward removes from full set

Answer

Forward removes features; backward adds features

Answer

Both start from the full feature set

Answer

Both are filter methods using correlation

Question 13

Why is feature selection conceptually different from dimensionality reduction like PCA?

Accepted Answer

Selection keeps a subset of original features; PCA creates new combined features

Answer

They are identical techniques

Answer

PCA keeps original features; selection creates new ones

Answer

Both always require the target variable

Question 14

What is stability selection and what problem does it address?

Accepted Answer

It aggregates selections over many subsamples to find stable, robust features

Answer

It selects features by maximizing variance

Answer

It removes outliers before selection

Answer

It is a wrapper using a single train/test split

Question 15

How does the ANOVA F-test serve as a feature selection filter for classification?

Accepted Answer

It tests differences in feature means across classes (between vs within variance)

Answer

It measures nonlinear dependence between two continuous variables

Answer

It is only valid for regression targets

Answer

It requires non-negative integer features

Question 16

What does a VarianceThreshold filter do, and what is its key limitation?

Accepted Answer

Removes near-constant (low-variance) features; ignores the target

Answer

Removes high-variance features that cause overfitting

Answer

Selects features most correlated with the target

Answer

Normalizes feature variance to one

Question 17

How can SHAP values be used for feature selection, and what is a caveat?

Accepted Answer

Mean |SHAP| gives consistent importance but splits credit among correlated features

Answer

SHAP can only explain linear models

Answer

SHAP values are always identical to permutation importance

Answer

SHAP requires the target during inference

Question 18

What is the Relief / ReliefF family of feature selection algorithms based on?

Accepted Answer

It weights features by how well they separate nearest neighbors of different classes

Answer

It ranks features by global variance

Answer

It is a wrapper that retrains a model per feature

Answer

It uses L1 penalties to zero out weights

Question 19

In the bias-variance sense, why can adding more features eventually hurt model performance (curse of dimensionality)?

Accepted Answer

Sparse high-dimensional data raises variance/overfitting from noisy, redundant features

Answer

More features always improve performance

Answer

Adding features only affects training time, never accuracy

Answer

It only matters for deep learning models

Question 20

When would you prefer Spearman over Pearson correlation as a univariate filter?

Accepted Answer

When the relationship is monotonic/nonlinear or data has outliers/ranks

Answer

When the relationship is strictly linear with no outliers

Answer

When the feature is purely categorical nominal

Answer

When you need to detect U-shaped relationships

Question 21

What is Elastic Net and why is it sometimes preferred over plain Lasso for feature selection?

Accepted Answer

It combines L1 and L2, handling correlated features better than Lasso

Answer

It uses only an L2 penalty

Answer

It removes the sparsity property of Lasso

Answer

It is unsupervised like PCA

Question 22

What is the difference between 'minimal-optimal' and 'all-relevant' feature selection?

Accepted Answer

Minimal-optimal finds smallest predictive subset; all-relevant finds every informative feature

Answer

They are the same goal phrased differently

Answer

All-relevant always returns fewer features

Answer

Minimal-optimal ignores predictive performance

Question 23

How does SelectFromModel decide which features to keep, and what is the threshold parameter?

Accepted Answer

It keeps features with importance above a threshold (numeric or 'mean'/'median')

Answer

It always keeps exactly half the features

Answer

It retrains the model once per feature like RFE

Answer

It selects by univariate correlation only

Question 24

Why can univariate filter methods select redundant features, and how do multivariate methods help?

Accepted Answer

Univariate scoring ignores inter-feature relationships, so it can keep redundant features

Answer

Univariate filters always remove redundancy automatically

Answer

Multivariate methods are always faster than univariate

Answer

Univariate filters capture feature interactions by design

Question 25

What is the regularization path in Lasso and how is it useful for feature selection?

Accepted Answer

It shows how coefficients enter/shrink as the L1 penalty changes

Answer

It shows accuracy vs training-set size

Answer

It plots feature variance vs threshold

Answer

It is only defined for Ridge regression

Question 26

What is a Markov blanket of the target, and why is it the theoretical optimum for feature selection?

Accepted Answer

The minimal feature set that makes the target conditionally independent of all others

Answer

The set of features with highest variance

Answer

The set of features uncorrelated with the target

Answer

The principal components of the feature matrix

Question 27

How do genetic algorithms approach feature subset selection?

Accepted Answer

They evolve binary feature masks using selection, crossover, and mutation

Answer

They rank features by mutual information only

Answer

They are filter methods independent of any model

Answer

They guarantee the globally optimal subset

Question 28

What is drop-column importance and how does it differ from permutation importance?

Accepted Answer

Drop-column retrains without the feature; permutation shuffles on a fixed trained model

Answer

Drop-column shuffles values; permutation retrains the model

Answer

They are identical methods

Answer

Drop-column never requires retraining

Question 29

Why should categorical-target feature selection use stratified folds, and what metric pitfall arises with imbalanced classes?

Accepted Answer

Stratify folds and avoid accuracy; use AUC/F1/balanced accuracy under imbalance

Answer

Accuracy is always the safest metric for imbalanced data

Answer

Imbalance has no effect on feature selection

Answer

Only filter methods are affected by imbalance

Question 30

What is Correlation-based Feature Selection (CFS) and its core heuristic?

Accepted Answer

It rewards feature-target correlation and penalizes feature-feature redundancy

Answer

It maximizes inter-feature correlation

Answer

It is a wrapper requiring model retraining

Answer

It selects features purely by variance

Test yourself