Feature importance quiz

Question 1

What is feature importance in machine learning?

Accepted Answer

A score measuring how much each feature influences a model's predictions

Answer

The number of unique values in a feature column

Answer

The Pearson correlation between two input features

Answer

The accuracy achieved when only one feature is used to train the model

Question 2

Which of the following is NOT a benefit of computing feature importance?

Accepted Answer

Guaranteeing higher model accuracy after feature removal

Answer

Understanding which variables drive predictions

Answer

Guiding feature selection to simplify models

Answer

Informing which data to collect for future model versions

Question 3

How does a Decision Tree measure the importance of each feature?

Accepted Answer

By measuring how much each feature reduces node impurity across all splits

Answer

By counting how many rows each feature appears in

Answer

By computing the Pearson correlation of each feature with the target

Answer

By calculating the variance of each feature column

Question 4

What does Gini impurity measure, and how is it linked to feature importance in Decision Trees?

Accepted Answer

A feature's importance equals the total reduction in Gini impurity it causes across all splits

Answer

Gini impurity measures the variance of a numerical feature column

Answer

Features with the lowest Gini impurity after splitting are least important

Answer

Gini impurity is only used in regression trees, not classification trees

Question 5

Which sklearn attribute gives you the feature importances of a trained Random Forest?

Accepted Answer

model.feature_importances_

Answer

model.coef_

Answer

model.score_

Answer

model.weights_

Question 6

Why is Random Forest feature importance generally more reliable than a single Decision Tree's importance?

Accepted Answer

Random Forest averages importance over many trees, reducing variance and improving stability

Answer

A single tree uses all features at every split, making it more thorough

Answer

Single tree importance is always more accurate because it is not averaged

Answer

Random Forest and single trees always produce identical feature importance rankings

Question 7

What is permutation importance and how does it differ from impurity-based (MDI) importance?

Accepted Answer

It measures performance drop on a validation set when a feature's values are randomly shuffled

Answer

Permutation importance is computed only for tree-based models during training

Answer

It computes importance from how many times a feature appears in splits, like MDI

Answer

Permutation importance always gives results identical to impurity-based importance

Question 8

What are SHAP values and what makes them uniquely powerful for feature importance?

Accepted Answer

SHAP values assign each feature a contribution to each individual prediction, using game theory

Answer

SHAP values only work with linear regression models

Answer

SHAP provides only a single global importance score with no per-sample detail

Answer

SHAP replaces cross-validation for model evaluation

Question 9

Can a feature with near-zero correlation to the target still have high feature importance? Explain why.

Accepted Answer

Yes, because feature importance captures non-linear and interaction effects that correlation misses

Answer

No, features with near-zero correlation always have near-zero feature importance

Answer

Correlation and feature importance always produce identical rankings

Answer

High correlation guarantees high feature importance in all model types

Question 10

What is Mean Decrease in Impurity (MDI) and what is its well-known bias?

Accepted Answer

MDI tends to overestimate importance for high-cardinality features due to more split opportunities

Answer

MDI is biased toward low-cardinality binary features

Answer

MDI is an unbiased estimator and never favors any feature type

Answer

MDI bias only affects regression trees, not classification trees

Question 11

What is Mean Decrease in Accuracy (MDA) and how is it computed in a Random Forest?

Accepted Answer

MDA measures the drop in out-of-bag accuracy when each feature's values are randomly shuffled

Answer

MDA measures how many times each feature is used in tree splits

Answer

MDA is computed using training set accuracy, which makes it prone to overfitting

Answer

MDA and MDI are computed identically and always give the same result

Question 12

In XGBoost, what is the difference between 'gain' and 'weight' (frequency) importance?

Accepted Answer

'Gain' measures the average improvement in the objective per split; 'weight' counts split frequency

Answer

'Weight' measures loss reduction per split; 'gain' counts how often the feature is used

Answer

Both 'gain' and 'weight' always rank features identically in XGBoost

Answer

XGBoost's 'gain' is identical to sklearn's MDI from Random Forest

Question 13

What is Information Gain and how is it used to rank features in Decision Trees?

Accepted Answer

Information gain measures the reduction in entropy when splitting on a feature

Answer

Information gain counts the number of samples in each child node after a split

Answer

Information gain is identical to Gini impurity and always produces the same splits

Answer

Information gain is only applicable to regression decision trees

Question 14

How does L1 (Lasso) regularization act as a feature selector through feature importance?

Accepted Answer

L1 regularization drives some coefficients to exactly zero, effectively eliminating those features

Answer

L1 regularization increases all feature coefficients proportionally

Answer

Both L1 and L2 regularization drive coefficients to exactly zero

Answer

L1 regularization only works for tree-based models

Question 15

What is Recursive Feature Elimination (RFE) and how does it leverage feature importance?

Accepted Answer

RFE repeatedly trains a model, removes the least important features, and retrains until done

Answer

RFE adds features one by one starting from the single most important feature

Answer

RFE removes all features below a fixed threshold in one step without retraining

Answer

RFE only works with linear regression models

Question 16

Why can multicollinearity mislead feature importance scores, and how does it affect different model types differently?

Accepted Answer

Multicollinearity splits importance between correlated features, making each appear less important individually

Answer

Multicollinearity has no effect on feature importance since models handle it automatically

Answer

Multicollinearity inflates the importance of all correlated features simultaneously

Answer

Multicollinearity only affects feature importance in neural networks

Question 17

What is the key difference between SHAP values and permutation importance in terms of granularity of explanation?

Accepted Answer

SHAP provides both per-prediction AND global explanations; permutation importance only gives global scores

Answer

Permutation importance gives per-prediction explanations; SHAP only provides global scores

Answer

Both SHAP and permutation importance always give identical feature rankings

Answer

SHAP values cannot be used with ensemble models like XGBoost or Random Forest

Question 18

How can we estimate feature importance for a neural network, which lacks a built-in importance attribute?

Accepted Answer

Model-agnostic methods like permutation importance or SHAP can estimate feature importance for neural networks

Answer

Neural networks output feature importances directly via the feature_importances_ attribute

Answer

Feature importance cannot be computed for neural networks under any circumstances

Answer

Only linear models can have feature importance computed

Question 19

Why does tree-based MDI importance overestimate the importance of high-cardinality continuous or categorical features?

Accepted Answer

MDI is biased toward high-cardinality features because they offer more split opportunities by chance

Answer

MDI is biased toward low-cardinality binary features because they are simpler to split

Answer

MDI gives perfectly unbiased importance regardless of the number of unique values

Answer

MDI bias only occurs in regression forests, not classification forests

Question 20

Does the type of problem (classification vs. regression) change how Random Forest computes feature importance?

Accepted Answer

The concept is the same but impurity measure differs: Gini/entropy for classification, MSE for regression

Answer

Feature importance uses Gini impurity for both classification and regression forests

Answer

Feature importance cannot be computed for regression Random Forests

Answer

Regression forests always produce higher importance scores than classification forests

Question 21

What is drop-column importance, and what makes it accurate but impractical for large feature sets?

Accepted Answer

Drop-column importance requires retraining the full model for each feature, making it computationally expensive

Answer

Drop-column importance is the fastest method for large feature sets

Answer

Drop-column importance only works for linear models

Answer

Drop-column importance never changes the model's performance compared to the full model

Question 22

Why can target encoding a high-cardinality categorical feature artificially inflate its feature importance?

Accepted Answer

Target encoding computed on the training set can inflate feature importance due to data leakage

Answer

Target encoding always reduces a feature's importance score below its true value

Answer

Target encoding has no effect on feature importance in any scenario

Answer

Target encoding only applies to regression problems, not classification

Question 23

How can using training-set feature importances for feature selection lead to overfitting?

Accepted Answer

Using training-set feature importance for selection can cause data leakage and overestimate performance

Answer

Overfitting cannot occur when using feature importance because it simplifies the model

Answer

Feature importance automatically prevents overfitting by removing noise features

Answer

Feature selection always improves generalization regardless of which data is used

Question 24

Why might feature importance rankings be unstable across different training runs, and how do you stabilize them?

Accepted Answer

Averaging importances across multiple runs or cross-validation folds improves stability

Answer

Feature importance is always perfectly stable since the tree algorithm is deterministic

Answer

Instability only occurs in neural networks, never in tree-based models

Answer

Using a larger dataset always fully eliminates importance instability

Question 25

What is a Partial Dependence Plot (PDP) and how does it complement feature importance scores?

Accepted Answer

PDP shows the direction and shape of a feature's effect; feature importance shows the magnitude

Answer

PDP and feature importance are the same thing displayed in different chart styles

Answer

PDP replaces the need to compute feature importance entirely

Answer

PDPs can only be computed for linear models

Question 26

What is variance threshold feature selection and when is it most useful?

Accepted Answer

Variance threshold removes near-constant features that have very low variance across samples

Answer

Variance threshold removes features that are highly correlated with the target variable

Answer

Variance threshold is a supervised method that requires the target variable to run

Answer

Variance threshold is only applicable to categorical features

Question 27

How do the coefficients of a standardized linear regression model represent feature importance?

Accepted Answer

Standardized linear model coefficients can serve as feature importance because scale differences are removed

Answer

Larger raw coefficients always indicate more important features regardless of feature scale

Answer

Linear model coefficients cannot be used to assess feature importance under any conditions

Answer

The feature with the smallest coefficient is always the most important in a linear model

Question 28

Does feature scaling (standardization or normalization) affect feature importance in a Random Forest? Why or why not?

Accepted Answer

Tree-based feature importance is unaffected by scaling because trees use value ordering, not magnitude

Answer

Scaling features always significantly changes tree-based feature importance scores

Answer

Standardization makes all tree-based feature importances equal to each other

Answer

Normalization is required before computing any Random Forest feature importance

Question 29

What is 'data leakage' in the context of feature importance, and give a concrete example?

Accepted Answer

Leaky features inflate their own importance by encoding target information not available at prediction time

Answer

Data leakage reduces feature importance scores for all features equally

Answer

Leaky features appear unimportant because they confuse the model

Answer

Data leakage only occurs in unsupervised learning problems

Question 30

Why is computing feature importance for time-series data more challenging than for independent tabular data?

Accepted Answer

Time series importance is harder because temporal dependencies break the assumptions of standard shuffling methods

Answer

Feature importance for time series is computed identically to i.i.d. data with no modifications

Answer

Randomly shuffling feature values works well for time series because time order does not matter

Answer

Tree-based models cannot be applied to time series data at all

Test yourself