Classical ML algorithms quiz

Question 1

In gradient-boosted decision trees (e.g., XGBoost), what role does the second-order Taylor expansion of the loss play?

Accepted Answer

It uses gradient and Hessian to compute Newton-like leaf weights and split gains

Answer

It uses only the gradient to fit residuals like classic gradient boosting

Answer

It performs exact loss minimization with no approximation

Answer

It expands the loss only to regularize tree depth

Question 2

Why does the bias-variance decomposition favor bagging for reducing variance but boosting for reducing bias?

Accepted Answer

Bagging reduces variance; boosting reduces bias

Answer

Bagging reduces bias; boosting reduces variance

Answer

Both reduce only variance

Answer

Both reduce only bias

Question 3

What is the effect of the L1 (Lasso) penalty on the solution path compared to L2 (Ridge) in linear models?

Accepted Answer

L1 yields exact zeros (sparsity); L2 shrinks without exact zeros

Answer

L1 shrinks smoothly; L2 yields exact zeros

Answer

Both yield exact zeros equally often

Answer

Neither penalty shrinks coefficients

Question 4

In a Support Vector Machine, what does the kernel trick allow you to avoid computing explicitly?

Accepted Answer

The explicit feature mapping phi(x) into high-dimensional space

Answer

The computation of support vectors

Answer

The margin maximization objective

Answer

The Lagrange multipliers

Question 5

Why is feature scaling critical for k-Nearest Neighbors and SVMs but not for decision trees?

Accepted Answer

kNN/SVM need scaling because they use distances; trees are scale-invariant

Answer

Trees need scaling but kNN/SVM do not

Answer

All three require scaling equally

Answer

None of them are affected by scaling

Question 6

What does the 'naive' assumption in Naive Bayes refer to, and why does the classifier often still work well?

Accepted Answer

Features are conditionally independent given the class; argmax is robust to this violation

Answer

Features are assumed Gaussian; it works due to CLT

Answer

Classes are assumed equiprobable; priors are ignored

Answer

Features are assumed independent unconditionally; it fails otherwise

Question 7

How does the choice of K in K-Means relate to the within-cluster sum of squares (WCSS / inertia) and the elbow method?

Accepted Answer

Inertia decreases with K; the elbow marks diminishing returns

Answer

Inertia increases with K, so pick the maximum

Answer

Inertia is constant in K; use silhouette only

Answer

Inertia is minimized at the optimal K, so minimize it

Question 8

In logistic regression, what does a coefficient beta_j represent in terms of the odds?

Accepted Answer

The change in log-odds per unit x_j; exp(beta_j) is the odds ratio

Answer

The change in probability per unit x_j

Answer

The correlation between x_j and y

Answer

The variance explained by x_j

Question 9

What distinguishes Principal Component Analysis (PCA) from Linear Discriminant Analysis (LDA) for dimensionality reduction?

Accepted Answer

PCA maximizes variance (unsupervised); LDA maximizes class separability (supervised)

Answer

Both are supervised; PCA maximizes class separation

Answer

PCA is supervised; LDA is unsupervised

Answer

They are identical up to scaling

Question 10

Why can random forests provide a built-in estimate of generalization error without a separate validation set?

Accepted Answer

About 37% of samples are out-of-bag per tree, enabling OOB error estimation

Answer

Trees are trained on all data, so error is exact

Answer

Random forests use cross-validation internally by default

Answer

The error is estimated from feature importances

Question 11

What problem does the kernel bandwidth (gamma) control in an RBF-kernel SVM, and what happens at extremes?

Accepted Answer

Large gamma overfits (narrow influence); small gamma underfits (broad influence)

Answer

Large gamma underfits; small gamma overfits

Answer

Gamma only affects training speed, not the boundary

Answer

Gamma controls the soft-margin penalty like C

Question 12

In gradient boosting, why is a small learning rate (shrinkage) combined with more trees usually preferred?

Accepted Answer

Small eta with more trees regularizes and improves generalization

Answer

Large eta with few trees always generalizes best

Answer

eta has no effect on overfitting

Answer

Small eta requires fewer trees

Question 13

What is the key difference in how Gini impurity and entropy behave as splitting criteria in decision trees?

Accepted Answer

They behave similarly; entropy uses a log and is slightly costlier

Answer

Gini and entropy produce drastically different trees

Answer

Entropy cannot be used for classification

Answer

Gini measures variance, not impurity

Question 14

How does the elastic net penalty combine L1 and L2, and when is it preferred over pure Lasso?

Accepted Answer

It blends L1 and L2; preferred with correlated features and p>>n

Answer

It uses only L2; preferred for sparsity

Answer

It blends L1 and L2 but worsens correlated-feature handling

Answer

It is identical to Lasso when rho=0

Question 15

Why does the curse of dimensionality degrade k-Nearest Neighbors performance?

Accepted Answer

Distances concentrate, making neighbors nearly equidistant and kNN unreliable

Answer

High dimensions make distances more discriminative

Answer

kNN is unaffected by dimensionality

Answer

Only training time increases, accuracy is unchanged

Question 16

In AdaBoost, how are training examples reweighted between rounds?

Accepted Answer

Misclassified examples get higher weight; correct ones lower

Answer

All examples keep equal weight throughout

Answer

Correctly classified examples get higher weight

Answer

Weights are random each round

Question 17

What is the difference between generative and discriminative classifiers, with examples?

Accepted Answer

Generative learn P(x,y); discriminative learn P(y|x) or the boundary

Answer

Generative learn P(y|x); discriminative learn P(x,y)

Answer

Both learn P(x,y) identically

Answer

Discriminative models cannot do classification

Question 18

Why does K-Means converge to a local optimum, and how does k-means++ help?

Accepted Answer

The objective is non-convex; k-means++ spreads seeds to avoid poor local optima

Answer

K-Means is convex and finds the global optimum

Answer

k-means++ guarantees the global optimum

Answer

Initialization does not affect the result

Question 19

What does multicollinearity do to ordinary least squares coefficient estimates, and how is it detected?

Accepted Answer

It inflates coefficient variance; detected via VIF or condition number

Answer

It biases predictions but stabilizes coefficients

Answer

It has no effect on OLS

Answer

It always reduces training error to zero

Question 20

How does DBSCAN differ from K-Means in handling cluster shape and noise?

Accepted Answer

DBSCAN finds arbitrary shapes, marks noise, and needs no K; K-Means assumes spherical clusters

Answer

Both require K and assume spherical clusters

Answer

DBSCAN requires K; K-Means handles noise

Answer

DBSCAN only works on spherical clusters

Question 21

What is the purpose of the C hyperparameter in soft-margin SVM?

Accepted Answer

C trades margin width against slack; large C overfits, small C regularizes

Answer

Large C widens the margin and underfits

Answer

C controls the kernel bandwidth

Answer

C sets the number of support vectors directly

Question 22

Why is accuracy a misleading metric on imbalanced datasets, and what alternatives are better?

Accepted Answer

Majority-class prediction inflates accuracy; use F1/PR-AUC/MCC instead

Answer

Accuracy is always reliable regardless of balance

Answer

Only ROC-AUC should ever be used

Answer

Recall is identical to accuracy

Question 23

In a confusion-matrix context, define precision and recall and the trade-off the threshold induces.

Accepted Answer

Precision=TP/(TP+FP); recall=TP/(TP+FN); threshold trades them off

Answer

Precision=TP/(TP+FN); recall=TP/(TP+FP)

Answer

Both equal TP/(TP+TN)

Answer

Threshold changes neither precision nor recall

Question 24

What assumption about the error distribution does ordinary least squares require for its estimator to be BLUE (Gauss-Markov)?

Accepted Answer

Homoscedastic, uncorrelated, zero-mean errors give BLUE; normality not required

Answer

Errors must be normally distributed for BLUE

Answer

Only normality is required

Answer

OLS is biased under these assumptions

Question 25

How does the EM algorithm fit a Gaussian Mixture Model, and what does each step do?

Accepted Answer

E-step computes responsibilities; M-step updates means/covariances/weights

Answer

E-step updates means; M-step assigns points

Answer

Both steps update only the mixing weights

Answer

EM maximizes likelihood globally in one step

Question 26

What does it mean that decision trees are high-variance, and how do ensembles address this?

Accepted Answer

Trees are high-variance; averaging decorrelated trees reduces it

Answer

Trees are low-variance; ensembles add variance

Answer

Trees are unbiased and ensembles fix bias only

Answer

Variance is irrelevant to trees

Question 27

Why might you choose stratified k-fold cross-validation over plain k-fold?

Accepted Answer

Stratified folds preserve class proportions, reducing estimate variance

Answer

Stratified folds randomize classes more

Answer

Stratified k-fold is only for regression

Answer

Plain k-fold always preserves class balance

Question 28

What is the difference between bagging and boosting regarding how base learners are trained?

Accepted Answer

Bagging trains learners in parallel on bootstraps; boosting trains them sequentially on errors

Answer

Both train sequentially on residuals

Answer

Bagging is sequential; boosting is parallel

Answer

They are the same algorithm

Question 29

How does Laplace (additive) smoothing prevent zero-probability problems in Naive Bayes?

Accepted Answer

It adds pseudocounts so no probability is zero

Answer

It removes rare features entirely

Answer

It normalizes features to unit variance

Answer

It only affects the class priors

Question 30

In XGBoost/LightGBM, what does the regularization term lambda (L2 on leaf weights) and gamma (min split loss) each control?

Accepted Answer

Lambda is L2 on leaf weights; gamma is the minimum gain to allow a split

Answer

Lambda sets tree depth; gamma sets learning rate

Answer

Both control the learning rate

Answer

Gamma is L1 on weights; lambda is row subsampling

Test yourself