Hyperparameter tuning quiz

Question 1

What is a hyperparameter in machine learning, and how does it differ from a model parameter?

Accepted Answer

A value set before training that controls the learning process, unlike parameters learned from data

Answer

A weight that the model learns automatically during training

Answer

A metric used to evaluate accuracy after training

Answer

A type of input feature in the dataset

Question 2

Why can't we simply learn hyperparameters from the training data the same way we learn model weights?

Accepted Answer

Because tuning them on training data favors overfitting, so they need a validation set to estimate generalization

Answer

Because hyperparameters cannot be represented numerically

Answer

Because the training algorithm is too slow to include them

Answer

Because hyperparameters are always fixed by law

Question 3

What is grid search, and what is its main disadvantage?

Accepted Answer

It tests all combinations exhaustively, but cost grows exponentially with the number of hyperparameters

Answer

It randomly samples values, which makes results unreproducible

Answer

It uses gradient descent on hyperparameters, which is unstable

Answer

It only works for neural networks

Question 4

How does random search differ from grid search, and why is it often more efficient?

Accepted Answer

It samples random combinations, exploring important hyperparameters more thoroughly per budget

Answer

It always tests more combinations than grid search

Answer

It guarantees finding the global optimum

Answer

It only changes one hyperparameter at a time

Question 5

What role does a validation set play in hyperparameter tuning?

Accepted Answer

It evaluates hyperparameter choices as a proxy for unseen data, keeping the test set untouched

Answer

It is used to learn the model weights

Answer

It is the same as the training set

Answer

It is only used after the model is deployed

Question 6

What is k-fold cross-validation and why is it commonly used during hyperparameter tuning?

Accepted Answer

It averages performance over k train/validation splits for a more reliable estimate

Answer

It trains the model k times faster

Answer

It removes the need for a test set entirely

Answer

It only works with k equal to the number of samples

Question 7

In gradient descent, what happens if the learning rate is set too high?

Accepted Answer

The loss may oscillate or diverge because updates overshoot the minimum

Answer

Training always converges faster with no downside

Answer

The model underfits because updates are too small

Answer

The learning rate has no effect on convergence

Question 8

What is the difference between underfitting and overfitting, and how do hyperparameters influence each?

Accepted Answer

Underfitting is too simple (high bias); overfitting is too complex (high variance); hyperparameters shift between them

Answer

Underfitting and overfitting are the same problem

Answer

Hyperparameters have no effect on either

Answer

Overfitting only happens with small learning rates

Question 9

Why is it important NOT to use the test set when selecting hyperparameters?

Accepted Answer

Using it for tuning leaks information and inflates the test score, hiding true generalization

Answer

The test set is too small to tune on

Answer

Tuning on the test set is fine if you only do it once

Answer

The test set has no labels

Question 10

What is Bayesian optimization in the context of hyperparameter tuning?

Accepted Answer

It builds a surrogate model of performance to choose promising configurations with fewer evaluations

Answer

It tries every combination exhaustively like grid search

Answer

It picks hyperparameters completely at random with no model

Answer

It only optimizes the learning rate

Question 11

What does the regularization strength hyperparameter (like alpha in Ridge regression) control?

Accepted Answer

How strongly large weights are penalized, controlling complexity and overfitting

Answer

The number of training epochs

Answer

The learning rate of gradient descent

Answer

The number of features selected

Question 12

In k-Nearest Neighbors, what is the effect of choosing a very small value of k versus a very large k?

Accepted Answer

Small k overfits (noise-sensitive); large k underfits (over-smoothed boundary)

Answer

Small k underfits; large k overfits

Answer

k has no effect on the decision boundary

Answer

Only k=1 ever works

Question 13

What is nested cross-validation and why is it used?

Accepted Answer

An inner loop tunes hyperparameters and an outer loop gives an unbiased performance estimate

Answer

It is a faster version of single cross-validation

Answer

It uses the test set in both loops

Answer

It only applies to neural networks

Question 14

What is early stopping, and how does it act as a hyperparameter tuning technique?

Accepted Answer

It stops training when validation performance stops improving, tuning the number of epochs

Answer

It stops training as soon as training loss reaches zero

Answer

It removes features that are not useful

Answer

It increases the learning rate over time

Question 15

Why are some hyperparameters, like learning rate or regularization strength, often searched on a logarithmic scale?

Accepted Answer

Because their effect is multiplicative and they span many orders of magnitude

Answer

Because logarithms are faster to compute

Answer

Because they must always be negative

Answer

Because only log values are valid hyperparameters

Question 16

In a decision tree, what does the max_depth hyperparameter control and how does it affect overfitting?

Accepted Answer

It limits tree levels; large depth overfits, small depth underfits

Answer

It sets the number of features per split only

Answer

It controls the learning rate of the tree

Answer

It has no effect on overfitting

Question 17

What is the main trade-off when deciding how many configurations to evaluate during hyperparameter search?

Accepted Answer

Between compute cost and the quality/likelihood of finding a good configuration

Answer

Between accuracy and the number of features

Answer

Between training and test set sizes

Answer

There is no trade-off; more is always free

Question 18

What is the 'curse of dimensionality' as it applies to grid search over hyperparameters?

Accepted Answer

Combinations grow exponentially as more hyperparameters are added, making grid search infeasible

Answer

The data becomes too small to train on

Answer

The learning rate becomes unstable

Answer

Models lose accuracy with more dimensions automatically

Question 19

What is the difference between RandomizedSearchCV's n_iter and GridSearchCV's behavior in scikit-learn?

Accepted Answer

n_iter caps the number of sampled combinations; grid search runs all combinations

Answer

n_iter sets the number of CV folds only

Answer

Both run the same fixed number of fits

Answer

n_iter controls the learning rate

Question 20

Why should hyperparameter tuning typically be combined with a fixed random seed when comparing methods?

Accepted Answer

To make results reproducible so differences reflect hyperparameters, not randomness

Answer

To make training run faster

Answer

To increase model accuracy directly

Answer

Because seeds are required by all algorithms

Question 21

What is the relationship between batch size and learning rate when training neural networks?

Accepted Answer

Larger batches give stabler gradients, often allowing a larger learning rate; they are tuned together

Answer

Batch size and learning rate are completely independent

Answer

Smaller batches always need larger learning rates

Answer

Batch size only affects memory, never the learning rate

Question 22

What is successive halving (and the Hyperband algorithm) in hyperparameter optimization?

Accepted Answer

It runs many configs cheaply, keeps the best, and gives them more resources iteratively

Answer

It trains all configs to completion before comparing

Answer

It is another name for grid search

Answer

It only tunes the number of layers

Question 23

In scikit-learn's GridSearchCV, what does the best_estimator_ attribute give you after fitting?

Accepted Answer

The model refit on all training data using the best-scoring hyperparameters

Answer

The list of every parameter combination tried

Answer

The validation accuracy as a number

Answer

The raw training data

Question 24

What is the role of the n_estimators hyperparameter in a Random Forest, and what is the trade-off when increasing it?

Accepted Answer

More trees reduce variance but increase compute, with diminishing accuracy gains

Answer

More trees always cause overfitting quickly

Answer

Fewer trees are always better

Answer

n_estimators sets the tree depth

Question 25

Why might tuning hyperparameters on a model give diminishing returns compared to improving the data or features?

Accepted Answer

Tuning optimizes use of existing data, while better data/features raise the achievable ceiling

Answer

Tuning always beats improving data

Answer

Data quality never affects model performance

Answer

Hyperparameters can fix any data problem

Question 26

What does the C hyperparameter control in a Support Vector Machine (SVM)?

Accepted Answer

The trade-off between margin width and training errors (inverse regularization)

Answer

The number of support vectors to keep

Answer

The kernel type used

Answer

The learning rate of the SVM

Question 27

What is a hyperparameter search space, and why does defining it well matter?

Accepted Answer

The allowed hyperparameters and value ranges; a good one balances coverage and efficiency

Answer

The set of all training examples

Answer

The final accuracy of the model

Answer

The number of CV folds used

Question 28

How does scikit-learn's cv parameter in GridSearchCV affect the total number of models trained?

Accepted Answer

Total fits = (combinations x cv) plus a final refit, so higher cv multiplies cost

Answer

cv only affects the final accuracy, not the number of fits

Answer

cv divides the number of combinations

Answer

cv sets the learning rate

Question 29

What is the danger of overfitting to the validation set during extensive hyperparameter tuning?

Accepted Answer

Repeatedly picking the best on one validation set can fit noise, inflating validation scores

Answer

It makes training slower but is otherwise harmless

Answer

It only affects the training set

Answer

Validation sets cannot be overfit

Question 30

In gradient boosting (e.g., XGBoost), how do the learning rate and number of estimators interact?

Accepted Answer

A smaller learning rate needs more estimators but often generalizes better; they are tuned together

Answer

They are independent and never affect each other

Answer

A larger learning rate always needs more trees

Answer

Number of estimators sets the tree depth

Test yourself