Cross-validation and data splits quiz

Question 1

Why do we split a dataset into a training set and a test set instead of training and evaluating on all the data?

Accepted Answer

To estimate how well the model generalizes to unseen data

Answer

To make training run faster on a smaller set

Answer

To increase the model's accuracy automatically

Answer

Because algorithms require exactly two datasets

Question 2

What is the main purpose of a validation set, as distinct from the test set?

Accepted Answer

To tune hyperparameters and select models during development

Answer

To train the final model parameters

Answer

To replace the training set when it is too small

Answer

To store data the model never uses

Question 3

In k-fold cross-validation, how is the dataset used across the k iterations?

Accepted Answer

Each fold is used once as validation while the rest train, repeated k times

Answer

One fold trains and the rest validate, repeated k times

Answer

All folds train and validate simultaneously

Answer

Folds are chosen randomly with replacement each round

Question 4

What is a key advantage of k-fold cross-validation over a single train/test split?

Accepted Answer

It gives a more reliable estimate by averaging over multiple splits

Answer

It always produces higher accuracy than a single split

Answer

It removes the need for any test data

Answer

It guarantees the model will not overfit

Question 5

What does stratified k-fold cross-validation preserve that ordinary k-fold may not?

Accepted Answer

The class proportions in each fold

Answer

The original order of the rows

Answer

The exact number of features per fold

Answer

The same random seed across folds

Question 6

What is Leave-One-Out Cross-Validation (LOOCV)?

Accepted Answer

k-fold CV where k equals the number of samples

Answer

A method that leaves out one whole feature each round

Answer

Training on one point and testing on the rest

Answer

A split that discards one outlier before training

Question 7

What is data leakage in the context of data splitting?

Accepted Answer

Test or future information leaking into training, inflating scores

Answer

The dataset accidentally being deleted during splitting

Answer

Random noise added to the training labels

Answer

Using too many features in the model

Question 8

Why should feature scaling (e.g., standardization) be fit only on the training set, not the whole dataset?

Accepted Answer

To avoid leaking test-set statistics into the model

Answer

Because scaling the test set is impossible

Answer

To make the test set larger

Answer

Because scaling slows down training otherwise

Question 9

Why is random shuffling usually applied before creating a train/test split?

Accepted Answer

To ensure both sets are representative of the whole dataset

Answer

To increase the total amount of data

Answer

To remove duplicate rows automatically

Answer

To sort the data by the target value

Question 10

For time-series data, why is ordinary random k-fold cross-validation inappropriate?

Accepted Answer

It lets the model train on future data to predict the past

Answer

Time-series data cannot be split at all

Answer

It always uses too few folds

Answer

Random folds make training too slow

Question 11

What does the test_size parameter control in scikit-learn's train_test_split?

Accepted Answer

The proportion of data allocated to the test set

Answer

The number of features to keep

Answer

The learning rate of the model

Answer

The number of cross-validation folds

Question 12

What is the role of the random_state argument when splitting data?

Accepted Answer

It makes the split reproducible by fixing the random seed

Answer

It sets how many random features are dropped

Answer

It controls the model's accuracy

Answer

It decides the number of folds

Question 13

In a three-way split, what is each of the train/validation/test sets typically used for?

Accepted Answer

Train fits parameters, validation tunes, test gives final score

Answer

Train tunes, validation fits, test trains

Answer

All three are used to fit parameters

Answer

Train and test tune while validation reports

Question 14

What is nested cross-validation and why is it used?

Accepted Answer

An inner loop tunes hyperparameters while an outer loop estimates performance

Answer

Two test sets used at the same time

Answer

Running CV twice on the same fold split

Answer

Cross-validating only the test set

Question 15

What trade-off do you face when choosing the number of folds k in k-fold cross-validation?

Accepted Answer

Larger k lowers bias but costs more computation

Answer

Larger k always lowers both bias and variance for free

Answer

Smaller k always gives the most accurate estimate

Answer

The choice of k has no effect on the estimate

Question 16

If a model scores 98% on training data but 70% on the test set, what does this most likely indicate?

Accepted Answer

The model is overfitting the training data

Answer

The model is underfitting

Answer

The test set is corrupted

Answer

The model is perfectly generalizing

Question 17

Why should the test set be used only once, at the very end of the modeling process?

Accepted Answer

Repeated use turns it into tuning data and biases the estimate

Answer

Using it multiple times damages the data

Answer

The test set can only be loaded once technically

Answer

It saves memory to use it once

Question 18

What is the purpose of the stratify argument in scikit-learn's train_test_split?

Accepted Answer

It preserves class proportions across train and test sets

Answer

It sorts the data before splitting

Answer

It increases the test size automatically

Answer

It removes minority-class samples

Question 19

How does cross-validation help make better use of a small dataset?

Accepted Answer

Every example is used for both training and validation across folds

Answer

It generates new synthetic samples

Answer

It removes the need for a training set

Answer

It duplicates the data to make it larger

Question 20

When tuning hyperparameters with GridSearchCV, on which data does it perform cross-validation?

Accepted Answer

On the training data passed to fit, keeping the test set aside

Answer

On the test set directly

Answer

On the full dataset including test

Answer

On a separate random dataset it creates

Question 21

What is the holdout method of model evaluation?

Accepted Answer

A single split into training and a held-out test set

Answer

Holding out one feature for testing

Answer

Repeatedly resampling the data with replacement

Answer

Training on the test set and testing on training

Question 22

What is repeated k-fold cross-validation and what benefit does it provide?

Accepted Answer

Running k-fold several times with different partitions and averaging

Answer

Using the same folds twice in a row

Answer

Doubling the number of folds

Answer

Repeating only the worst-performing fold

Question 23

In a 5-fold cross-validation on 1000 samples, how many samples are used for training in each fold?

Accepted Answer

800 samples for training, 200 for validation

Answer

200 samples for training, 800 for validation

Answer

1000 samples for training each round

Answer

500 samples for training each round

Question 24

Why can evaluating only on training data give a misleadingly optimistic accuracy?

Accepted Answer

The model has already seen and fit those exact examples

Answer

Training data is always cleaner than test data

Answer

Accuracy is computed differently on training data

Answer

Training sets are always larger

Question 25

When using a scikit-learn Pipeline inside cross-validation, why does it prevent leakage?

Accepted Answer

Preprocessing is fit only on each fold's training portion

Answer

It skips preprocessing on the test fold entirely

Answer

It uses the same scaler for every dataset

Answer

It removes the need for cross-validation

Question 26

What problem can arise if duplicate or near-duplicate records end up in both the training and test sets?

Accepted Answer

Leakage that makes the test score overly optimistic

Answer

The model trains too slowly

Answer

The test set becomes too small

Answer

Accuracy drops below random chance

Question 27

What is group k-fold (GroupKFold) cross-validation designed to handle?

Accepted Answer

Keeping all samples of one group within a single fold

Answer

Grouping features instead of samples

Answer

Making every fold the same size exactly

Answer

Splitting only the largest group

Question 28

How is the final cross-validation score typically reported?

Accepted Answer

The mean of fold scores, often with the standard deviation

Answer

Only the single best fold's score

Answer

The sum of all fold scores

Answer

The score from the last fold only

Question 29

Which scenario is the strongest argument for using cross-validation rather than a single holdout split?

Accepted Answer

A small dataset where a single split would be unreliable

Answer

A massive dataset with millions of rows

Answer

When you want the fastest possible evaluation

Answer

When the data has no labels at all

Question 30

What does it mean if cross-validation scores vary a lot across folds (high standard deviation)?

Accepted Answer

The model's performance is unstable and split-sensitive

Answer

The model is perfectly reliable

Answer

The dataset is definitely too large

Answer

Cross-validation was done incorrectly every time

Test yourself