Dropout and preventing overfitting quiz

Question 1

What does it mean for a machine learning model to 'overfit' the training data?

Accepted Answer

The model learns training data including noise, performing well on training but poorly on unseen data

Answer

The model performs equally well on training and test data

Answer

The model is too simple to capture patterns in the data

Answer

The model trains too slowly to converge

Question 2

A model achieves 55% accuracy on training data and 54% accuracy on test data for a 10-class problem. Which condition does this most likely represent?

Accepted Answer

Underfitting — the model is too simple to capture the data's patterns

Answer

Overfitting — training accuracy is low because the model memorised noise

Answer

Good generalisation — similar train and test accuracy is always desirable

Answer

Data leakage — the test set was accidentally included in training

Question 3

In the bias-variance tradeoff, what happens to bias and variance as you increase model complexity (e.g., adding more layers to a neural network)?

Accepted Answer

Bias decreases and variance increases with complexity

Answer

Both bias and variance increase with complexity

Answer

Bias increases and variance decreases with complexity

Answer

Both bias and variance decrease with complexity

Question 4

What is dropout in neural networks, and at which phase of the learning pipeline is it applied?

Accepted Answer

Randomly deactivating neurons during training with probability p; disabled at inference

Answer

Randomly removing training examples from the dataset before each epoch

Answer

Permanently pruning the weakest neurons after training completes

Answer

Reducing the learning rate by a random factor during training

Question 5

Why is dropout turned off during inference, and how does 'inverted dropout' ensure predictions stay correctly scaled?

Accepted Answer

All neurons are active at inference; inverted dropout scales activations during training so no test-time change is needed

Answer

Dropout is kept on at inference by default to add uncertainty

Answer

All neurons are active at inference and the output is multiplied by (1−p) to compensate

Answer

Half the neurons stay active at inference to approximate the training average

Question 6

In Keras, what does Dropout(0.3) mean, and what dropout rates are typically recommended for fully-connected hidden layers?

Accepted Answer

Each neuron has a 30% chance of being turned off during each training step

Answer

30% of training examples are dropped from each mini-batch

Answer

The learning rate is reduced by 30% at each step

Answer

30% of weights are permanently zeroed after training

Question 7

Intuitively, why does randomly dropping neurons during training prevent neurons from 'co-adapting' and reduce overfitting?

Accepted Answer

It forces neurons to learn independently useful features by preventing co-adaptation

Answer

It permanently reduces the number of parameters, making the model simpler

Answer

It increases the learning rate so the model escapes sharp overfitting minima

Answer

It removes noisy training examples automatically each epoch

Question 8

In inverted dropout, if the keep probability k = 0.8 (dropout rate = 0.2), what scaling is applied to the surviving activations and why?

Accepted Answer

Surviving activations are multiplied by 1/0.8 = 1.25 to preserve expected magnitude

Answer

Surviving activations are multiplied by 0.8 to keep values small

Answer

Surviving activations are left unchanged; only zero-valued neurons are adjusted

Answer

Surviving activations are multiplied by 0.2 (the dropout rate)

Question 9

What is L2 regularisation (also called weight decay), and how does it modify the loss function to reduce overfitting?

Accepted Answer

It adds a penalty equal to λ times the sum of squared weights to the loss

Answer

It adds a penalty equal to the sum of absolute values of weights (|w|)

Answer

It randomly drops neurons during each forward pass

Answer

It clips all gradients to a fixed maximum value

Question 10

How does L1 regularisation differ from L2, and which one naturally drives many weights to exactly zero?

Accepted Answer

L1 uses absolute weights and drives many to exactly zero (sparsity); L2 uses squared weights and shrinks all but rarely zeroes them

Answer

L1 uses squared weights; L2 uses absolute weights — they are simply swapped names

Answer

L1 increases model complexity; L2 reduces it

Answer

L1 and L2 behave identically; the only difference is the default λ value

Question 11

With L2 regularisation (λ = 0.01) and learning rate η = 0.1, a weight w = 2.0 has a data-loss gradient ∂L/∂w = 0.5. What is the updated weight after one gradient-descent step?

Accepted Answer

1.946 (data gradient plus L2 penalty gradient 2λw)

Answer

1.95 (only the data gradient is applied, ignoring the L2 term)

Answer

1.948 (λ applied once without the factor of 2, giving total gradient 0.52)

Answer

1.99 (regularisation incorrectly added to the weight instead of subtracted)

Question 12

What is early stopping, and how does monitoring the validation loss help decide when to stop training?

Accepted Answer

Stopping training when validation loss stops improving even if training loss still decreases

Answer

Stopping as soon as training loss reaches zero

Answer

Training for a fixed number of epochs regardless of performance

Answer

Halving the learning rate to zero at a predetermined scheduled step

Question 13

How does data augmentation help prevent overfitting in image classification, and give three common augmentation transformations?

Accepted Answer

It creates new training examples via transformations like flips, crops, and colour jitter

Answer

It removes duplicate images to make training data cleaner

Answer

It reduces model size to prevent memorisation

Answer

It filters out mislabelled images from the dataset automatically

Question 14

How does batch normalisation differ from dropout as a way to improve generalisation, and can they be used together in the same network?

Accepted Answer

Batch norm normalises activations to stabilise training; dropout adds noise by zeroing neurons — both can be combined

Answer

They are fully interchangeable; only one should ever be used in a model

Answer

Batch norm is for RNNs only; dropout is for CNNs only

Answer

Batch norm increases overfitting while dropout reduces it

Question 15

What is k-fold cross-validation and why does it give a more reliable overfitting signal than a single train/validation split?

Accepted Answer

Splitting into k folds, training k times each with a different fold as validation, and averaging performance

Answer

Training the model k times on the full dataset with different random seeds

Answer

Removing k% of the data as a permanent held-out test set

Answer

Running k different model architectures and picking the best one

Question 16

Why do we need three separate data splits — training, validation, and test — when building and evaluating a machine learning model?

Accepted Answer

Training set trains weights; validation set tunes hyperparameters; test set gives final unbiased performance estimate

Answer

All three sets are combined to maximise the total amount of data available for training

Answer

Training and validation sets are the same thing; only the test set is different

Answer

The test set is used to make final weight updates after training on the other two sets

Question 17

When plotting training loss and validation loss against training epochs, what pattern in the curves is the clearest sign of overfitting?

Accepted Answer

Training loss falls to near zero while validation loss rises or plateaus at a much higher value

Answer

Both training and validation loss decrease together and converge at a low value

Answer

Training loss increases while validation loss decreases

Answer

Both training and validation loss increase steadily throughout training

Question 18

Standard SGD with L2 regularisation and AdamW both implement 'weight decay'. Why does AdamW behave differently from simply adding an L2 penalty term to the loss when using Adam?

Accepted Answer

In Adam, L2 loss gradient is scaled down by the adaptive factor; AdamW applies decay directly to weights, bypassing that scaling

Answer

They are completely different and unrelated techniques

Answer

Weight decay increases the learning rate while L2 decreases it

Answer

Weight decay only applies to bias terms; L2 regularisation applies to all weights

Question 19

How do ensemble methods like Random Forests use bagging to reduce overfitting compared to a single deep decision tree?

Accepted Answer

Many diverse trees are trained on bootstrap samples and averaged; uncorrelated errors cancel, reducing variance

Answer

Ensemble trees are deliberately shallow so they cannot overfit

Answer

They apply L2 regularisation to each tree's leaf predictions

Answer

Ensemble methods increase bias to prevent overfitting

Question 20

In which parts of a deep neural network is dropout most effective, and where is it typically NOT applied?

Accepted Answer

Most effective in large fully-connected hidden layers; not applied to input, output, or batch-norm layers

Answer

Applied only to the output layer to prevent overconfident predictions

Answer

Applied uniformly to every layer including the input and output layers

Answer

Applied only to the very first hidden layer

Question 21

Why is standard dropout less effective in convolutional layers, and what does 'spatial dropout' do differently to address this?

Accepted Answer

Standard dropout is ineffective in CNNs because correlated pixels leak information; spatial dropout drops entire feature maps (channels)

Answer

Standard dropout drops whole channels; spatial dropout drops individual pixels

Answer

Spatial dropout is the same as standard dropout but with a higher rate

Answer

Standard dropout works better in CNNs; spatial dropout is designed for RNNs

Question 22

What is Monte Carlo (MC) Dropout, and how does keeping dropout active at inference time allow a model to express uncertainty?

Accepted Answer

Running multiple forward passes with dropout active at inference to estimate prediction uncertainty

Answer

Running the model once with all dropout disabled for faster inference

Answer

Using dropout to select the most informative training examples for active learning

Answer

Applying dropout only to the final dense layer for uncertainty in classification

Question 23

What is max-norm regularisation, and how does it differ from L2 regularisation in the way it constrains weights?

Accepted Answer

It constrains the L2 norm of each neuron's incoming weight vector to be at most c, rescaling only when exceeded

Answer

It clips each individual weight scalar to the range [−c, c]

Answer

It is identical to L2 regularisation but applied only to the output layer

Answer

It sets the maximum allowed learning rate to c

Question 24

How does DropConnect generalise standard dropout, and what specifically does it zero out during training?

Accepted Answer

DropConnect randomly zeros individual weight connections during training rather than entire neuron activations

Answer

DropConnect drops entire training examples from the mini-batch rather than neurons or weights

Answer

DropConnect applies dropout only to the connections between the last two layers

Answer

DropConnect skips random layers entirely during the forward pass

Question 25

Besides standard dropout, name and briefly describe three other noise-based or perturbation-based regularisation techniques used in deep learning.

Accepted Answer

Input noise injection, weight noise, and label smoothing are all noise-based regularisation techniques

Answer

Only dropout can add noise; other techniques always remove noise from the model

Answer

Noise can only be injected into the output layer; nowhere else

Answer

Noise is always harmful and must be minimised throughout the training pipeline

Question 26

What is label smoothing and how does it prevent overconfident predictions that contribute to overfitting?

Accepted Answer

Replacing hard one-hot labels with soft targets that assign small probabilities to wrong classes

Answer

Removing incorrectly labelled examples from the training set

Answer

Using a lower learning rate to prevent sharp weight changes near convergence

Answer

Reducing the number of output classes to simplify the problem

Question 27

Consider the Keras model below. What happens to the Dropout(0.5) layers when you call model.predict() versus model.fit()?

Accepted Answer

Dropout zeros 50% of neurons during training but is automatically disabled during prediction; all neurons are fully active

Answer

Dropout is active with rate 0.5 during both training and prediction

Answer

Dropout is active during training but outputs are halved at prediction time

Answer

Dropout is applied only during the first epoch of training, then disabled

Question 28

If you increase the dropout rate from 0.3 to 0.7 in a network that was previously overfitting, what effects would you expect on training accuracy, validation accuracy, and training speed?

Accepted Answer

Training accuracy drops; validation accuracy may improve initially but can worsen if underfitting occurs; training is slower

Answer

Training accuracy increases because the network is forced to use fewer neurons more efficiently

Answer

Both training and validation accuracy always improve with a higher dropout rate

Answer

Higher dropout rate has no effect on accuracy, only on wall-clock training speed

Question 29

What is the Lottery Ticket Hypothesis, and why does it suggest that overparameterised networks are not wasteful?

Accepted Answer

Overparameterised networks contain small subnetworks ('winning tickets') that match full-network accuracy when trained from their original initialisation

Answer

Large networks always need all their neurons; sparse networks always underperform full ones

Answer

Dropout randomly selects the winning ticket at inference time to speed up prediction

Answer

Lottery-style random initialisation always beats careful techniques like Xavier/He initialisation

Question 30

Why is collecting more labelled training data generally considered the most fundamental way to reduce overfitting, compared to regularisation tricks like dropout?

Accepted Answer

More data provides diverse examples that make memorisation hard, addressing the root cause of overfitting

Answer

More data increases model complexity, allowing it to fit more complex patterns

Answer

More data slows training, which naturally reduces overfitting as a side effect

Answer

More data is less effective than regularisation; dropout always outperforms adding more examples

Test yourself