Learning rate schedules quiz

Question 1

What is a learning rate schedule in the context of training neural networks?

Accepted Answer

A rule that changes the learning rate during training

Answer

A fixed number that never changes

Answer

The number of layers in the network

Answer

The size of the training dataset

Question 2

Why might a constant learning rate that is too high cause problems late in training?

Accepted Answer

It causes updates to overshoot and oscillate near the minimum

Answer

It makes the model train too slowly from the start

Answer

It increases the number of parameters

Answer

It has no effect on convergence

Question 3

What is step decay (also called step-based) learning rate scheduling?

Accepted Answer

The rate is multiplied by a factor at fixed epoch intervals

Answer

The rate increases steadily each epoch

Answer

The rate is random at each step

Answer

The rate is tied to the batch size

Question 4

In exponential decay, how does the learning rate change with the epoch number t?

Accepted Answer

lr = lr0 * exp(-k*t)

Answer

lr = lr0 + k*t

Answer

lr = lr0 * t

Answer

lr = lr0 / exp(k)

Question 5

What is a learning rate warmup and why is it used?

Accepted Answer

Gradually increasing the LR from small to target at the start

Answer

Setting the LR to its maximum immediately

Answer

Removing the learning rate entirely

Answer

Decreasing the LR to zero at the start

Question 6

Describe cosine annealing as a learning rate schedule.

Accepted Answer

LR follows a half cosine curve from max down to a minimum

Answer

LR increases linearly forever

Answer

LR stays constant then jumps up

Answer

LR is chosen randomly each epoch

Question 7

What does 'cosine annealing with warm restarts' add to plain cosine annealing?

Accepted Answer

It periodically resets the LR to the maximum and anneals again

Answer

It never lets the LR decrease

Answer

It removes the cosine shape

Answer

It fixes the LR after one cycle

Question 8

How does the 'reduce on plateau' schedule decide when to lower the learning rate?

Accepted Answer

When a monitored metric stops improving for 'patience' epochs

Answer

At a fixed epoch decided before training

Answer

Whenever the batch size changes

Answer

Only at the very last epoch

Question 9

In a linear decay schedule, what is the learning rate at the final step if it decays linearly to zero?

Accepted Answer

Zero (or the specified end value)

Answer

Equal to the initial value

Answer

Twice the initial value

Answer

Undefined

Question 10

What is the '1cycle' policy proposed by Leslie Smith?

Accepted Answer

LR rises to a max then falls in one cycle, with inverse momentum

Answer

LR stays constant throughout

Answer

LR only ever decreases

Answer

LR is set per layer randomly

Question 11

What is the purpose of a learning rate range test (LR finder)?

Accepted Answer

To find a good LR by sweeping it and watching the loss

Answer

To count the number of epochs needed

Answer

To measure GPU memory usage

Answer

To pick the batch size automatically

Question 12

How does the learning rate interact with the size of gradient descent steps?

Accepted Answer

It multiplies the gradient to set the step size

Answer

It replaces the gradient entirely

Answer

It only affects the bias terms

Answer

It controls the number of layers

Question 13

Why is a smooth schedule like cosine often preferred over abrupt step drops?

Accepted Answer

Smooth changes are more stable and need no milestone tuning

Answer

Smooth schedules use less memory

Answer

Step drops are impossible to implement

Answer

Cosine always trains faster in wall-clock time

Question 14

What is polynomial decay of the learning rate?

Accepted Answer

lr = lr0 * (1 - t/T)^p

Answer

lr = lr0 * t^p

Answer

lr = lr0 + p*t

Answer

lr = lr0 / (p*t)

Question 15

What is inverse time decay (also called 1/t decay)?

Accepted Answer

lr = lr0 / (1 + k*t)

Answer

lr = lr0 * (1 + k*t)

Answer

lr = lr0 * t

Answer

lr = lr0 - k*t

Question 16

If a step decay uses initial lr=0.1 and multiplies by 0.1 every 30 epochs, what is the learning rate at epoch 65?

Accepted Answer

0.001

Answer

0.01

Answer

0.1

Answer

0.0001

Question 17

How do adaptive optimizers like Adam relate to learning rate schedules?

Accepted Answer

Schedules can still be applied to Adam's global base learning rate

Answer

Adam makes schedules impossible to use

Answer

Adam removes the learning rate completely

Answer

Schedules only work with plain SGD and never Adam

Question 18

What problem can occur if the learning rate is decayed too quickly early in training?

Accepted Answer

The model converges prematurely to a poor solution

Answer

Training becomes unstable and diverges

Answer

The number of parameters increases

Answer

Gradients become exactly zero immediately

Question 19

In PyTorch, when should you typically call scheduler.step() for an epoch-based schedule like StepLR?

Accepted Answer

Once at the end of each epoch

Answer

Once before training starts only

Answer

After every single weight value

Answer

Never, it updates automatically

Question 20

What is a 'triangular' cyclical learning rate (CLR)?

Accepted Answer

LR oscillates linearly between a lower and upper bound

Answer

LR only decreases each cycle

Answer

LR is held constant within a cycle

Answer

LR follows a perfect circle shape

Question 21

Why might increasing the learning rate occasionally (as in cyclical schedules) be beneficial?

Accepted Answer

Larger steps can escape poor local minima and saddle points

Answer

It always reduces the final loss to zero

Answer

It guarantees faster wall-clock training

Answer

It removes the need for gradients

Question 22

How is warmup commonly combined with decay in transformer training?

Accepted Answer

Linear warmup then inverse-square-root decay

Answer

Constant rate the whole time

Answer

Decay first then warmup

Answer

Random rate each step

Question 23

What does the 'patience' parameter control in ReduceLROnPlateau?

Accepted Answer

Epochs with no improvement to wait before reducing the LR

Answer

The factor the LR is multiplied by

Answer

The maximum learning rate allowed

Answer

The batch size used during training

Question 24

What is the difference between an epoch-based and a step-based (iteration-based) schedule?

Accepted Answer

Epoch-based updates per dataset pass; step-based updates per mini-batch

Answer

They are identical in all cases

Answer

Step-based updates only once total

Answer

Epoch-based updates after each sample

Question 25

If cosine annealing goes from lr_max=0.1 to lr_min=0 over T epochs, what is the learning rate at the halfway point t=T/2?

Accepted Answer

0.05

Answer

0.1

Answer

0.0

Answer

0.025

Question 26

Why are learning rate schedules especially important when using large batch sizes?

Accepted Answer

Large batches need higher peak LRs that require careful warmup

Answer

Large batches make the LR irrelevant

Answer

Schedules only matter for batch size of 1

Answer

Large batches forbid any LR changes

Question 27

What is a simple sign that your learning rate (or schedule) is set too high during training?

Accepted Answer

The loss diverges, oscillates, or becomes NaN

Answer

The loss decreases too smoothly

Answer

The model trains with zero error instantly

Answer

The GPU uses less memory

Question 28

What does the 'factor' (or gamma) argument represent in step or plateau schedulers?

Accepted Answer

The multiplier applied to the LR at each reduction

Answer

The total number of epochs

Answer

The initial learning rate value

Answer

The momentum coefficient

Question 29

How can a learning rate schedule help with generalization, not just optimization speed?

Accepted Answer

Lower late-stage rates settle into flatter, better-generalizing minima

Answer

Schedules only affect training speed, never test accuracy

Answer

They increase overfitting in every case

Answer

They have no effect on the final model

Question 30

In a warmup phase that linearly increases lr from 0 to 0.001 over 100 steps, what is the learning rate at step 40?

Accepted Answer

0.0004

Answer

0.001

Answer

0.004

Answer

0.0001

Test yourself