Reinforcement learning quiz

Question 1

Explain the difference between on-policy and off-policy reinforcement learning, and give one canonical algorithm for each.

Accepted Answer

On-policy evaluates and improves the same policy that generates data; off-policy learns a target policy from a different behavior policy

Answer

On-policy learns from any data; off-policy requires fresh data each step

Answer

On-policy is always model-based; off-policy is always model-free

Answer

On-policy uses value functions only; off-policy uses policies only

Question 2

Write the Bellman optimality equation for the action-value function Q*(s,a) and state what makes it nonlinear.

Accepted Answer

Q*(s,a) = E[r + γ max_{a'} Q*(s',a')]; nonlinear due to the max operator

Answer

Q*(s,a) = E[r + γ Q*(s',a')]; nonlinear due to expectation

Answer

Q*(s,a) = r + γ Σ_{a'} Q*(s',a'); nonlinear due to the sum

Answer

Q*(s,a) = max_{a'} E[r + γ Q*(s',a')]; linear because of the expectation

Question 3

Why does vanilla Q-learning with function approximation often diverge, and what is the 'deadly triad'?

Accepted Answer

Function approximation, bootstrapping, and off-policy training

Answer

Large learning rate, small batch, and sparse reward

Answer

Discount factor, exploration, and reward scaling

Answer

Replay buffer, target network, and gradient clipping

Question 4

What problem does Double DQN solve, and how does it modify the target computation?

Accepted Answer

It reduces overestimation bias by using the online net to select and the target net to evaluate the max action

Answer

It reduces variance by averaging two replay buffers

Answer

It speeds up training by using two GPUs

Answer

It eliminates the need for a target network entirely

Question 5

Derive the policy gradient theorem result and explain the role of the baseline.

Accepted Answer

Subtracting a state-dependent baseline reduces variance while keeping the gradient unbiased

Answer

The baseline biases the gradient toward higher-reward actions

Answer

The baseline is required to make the gradient estimator unbiased

Answer

The baseline must depend on the action to be valid

Question 6

What is the clipped surrogate objective in PPO and why is clipping used instead of a hard KL constraint?

Accepted Answer

Clipping limits the probability-ratio change to constrain policy updates without a hard KL constraint

Answer

Clipping increases the learning rate adaptively

Answer

Clipping normalizes advantages to unit variance

Answer

Clipping forces exploration by bounding the entropy

Question 7

Explain Generalized Advantage Estimation (GAE) and the bias-variance trade-off controlled by λ.

Accepted Answer

λ interpolates between low-variance/high-bias TD (λ=0) and high-variance/low-bias Monte Carlo (λ=1) advantage estimates

Answer

λ controls the discount factor for future rewards

Answer

λ sets the clipping range for the policy ratio

Answer

λ scales the entropy bonus

Question 8

In SARSA vs Q-learning, why is SARSA considered safer in environments with risky exploration (e.g., the Cliff Walking problem)?

Accepted Answer

SARSA's on-policy target accounts for exploratory actions, learning a safer path

Answer

SARSA uses a larger discount factor by default

Answer

Q-learning ignores rewards near terminal states

Answer

SARSA does not bootstrap, avoiding error propagation

Question 9

What is the entropy regularization term in Soft Actor-Critic (SAC), and what objective does it optimize?

Accepted Answer

It adds policy entropy to the reward, optimizing the maximum-entropy RL objective for exploration and robustness

Answer

It penalizes large gradients to prevent divergence

Answer

It maximizes the entropy of the reward distribution

Answer

It constrains the KL divergence between consecutive policies

Question 10

Why does DDPG use a deterministic policy, and how is exploration achieved during training?

Accepted Answer

It enables the deterministic policy gradient; exploration is added via action noise (e.g., Gaussian/OU)

Answer

A deterministic policy guarantees convergence to the global optimum

Answer

Deterministic policies need no critic

Answer

Exploration is achieved by ε-greedy over continuous actions

Question 11

What three tricks does TD3 add to DDPG, and what problem does each address?

Accepted Answer

Twin critics (min), delayed actor updates, and target policy smoothing

Answer

Double buffers, dropout, and batch norm

Answer

Prioritized replay, dueling heads, and noisy nets

Answer

Entropy bonus, automatic temperature, and reparameterization

Question 12

Explain the exploration-exploitation trade-off and how UCB addresses it in multi-armed bandits.

Accepted Answer

UCB adds an optimism bonus inversely related to how often an arm was pulled, achieving logarithmic regret

Answer

UCB picks actions uniformly at random with probability ε

Answer

UCB always exploits the empirically best arm

Answer

UCB uses a softmax over Q-values with temperature decay

Question 13

What is Thompson Sampling and why is it effective for exploration?

Accepted Answer

It samples a reward estimate from each action's posterior and picks the max, exploring via posterior uncertainty

Answer

It deterministically picks the action with the highest posterior mean

Answer

It adds Gaussian noise to the Q-values

Answer

It requires the true reward distribution to be known

Question 14

What is reward shaping, and what condition guarantees the optimal policy is preserved (potential-based shaping)?

Accepted Answer

Potential-based shaping F = γΦ(s') − Φ(s) preserves the optimal policy

Answer

Any bounded shaping reward preserves the optimal policy

Answer

Only constant shaping rewards preserve the optimal policy

Answer

Shaping never changes the optimal policy regardless of form

Question 15

Define the discount factor γ and explain the consequences of choosing γ close to 1 vs close to 0.

Accepted Answer

γ near 1 makes the agent far-sighted but increases variance and horizon; γ near 0 makes it myopic

Answer

γ near 1 makes the agent myopic; γ near 0 makes it far-sighted

Answer

γ only affects the learning rate

Answer

γ must equal 1 for the value function to be finite

Question 16

What is the difference between Monte Carlo and Temporal Difference methods for value estimation?

Accepted Answer

MC uses full returns (unbiased, high variance); TD bootstraps (biased, lower variance, online)

Answer

MC bootstraps and is biased; TD uses full returns and is unbiased

Answer

Both require complete episodes before updating

Answer

TD cannot be used with function approximation

Question 17

Explain prioritized experience replay (PER) and the bias it introduces.

Accepted Answer

PER samples by TD-error magnitude and corrects the induced bias with importance-sampling weights

Answer

PER samples uniformly and needs no correction

Answer

PER removes old transitions to avoid bias

Answer

PER prioritizes by reward magnitude and introduces no bias

Question 18

What is the dueling network architecture in DQN and why does it help?

Accepted Answer

It decomposes Q into V(s) and a mean-centered advantage A(s,a), aiding learning when actions have similar values

Answer

It uses two separate replay buffers for value and advantage

Answer

It duels two agents against each other

Answer

It replaces the max operator with a softmax

Question 19

Describe Hindsight Experience Replay (HER) and the type of problem it targets.

Accepted Answer

HER relabels achieved states as goals so sparse-reward failed episodes still provide learning signal

Answer

HER discards failed episodes to focus on successes

Answer

HER adds dense shaping rewards to every transition

Answer

HER is a model-based planning method

Question 20

What distinguishes model-based RL from model-free RL, and what is a key advantage and risk of model-based approaches?

Accepted Answer

Model-based RL learns dynamics for planning: more sample-efficient but risks model bias

Answer

Model-based RL never uses neural networks

Answer

Model-free RL is always more sample-efficient

Answer

Model-based RL cannot handle continuous states

Question 21

In MuZero, what three quantities does the learned model predict, and why is it notable that it does not model raw observations?

Accepted Answer

Policy, value, and reward via representation/dynamics/prediction functions in latent space

Answer

Next observation, reward, and termination

Answer

Q-values, advantages, and entropy

Answer

Pixel reconstructions, optical flow, and segmentation

Question 22

What is the credit assignment problem in RL, and name two mechanisms that help address it.

Accepted Answer

Eligibility traces/TD(λ) and advantage/return-redistribution methods

Answer

Increasing the batch size and learning rate

Answer

Using a larger replay buffer only

Answer

Reducing the discount factor to zero

Question 23

Explain the actor-critic architecture and why it reduces variance compared to REINFORCE.

Accepted Answer

The critic provides a low-variance bootstrapped advantage, reducing the variance of policy-gradient updates vs MC returns

Answer

The critic replaces the actor entirely

Answer

Actor-critic eliminates all bias from the gradient

Answer

It uses two actors to vote on actions

Question 24

What is offline (batch) RL, and what is the core challenge of distributional shift / extrapolation error?

Accepted Answer

Offline RL trains on a fixed dataset; the challenge is extrapolation error on out-of-distribution actions

Answer

Offline RL collects data online but trains offline; the challenge is latency

Answer

Offline RL has no exploration challenge because data is fixed

Answer

The main challenge is GPU memory for large datasets

Question 25

How does Conservative Q-Learning (CQL) mitigate overestimation in offline RL?

Accepted Answer

CQL lower-bounds Q by penalizing out-of-distribution action values while supporting dataset actions

Answer

CQL maximizes Q for all actions equally

Answer

CQL removes the Bellman backup entirely

Answer

CQL adds noise to the dataset to regularize

Question 26

Describe RLHF for language models: what are the three stages and what does the reward model represent?

Accepted Answer

SFT, reward-model training on preferences, and RL (PPO) with a KL penalty

Answer

Pretraining, distillation, and quantization

Answer

Tokenization, embedding, and decoding

Answer

Reward shaping, imitation, and pruning

Question 27

What is Direct Preference Optimization (DPO) and how does it differ from PPO-based RLHF?

Accepted Answer

DPO directly optimizes a preference loss without a separate reward model or online RL, with implicit KL regularization

Answer

DPO requires a separately trained reward model and online rollouts

Answer

DPO is a model-based planning method for LLMs

Answer

DPO replaces the transformer with an actor-critic network

Question 28

What is GRPO (Group Relative Policy Optimization) and why is it attractive for LLM RL training?

Accepted Answer

GRPO normalizes rewards across a sampled group per prompt to form advantages, removing the need for a critic network

Answer

GRPO requires two critics like TD3

Answer

GRPO is a model-based world-model method

Answer

GRPO replaces the policy gradient with Q-learning

Question 29

What is reward hacking (specification gaming), and how is it commonly mitigated in RLHF pipelines?

Accepted Answer

It is the agent exploiting the proxy reward against true intent; mitigated by KL penalties, RM ensembles, and RM retraining

Answer

It is when training crashes due to NaN rewards

Answer

It is overfitting the value network only

Answer

It is when the discount factor is set too high

Question 30

Explain the role of the KL penalty in RLHF and what happens if its coefficient is too small or too large.

Accepted Answer

It keeps the policy near the reference model; too small → reward hacking/drift, too large → little learning

Answer

The KL penalty controls the exploration noise

Answer

It scales the reward model output

Answer

It replaces the advantage estimate

Test yourself