Reinforcement Learning quiz

Question 1

What is a Markov Decision Process (MDP)?

Accepted Answer

States, actions, transition probabilities, rewards, discount factor

Answer

Only states and rewards

Answer

Actions and neural networks

Answer

Policy and value functions only

Question 2

What does the Bellman equation express?

Accepted Answer

V(s) = max_a Σ P(s'|s,a)[R(s,a) + γ V(s')]

Answer

V(s) = R(s) + V(s+1)

Answer

V(s) = log π(a|s)

Answer

V(s) = Σ R(s,a)/|A|

Question 3

Explain the ε-greedy exploration strategy.

Accepted Answer

Random action with prob ε, best known action with prob 1−ε

Answer

Always pick the best action

Answer

Always pick a random action

Answer

Pick action proportional to Q-value

Question 4

What is the key difference between Q-learning and SARSA?

Accepted Answer

Q-learning uses max over next actions; SARSA uses the actual next action

Answer

SARSA uses max; Q-learning uses actual action

Answer

Both are on-policy

Answer

They are identical except for learning rate

Question 5

What is temporal difference (TD) learning?

Accepted Answer

Updates using current next-state estimate (bootstrapping)

Answer

Waits until episode ends to update

Answer

Uses only random sampling

Answer

Requires a model of the environment

Question 6

What is the REINFORCE algorithm?

Accepted Answer

∇J(θ) ≈ Σ G_t ∇ log π_θ(a_t|s_t)

Answer

∇J(θ) = Q(s,a)

Answer

∇J(θ) = −log π(a|s)

Answer

∇J(θ) = V(s) − Q(s,a)

Question 7

What problem does Deep Q-Network (DQN) solve, and what are its two key tricks?

Accepted Answer

Experience replay and target network

Answer

Batch norm and dropout

Answer

Actor-critic and entropy

Answer

LSTM and attention

Question 8

What does 'on-policy' vs 'off-policy' mean in RL?

Accepted Answer

On-policy learns from current behavior; off-policy learns from a different behavior policy

Answer

On-policy is faster

Answer

Off-policy cannot reuse data

Answer

They are the same for continuous actions

Question 9

What is value iteration and how does it find the optimal policy?

Accepted Answer

Iteratively applies Bellman optimality equation until convergence

Answer

Randomly samples actions and tracks rewards

Answer

Uses neural networks to estimate gradients

Answer

Runs episodes and averages returns

Question 10

What is the role of the replay buffer in DQN?

Accepted Answer

Stores past transitions for random mini-batch sampling, breaking correlations

Answer

Stores only the most recent transition

Answer

Buffers rewards before computing returns

Answer

Caches neural network weights

Question 11

What is model-based vs model-free RL?

Accepted Answer

Model-free learns from interactions only; model-based also learns environment dynamics

Answer

Model-free builds a transition model

Answer

Model-based doesn't use value functions

Answer

They always perform the same

Question 12

What is sparse reward and how is it addressed?

Accepted Answer

Reward shaping, HER, curiosity-driven exploration, curriculum learning

Answer

Only reward shaping works

Answer

Increase learning rate

Answer

Use longer episodes

Question 13

What is an episodic vs continuing task?

Accepted Answer

Episodic tasks have terminal states; continuing tasks run indefinitely

Answer

Episodic tasks use no discount

Answer

Continuing tasks always converge faster

Answer

They are the same if γ=1

Question 14

What is the policy improvement theorem?

Accepted Answer

Acting greedily w.r.t. V^π yields a policy at least as good

Answer

Acting greedily can worsen the policy

Answer

It only applies to deterministic policies

Answer

It requires knowing P(s'|s,a) exactly

Question 15

What is the credit assignment problem in RL?

Accepted Answer

Determining which past actions caused a delayed reward

Answer

Assigning reward to each network weight

Answer

Choosing the learning rate

Answer

Estimating the discount factor

Question 16

What is a Markov chain and how does it differ from an MDP?

Accepted Answer

Markov chain has no actions or rewards; MDP adds both

Answer

They are identical

Answer

Markov chain has rewards but no actions

Answer

MDP has no transition probabilities

Question 17

What is the state-action-reward-state-action (SARSA) update rule?

Accepted Answer

Q(s,a) ← Q(s,a) + α[r + γ Q(s',a') − Q(s,a)]

Answer

Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') − Q(s,a)]

Answer

Q(s,a) ← r + γ Q(s',a')

Answer

Q(s,a) ← α·r

Question 18

What is intrinsic motivation in RL?

Accepted Answer

Internal curiosity signals added to extrinsic reward to encourage exploration

Answer

Rewards from the true environment only

Answer

Penalty for exploring new states

Answer

Reward proportional to policy entropy

Question 19

What is DDPG (Deep Deterministic Policy Gradient)?

Accepted Answer

Deterministic actor + Q-critic for continuous action spaces

Answer

Stochastic actor with discrete actions

Answer

Only uses Monte Carlo returns

Answer

No target network needed

Question 20

What is Generalized Advantage Estimation (GAE)?

Accepted Answer

Smoothly interpolates between TD and MC advantage estimates via λ

Answer

Computes advantages using only MC returns

Answer

Estimates Q-values without a critic

Answer

Uses the policy gradient directly

Question 21

What is reward clipping and why is it used in DQN?

Accepted Answer

Clips rewards to [-1,+1] to normalize scales across games

Answer

Always improves performance

Answer

Clips actions not rewards

Answer

Only applies to continuous action spaces

Question 22

What is the difference between value-based and policy-based RL?

Accepted Answer

Value-based: implicit policy from Q; policy-based: direct policy optimization

Answer

Value-based only works for continuous actions

Answer

Policy-based learns Q-values

Answer

They are identical

Question 23

How does A3C differ from A2C?

Accepted Answer

A3C uses asynchronous workers; A2C synchronizes all workers first

Answer

A2C is faster than A3C always

Answer

A3C only works with GPUs

Answer

They use different advantage formulations

Question 24

What is Thompson Sampling?

Accepted Answer

Sample from posterior distributions and pull the arm with highest sample

Answer

Always pull the arm with highest mean

Answer

Randomly sample arms uniformly

Answer

Use UCB with Bayesian updates

Question 25

What is reward in RL and how should it be designed?

Accepted Answer

Scalar signal reflecting immediate desirability of a state or action

Answer

Only sparse terminal signal

Answer

Always a binary 0/1

Answer

Negative log-likelihood

Question 26

What is multi-agent RL?

Accepted Answer

Multiple agents learn simultaneously in a shared environment

Answer

Single agent with multiple reward signals

Answer

Parallel training of independent models

Answer

Only applicable to zero-sum games

Test yourself