Cheat sheet · Time Series & RL

Reinforcement learning cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Reinforcement Learning (RL) trains agents to maximize expected cumulative reward by interacting with an environment modeled as an MDP. This sheet covers core theory (value/policy methods), deep RL (DQN, PPO, SAC), exploration, model-based and offline RL, and the now-dominant RLHF/DPO/RLVR stack for aligning and reasoning-training LLMs.

Foundations: MDPs, Returns & Value Functions

An MDP is the tuple (S, A, P, R, gamma): states, actions, transition dynamics P(s'|s,a), reward function R(s,a,s'), and discount gamma in [0,1]. Model-free RL learns a policy pi(a|s) without knowing P/R; model-based RL learns P/R (or a surrogate) to plan.
Goal: maximize expected discounted return G_t = sum_{k>=0} gamma^k * r_{t+k+1}. gamma < 1 guarantees a finite return in continuing tasks and trades off myopia vs far-sighted credit assignment; gamma = 1 is valid only for episodic (terminating) tasks.
State-value V^pi(s) = E_pi[G_t | s_t=s]; action-value Q^pi(s,a) = E_pi[G_t | s_t=s, a_t=a]. Advantage A^pi(s,a) = Q^pi(s,a) - V^pi(s) measures how much better an action is than the policy's average.
Bellman expectation: V^pi(s) = sum_a pi(a|s) sum_{s'} P(s'|s,a)[R(s,a,s') + gamma V^pi(s')]. Bellman optimality: V*(s) = max_a sum_{s'} P(s'|s,a)[R(s,a,s') + gamma V*(s')]; the optimal policy is greedy w.r.t. Q*.
Markov property: the next state/reward depend only on the current state-action, not history. Real problems are often partially observable (POMDP) and are handled with recurrence (RNN/transformer) or frame stacking to recover an approximate Markov state.
Prediction (evaluate a fixed pi) vs control (find pi*). Generalized Policy Iteration alternates policy evaluation and greedy improvement, and is the backbone of nearly all RL algorithms.

Tabular & Value-Based Methods

Dynamic Programming (value/policy iteration) needs a known model and full state sweeps: exact but does not scale; mainly a conceptual baseline.
Monte Carlo: estimate values from complete-episode returns. Unbiased but high variance and requires episodic, terminating tasks.
Temporal Difference (TD) bootstraps from current estimates: TD(0) target = r + gamma*V(s'). Lower variance, learns online from incomplete episodes, but biased. TD(lambda) / eligibility traces interpolate between MC and TD.
Q-learning (off-policy): Q(s,a) += alpha*[r + gamma*max_a' Q(s',a') - Q(s,a)]. Converges to Q* regardless of the behavior policy (given sufficient exploration and decaying alpha).
SARSA (on-policy): uses the actually-taken next action a' instead of the max, so it learns the value of the policy it follows; more conservative near hazards (classic cliff-walking example).
Off-policy (learn a target policy from data generated by another) enables replay and data reuse; on-policy (SARSA, PPO) needs fresh data from the current policy.

Deep Value Methods: DQN Family

DQN approximates Q with a neural net, stabilized by two tricks: an experience replay buffer (breaks temporal correlation, reuses data) and a periodically-updated target network (provides stable bootstrap targets).
Loss: L = E[(r + gamma*max_a' Q_target(s',a') - Q_online(s,a))^2], minimized by SGD over sampled minibatches; for terminal s' the gamma term is dropped.
Double DQN fixes overestimation bias by decoupling action selection from evaluation: target = r + gamma*Q_target(s', argmax_a' Q_online(s',a')).
Dueling DQN splits the net into V(s) and A(s,a) streams; Prioritized Experience Replay (PER) samples high-TD-error transitions more often and corrects the bias with importance-sampling weights.
Rainbow combines Double + Dueling + PER + multi-step returns + distributional (C51) + noisy nets, and is a strong discrete-action baseline.
DQN handles discrete action spaces only; continuous control needs policy-gradient or actor-critic methods (DDPG/TD3/SAC).

Policy Gradient & Actor-Critic

Policy gradient theorem: grad J = E[ grad log pi(a|s) * Psi ], where Psi can be the return, Q, or (best) the advantage A. Subtracting a state-value baseline to form the advantage reduces variance without adding bias: the actor-critic setup (actor = policy, critic = value fn).
GAE (Generalized Advantage Estimation, parameter lambda) smoothly trades bias vs variance in the advantage estimate and is standard in PPO.
PPO is the practical default for on-policy RL: it maximizes a clipped surrogate so the new policy stays close to the old one, avoiding destructive updates. Robust, easy to tune, and dominant in RLHF.
TRPO is PPO's predecessor: a hard KL trust-region constraint solved via natural gradient; strong monotonic-improvement guarantees but expensive (second-order).
Continuous-control off-policy: DDPG (deterministic actor + critic), TD3 (twin critics + delayed policy updates + target-action smoothing to curb overestimation), and SAC (maximum-entropy: adds an entropy bonus for exploration, sample-efficient, with an auto-tuned temperature).
A2C/A3C are simpler synchronous/asynchronous advantage actor-critics: useful for understanding the family, less used at the frontier today.

Exploration, Model-Based & Offline RL

Exploration vs exploitation: epsilon-greedy and Boltzmann (softmax) are simple baselines; UCB and Thompson sampling are principled for bandits. Deep RL adds intrinsic rewards (curiosity/RND, count-based) for sparse-reward tasks.
Model-based RL learns dynamics to plan or generate synthetic rollouts (Dyna, MuZero, Dreamer/world models); far more sample-efficient but risks compounding model error.
Offline (batch) RL learns purely from a fixed dataset with no environment interaction: critical when exploration is unsafe or costly (healthcare, robotics, recsys).
The core offline challenge is distributional shift: the policy queries out-of-distribution actions the critic overestimates. Fixes: conservatism (CQL), policy constraints (BCQ, TD3+BC), or in-sample/implicit methods (IQL).
Reward shaping speeds learning but, if not potential-based, can change the optimal policy. Potential-based shaping F = gamma*phi(s') - phi(s) provably preserves the optimal policy (Ng et al.).
Imitation learning is the cousin: behavior cloning (supervised; suffers covariate shift, which DAgger mitigates) and inverse RL (infer the reward from expert demonstrations).

RL for LLMs: RLHF, DPO & RLVR

RLHF classic pipeline: (1) SFT on demonstrations, (2) train a reward model on human preference pairs (Bradley-Terry loss), (3) optimize the policy with PPO against the reward, plus a per-token KL penalty to the frozen SFT reference to prevent reward hacking and mode collapse.
DPO (Direct Preference Optimization) skips the explicit reward model and the RL loop, optimizing preferences directly with a classification-style loss on a frozen reference: cheaper and more stable, now widely used. Variants: IPO, KTO (pointwise good/bad labels), ORPO (reference-free), SimPO.
GRPO (Group Relative Policy Optimization, DeepSeek) drops the value-function critic: it samples a group of responses per prompt and normalizes rewards within the group to form the advantage; memory-efficient and central to reasoning-model training.
RLVR (RL from Verifiable Rewards) uses programmatic checkers (unit tests, math answer-matching) instead of a learned reward model; powers reasoning models (o-series, DeepSeek-R1) and avoids reward-model gaming.
Reward hacking remains the central failure: models exploit reward-model quirks. Mitigations: KL regularization, reward-model ensembles, and constitutional AI / RLAIF (AI feedback) to scale labeling.
Practical knobs: KL coefficient (too high = no learning, too low = drift/gibberish), frozen reference model, and batch/group sizes. Reasoning RL (long chains-of-thought with verifiable rewards) is the dominant 2025-2026 frontier.

Common pitfalls

Confusing on-policy vs off-policy: e.g. using a replay buffer with PPO, or expecting SARSA to converge to the optimal (rather than the safe on-policy) policy.
Ignoring overestimation bias: vanilla DQN/DDPG overestimate Q via the max operator; use Double DQN or TD3 twin critics.
Reward mis-specification and non-potential-based shaping that silently change the optimal policy, leading to reward hacking.
High variance from naive policy gradients without a baseline/advantage; forgetting GAE or to normalize advantages per batch.
Poor input handling: not normalizing observations/rewards, missing frame-stacking for POMDPs, or a gamma mismatched to the task horizon.
Treating offline RL like supervised learning and ignoring distributional shift; the critic overestimates OOD actions without conservatism (CQL/IQL).
Missetting the RLHF/DPO KL penalty: too low lets the policy drift into degenerate text, too high blocks any improvement.
Evaluating on a single seed/env: RL is notoriously seed- and hyperparameter-sensitive; report mean and variance over multiple seeds.
Assuming the Markov property holds when the true state is only partially observed, causing unstable or biased learning.

Interview tips

Derive the Bellman expectation and optimality equations and explain V vs Q vs advantage from memory.
Articulate on-policy vs off-policy and give a concrete consequence (sample reuse, replay compatibility, safety).
Explain why DQN needs BOTH a replay buffer and a target network, and which instability each one addresses.
Walk through the PPO clipped objective and why clipping (vs TRPO's hard KL constraint) keeps updates stable.
Discuss the bias-variance tradeoff across MC, TD(0), TD(lambda)/GAE, and the role of the discount factor.
Know the modern LLM stack end-to-end: SFT -> reward model -> PPO, and why DPO/GRPO/RLVR emerged as alternatives.
Define reward hacking and distributional shift, and name concrete mitigations (KL penalty, potential-based shaping, CQL/IQL).
Compare SAC vs TD3 vs PPO for a continuous-control task, justifying the choice on sample efficiency and stability.
Show practical awareness: hyperparameter/seed sensitivity, observation normalization, and how you'd debug a non-learning agent.

Read the full lesson Test yourself — 33 question quiz