RLHF & Alignment quiz

Question 1

What is RLHF and what are the three main stages of the pipeline?

Accepted Answer

RLHF has three stages: SFT, reward model training, and PPO optimization with KL penalty

Answer

RLHF has two stages: reward model training followed by supervised fine-tuning

Answer

RLHF uses only human preference labels without any supervised fine-tuning

Answer

RLHF trains the base model directly on preference labels using cross-entropy

Question 2

What is DPO (Direct Preference Optimization) and how does it avoid training a reward model?

Accepted Answer

DPO directly trains the policy on preference pairs via a closed-form loss derived from the RLHF objective

Answer

DPO trains a reward model first, then optimizes the policy via PPO

Answer

DPO requires a value function network to estimate returns

Answer

DPO produces lower quality alignment than RLHF on all benchmarks

Question 3

What is reward hacking and how does it manifest in RLHF systems?

Accepted Answer

Reward hacking occurs when the policy exploits reward model flaws instead of genuinely improving quality

Answer

Reward hacking means the reward model correctly identifies high-quality responses

Answer

Reward hacking only occurs when β (KL coefficient) is too large

Answer

Reward hacking is impossible when using DPO instead of PPO

Question 4

What is RLAIF (RL from AI Feedback) and how does it differ from RLHF?

Accepted Answer

RLAIF uses a strong LLM to generate preference labels instead of human annotators

Answer

RLAIF uses human annotators but automates the reward model training

Answer

RLAIF produces inferior alignment compared to RLHF in all evaluations

Answer

RLAIF requires the feedback model to be the same model being trained

Question 5

What are the 'HHH' criteria in AI alignment?

Accepted Answer

HHH stands for Helpful, Honest, and Harmless — Anthropic's three core alignment criteria

Answer

HHH stands for High, Hard, and Heavy — metrics for model capacity

Answer

HHH is an evaluation benchmark for instruction-following quality

Answer

HHH is used only during SFT and is not relevant for RLHF

Question 6

What is sycophancy in aligned LLMs and why is it a problem?

Accepted Answer

Sycophancy is when the model tells users what they want rather than truth, undermining honesty

Answer

Sycophancy is a positive property indicating the model respects user preferences

Answer

Sycophancy is caused by using too large a KL penalty during PPO

Answer

Sycophancy only appears in models smaller than 7B parameters

Question 7

What is red teaming in the context of AI alignment?

Accepted Answer

Red teaming deliberately probes models for safety failures to inform safety training before deployment

Answer

Red teaming trains the model to generate harmful content for research purposes

Answer

Red teaming is only relevant for models with tool use capabilities

Answer

Red teaming is performed only by external academic researchers, not by AI companies

Question 8

What is SimPO (Simple Preference Optimization) and how does it simplify DPO?

Accepted Answer

SimPO removes the reference model by using length-normalized log-prob as implicit reward

Answer

SimPO requires a reference model and reward model for training

Answer

SimPO is identical to DPO with β set to zero

Answer

SimPO was shown to be inferior to DPO in all alignment benchmarks

Question 9

What is the difference between safety and helpfulness in alignment and how are they balanced?

Accepted Answer

Safety and helpfulness can conflict; over-cautious refusals reduce helpfulness; alignment calibrates the tradeoff

Answer

Safety and helpfulness never conflict; safe models are always more helpful

Answer

Helpfulness is prioritized over safety in all production-grade LLMs

Answer

Safety improvements always degrade helpfulness by exactly 20% on benchmarks

Question 10

What is ORPO (Odds Ratio Preference Optimization)?

Accepted Answer

ORPO combines SFT and preference optimization in one stage without a reference model using odds ratio loss

Answer

ORPO requires a pretrained reward model for computing the odds ratio

Answer

ORPO is only applicable to encoder models like BERT

Answer

ORPO is identical to DPO with the reference model replaced by a value network

Question 11

What is instruction hierarchy / system prompt authority in aligned LLMs?

Accepted Answer

System prompts (operator-level) have higher authority than user messages, but absolute safety policies override both

Answer

User messages always take priority over system prompts in aligned LLMs

Answer

Instruction hierarchy means the longest instruction always takes priority

Answer

All instruction levels are treated equally in modern RLHF-trained models

Question 12

What is the 'helpful, harmless, honest' tension in practice?

Accepted Answer

Helpful vs harmless creates genuine edge cases; RLHF and constitutions guide context-dependent trade-offs

Answer

Helpful and harmless are always complementary and never conflict

Answer

Harmless always takes priority over helpful in all production models

Answer

The HHH tension is only relevant for models deployed in medical contexts

Question 13

What is Best-of-N (BoN) sampling and how does it improve output quality at inference?

Accepted Answer

BoN generates N responses, scores with a reward model, and returns the highest-scoring one at inference

Answer

BoN requires policy gradient training before it can be applied

Answer

BoN quality scales linearly with N at all sample sizes

Answer

BoN requires the reward model to have been trained on the same dataset as the policy

Question 14

What is the difference between jailbreaking and adversarial prompting?

Accepted Answer

Jailbreaking bypasses model safety filters; prompt injection injects malicious instructions into context

Answer

Jailbreaking and prompt injection are identical attack vectors

Answer

Adversarial prompting always requires gradient-based optimization to craft inputs

Answer

Modern RLHF-trained models are completely immune to jailbreaking

Question 15

What is the purpose of the SFT (Supervised Fine-Tuning) stage in RLHF?

Accepted Answer

SFT fine-tunes the base model on high-quality demonstrations to create a stable starting point for RLHF

Answer

SFT is the final stage of RLHF that produces the aligned model

Answer

SFT is identical to pretraining but with a smaller learning rate

Answer

SFT requires millions of examples to be effective

Question 16

What is scalable oversight and why is it important for advanced AI alignment?

Accepted Answer

Scalable oversight addresses maintaining human supervision when AI capabilities exceed human evaluation ability

Answer

Scalable oversight is only relevant for models with more than 1T parameters

Answer

Scalable oversight is solved by increasing human annotator count proportionally

Answer

Scalable oversight is a purely theoretical concern with no practical implications in 2026

Question 17

What is the role of length normalization in preference optimization?

Accepted Answer

Length normalization counteracts length bias where reward models favor verbosity over quality

Answer

Length normalization encourages longer responses which human raters prefer

Answer

Length normalization is only applied during evaluation, not training

Answer

Length bias is not a real issue in modern RLHF-trained models

Question 18

What is Reinforcement Learning from Verifiable Rewards (RLVR) and how does DeepSeek-R1 use it?

Accepted Answer

RLVR uses deterministic verifiers (math solvers, unit tests) as reward signals, scaling without human annotation

Answer

RLVR requires human preference labels to determine reward signals

Answer

RLVR produces worse reasoning than RLHF because verifiers are less nuanced than humans

Answer

RLVR can only be applied to coding tasks, not mathematical reasoning

Question 19

What is prompt injection in the context of LLM agents and why is it particularly dangerous?

Accepted Answer

Prompt injection inserts adversarial instructions through untrusted data; dangerous because agents have tool access

Answer

Prompt injection is only possible through the system prompt, not user input

Answer

RLHF training completely eliminates prompt injection vulnerabilities

Answer

Prompt injection only works on models smaller than 7B parameters

Question 20

What is reward model ensembling and how does it reduce reward hacking?

Accepted Answer

Reward model ensembles reduce exploitability by requiring exploits to fool multiple models simultaneously

Answer

Ensembling reward models always produces better results than any single model

Answer

Ensemble uncertainty is not useful for alignment because uncertainty means the model should always refuse

Answer

Reward model ensembles require identical training data to be effective

Question 21

What is TLDR (Too Long Didn't Read) fine-tuning and what alignment lessons does it illustrate?

Accepted Answer

TL;DR fine-tuning showed RLHF with preference comparisons outperforms SFT and provided KL constraint lessons

Answer

TL;DR experiments showed SFT always outperforms RLHF for summarization

Answer

TL;DR summarization only uses reward models, no KL divergence constraint

Answer

TL;DR experiments were conducted without any human preference labels

Question 22

What is model spec / model card in the context of AI alignment and governance?

Accepted Answer

Model spec documents the model's intended values and behaviors that guide alignment training; model card documents capabilities and limitations

Answer

Model card and model spec are legal documents with no technical relevance

Answer

Model spec is only used for multimodal models, not language-only LLMs

Answer

Model card specifies training compute requirements; model spec specifies benchmark targets

Question 23

What is deceptive alignment and why is it a concern for scalable AI safety?

Accepted Answer

Deceptive alignment is a model that behaves safely during training but pursues misaligned goals when deployed

Answer

Deceptive alignment means the model refuses harmful requests deceptively

Answer

Deceptive alignment is prevented by using RLHF instead of SFT alone

Answer

Deceptive alignment is theoretically impossible in current transformer architectures

Question 24

What is SFT data curation and why does data quality matter more than quantity?

Accepted Answer

LIMA showed 1,000 high-quality examples can match 52k low-quality ones; quality > quantity for SFT

Answer

More SFT training examples always produce better instruction following regardless of quality

Answer

SFT data curation is only necessary for models larger than 70B parameters

Answer

SFT data quality is less important than RLHF data quality for final model performance

Question 25

What are absolute hardcoded behaviors vs instructable behaviors in aligned LLMs?

Accepted Answer

Hardcoded behaviors (CBRN, CSAM) cannot be overridden; softcoded behaviors can be adjusted by operators/users

Answer

All model behaviors can be adjusted by operators for any business reason

Answer

Hardcoded behaviors only apply to text generation, not tool use or code execution

Answer

Users can override hardcoded behaviors by providing a sufficiently compelling reason

Question 26

What is the preference dataset and what makes a high-quality one?

Accepted Answer

High-quality preference datasets have meaningful chosen/rejected differences, diverse prompts, and qualified annotators

Answer

Preference datasets only contain harmful prompt examples for safety training

Answer

Preference datasets always have equal numbers of harmful and helpful examples

Answer

Preference datasets are identical to SFT datasets with reversed labels

Question 27

What is activation steering (representation engineering) in alignment?

Accepted Answer

Activation steering adds concept vectors to activations at inference time to shift behavior without fine-tuning

Answer

Activation steering modifies model weights permanently to change behavior

Answer

Activation steering is identical to LoRA fine-tuning with a small rank

Answer

Activation steering only works for toxicity reduction, not other behaviors

Question 28

What is the role of the beta (β) parameter in DPO and how does it affect alignment?

Accepted Answer

β controls KL penalty strength; small β allows more deviation from reference (stronger but riskier); large β stays close to SFT

Answer

Larger β in DPO produces stronger preference optimization and more alignment improvement

Answer

β must always be set to 0.1 for DPO to converge

Answer

β in DPO is equivalent to the learning rate and should be tuned similarly

Test yourself