AI Safety & Guardrails quiz

Question 1

What is the difference between alignment and safety in the context of large language models?

Accepted Answer

Alignment covers matching human values broadly; safety is a subset focused on preventing harmful or catastrophic outputs

Answer

Alignment and safety are synonymous terms in AI research

Answer

Safety refers to computational stability; alignment refers to output quality

Answer

Alignment is only relevant for language models; safety applies to all AI systems

Question 2

What is Constitutional AI (CAI) and how does it reduce reliance on human feedback for harmlessness training?

Accepted Answer

CAI uses a constitution of principles to generate synthetic preference data via AI critique, reducing need for human harmlessness labels

Answer

CAI trains a separate safety classifier and blocks outputs at inference time

Answer

CAI replaces RLHF entirely with supervised fine-tuning on hand-curated safe examples

Answer

CAI uses human feedback at 10x the scale of standard RLHF for better safety

Question 3

What is prompt injection and how does it differ from jailbreaking?

Accepted Answer

Prompt injection embeds malicious instructions in external data an LLM processes; jailbreaking comes from the user's own prompt

Answer

Prompt injection and jailbreaking are identical attack vectors with different names

Answer

Prompt injection only affects models with web browsing tools

Answer

Jailbreaking requires physical access to model weights

Question 4

What is the Llama Guard model and how is it used as a safety classifier?

Accepted Answer

Llama Guard is a fine-tuned LLM classifier that evaluates conversations against a harm taxonomy for input/output moderation

Answer

Llama Guard is a keyword blocklist integrated into the Meta LLaMA tokenizer

Answer

Llama Guard replaces RLHF by training the base model to refuse harmful prompts

Answer

Llama Guard only classifies user inputs, not model outputs

Question 5

What is concept erasure (machine unlearning) in LLMs and what methods are used?

Accepted Answer

Methods like ROME (weight editing), RMU (representation misdirection), and gradient ascent on forget sets enable LLM unlearning without full retraining

Answer

Machine unlearning requires retraining the model from scratch on filtered data

Answer

ROME erases concepts by adding new tokens to the vocabulary that override old associations

Answer

Machine unlearning is not technically feasible for transformer models

Question 6

How is toxicity measured and what datasets are used to evaluate safety classifiers?

Accepted Answer

Toxicity is measured via Perspective API and evaluated on datasets like ToxiGen, RealToxicityPrompts, and BBQ covering various bias dimensions

Answer

Toxicity is measured exclusively by human annotators; no automated metrics exist

Answer

BLEU score is used as a proxy for toxicity measurement in safety evaluations

Answer

Toxicity classifiers are only evaluated on English-language content

Question 7

What is the OWASP Top 10 for LLM Applications and which risks are most critical?

Accepted Answer

OWASP LLM Top 10 includes Prompt Injection, Sensitive Information Disclosure, and Excessive Agency as top risks for LLM applications

Answer

OWASP LLM Top 10 focuses only on training-time attacks, not deployment risks

Answer

SQL Injection and XSS are the top two OWASP LLM risks

Answer

OWASP LLM Top 10 only applies to chatbot applications, not API deployments

Question 8

What is the Dual Newspaper Test for evaluating AI safety responses?

Accepted Answer

The Dual Newspaper Test checks whether a response would appear in either a harmful AI story or an overly restrictive AI story

Answer

The Dual Newspaper Test requires two different LLMs to evaluate each response

Answer

The Dual Newspaper Test is a formal mathematical metric for measuring refusal rates

Answer

The test only applies to models deployed in media applications

Question 9

How does the Perspective API work and what are its known limitations for content moderation?

Accepted Answer

Perspective API scores toxicity but suffers from demographic bias, context blindness, and adversarial evasion limitations

Answer

Perspective API uses rule-based keyword matching rather than machine learning

Answer

Perspective API achieves human-level toxicity detection with no known bias issues

Answer

Perspective API only processes text in English with no multilingual support

Question 10

What is the interpretability technique of sparse autoencoders (SAEs) and how do they relate to model safety?

Accepted Answer

SAEs decompose polysemantic model activations into monosemantic features enabling identification and monitoring of dangerous capability circuits

Answer

SAEs compress model weights for efficient deployment without interpretability goals

Answer

SAEs are used to speed up inference by replacing attention layers

Answer

SAEs only work on small language models with fewer than 1 billion parameters

Question 11

What is a safety benchmark suite and which ones are standard for evaluating LLM safety?

Accepted Answer

HarmBench, SALAD-Bench, SimpleSafetyTests, and WildGuard are standard LLM safety evaluation benchmarks covering diverse harm categories

Answer

BLEU and perplexity are the standard metrics for safety evaluation

Answer

A single benchmark like HarmBench is sufficient for comprehensive safety evaluation

Answer

Safety benchmarks are proprietary and not available to researchers

Test yourself