LLM Pretraining quiz

Question 1

What is the difference between BPE, WordPiece, and Unigram tokenization?

Accepted Answer

BPE merges most frequent pairs; WordPiece merges to maximize LM likelihood; Unigram prunes from large vocab

Answer

BPE merges by likelihood; WordPiece merges by frequency

Answer

All three methods produce identical tokenizations for the same corpus

Answer

Unigram tokenization is the slowest due to requiring dynamic programming at inference

Question 2

What is Rotary Positional Encoding (RoPE) and why is it used in modern LLMs?

Accepted Answer

RoPE rotates Q and K vectors so their dot product encodes relative position, enabling extrapolation

Answer

RoPE adds a learnable positional vector to token embeddings like BERT

Answer

RoPE is a fixed sinusoidal encoding like the original Transformer

Answer

RoPE uses absolute position in key vectors only, not queries

Question 3

What is the scaling law for language model pretraining (Kaplan et al., 2020)?

Accepted Answer

Kaplan et al. showed loss follows power laws in N, D, C; Chinchilla revised optimal N≈D (20 tokens/param)

Answer

Scaling laws show that only dataset size matters; model size has no predictive power

Answer

Scaling laws predict that doubling model size always halves loss

Answer

Scaling laws only apply to encoder-only models like BERT

Question 4

What is causal language modeling (CLM) and how does its training objective differ from masked language modeling (MLM)?

Accepted Answer

CLM predicts next token with causal attention; MLM reconstructs masked tokens with bidirectional attention

Answer

CLM uses bidirectional attention; MLM uses causal attention

Answer

CLM and MLM produce identical representations for the same corpus

Answer

MLM is used for text generation; CLM is used for classification

Question 5

What is Flash Attention 2 and how does it improve over Flash Attention 1?

Accepted Answer

Flash Attention 2 improves work partitioning, reduces non-matmul FLOPs, and supports MQA/GQA natively

Answer

Flash Attention 2 uses a different attention formula producing different outputs than standard attention

Answer

Flash Attention 2 only improves backward pass efficiency, not forward

Answer

Flash Attention 2 requires FP8 precision to achieve its speedups

Question 6

What is the SwiGLU activation function and why is it preferred over ReLU in transformer FFN layers?

Accepted Answer

SwiGLU uses Swish-gated linear units with multiplicative gating, outperforming ReLU in LLMs

Answer

SwiGLU uses a ReLU gate with a separate projection for stability

Answer

SwiGLU requires four projection matrices instead of the standard two in FFN

Answer

SwiGLU is used only in encoder architectures, not decoder LLMs

Question 7

What is the difference between pre-norm and post-norm transformer architectures?

Accepted Answer

Pre-norm applies LN before the sublayer, improving gradient flow; post-norm applies after residual addition

Answer

Post-norm applies LN before the sublayer; pre-norm applies it after residual addition

Answer

Pre-norm and post-norm are identical for models with fewer than 12 layers

Answer

Post-norm is exclusively used in modern LLMs for better loss convergence

Question 8

What is sequence packing and why does it improve training efficiency?

Accepted Answer

Sequence packing concatenates short documents to fill context windows, avoiding padding waste

Answer

Sequence packing allows shorter sequences to attend across document boundaries freely

Answer

Sequence packing is only useful for code datasets, not natural language

Answer

Sequence packing requires quadratic memory scaling and is avoided in practice

Question 9

What is the purpose of a cooldown (decay) phase at the end of LLM pretraining?

Accepted Answer

Cooldown reduces learning rate near training end, allowing the optimizer to settle and reducing noise

Answer

Cooldown increases learning rate to escape local minima at the end of training

Answer

Cooldown is only needed for models larger than 70B parameters

Answer

Cooldown is applied at the start of training before warmup

Question 10

What is loss spike in LLM pretraining and how is it mitigated?

Accepted Answer

Loss spikes are sudden loss increases caused by bad batches or instability; mitigated by grad clipping and restarts

Answer

Loss spikes indicate the model has converged and training should stop

Answer

Loss spikes only occur in models using float32 precision

Answer

Loss spikes are desirable as they help the model escape local minima

Question 11

How does the Chinchilla paper (Hoffmann et al., 2022) change LLM pretraining strategy?

Accepted Answer

Chinchilla found ~20 tokens per parameter is optimal; GPT-3 was undertrained, smaller models can match it with more data

Answer

Chinchilla found 5 tokens per parameter is optimal for all model sizes

Answer

Chinchilla showed that only model size matters; data quantity is irrelevant

Answer

Chinchilla found optimal training requires 100 tokens per parameter for models over 10B

Question 12

What is continual pretraining and how does it differ from pretraining from scratch?

Accepted Answer

Continual pretraining resumes from an existing checkpoint on new data, risking catastrophic forgetting

Answer

Continual pretraining always outperforms pretraining from scratch on the new domain

Answer

Continual pretraining and instruction tuning are the same process

Answer

Continual pretraining requires freezing all layers except the last two

Question 13

What is the attention sink phenomenon observed in LLMs?

Accepted Answer

Attention sinks are tokens that absorb excess attention mass; retaining them enables long-context streaming

Answer

Attention sinks are bugs that should be fixed with better positional encoding

Answer

Attention sinks only occur in encoder models, not decoder LLMs

Answer

Attention sinks reduce model performance and are removed during post-training

Question 14

What is the difference between a decoder-only and encoder-decoder architecture for LLM pretraining?

Accepted Answer

Decoder-only uses causal next-token prediction; encoder-decoder has bidirectional encoder + causal decoder

Answer

Decoder-only models use bidirectional attention; encoder-decoder uses causal attention

Answer

Encoder-decoder models are preferred for all LLM tasks in 2026

Answer

Decoder-only models cannot perform translation or summarization tasks

Question 15

What is the FIM (Fill-In-the-Middle) objective used in code LLMs?

Accepted Answer

FIM rearranges prefix/suffix/middle so models can infill code given surrounding context

Answer

FIM converts CLM into a masked language model by randomly masking tokens

Answer

FIM is only used for natural language tasks, not code generation

Answer

FIM requires a separate encoder to process the suffix context

Question 16

What are the main sources of pretraining compute cost and how is it estimated?

Accepted Answer

Pretraining compute is ~6ND FLOPs (2N forward + 4N backward) per training token

Answer

Pretraining compute is estimated as 2ND FLOPs, accounting only for the forward pass

Answer

Compute cost scales linearly with batch size but not with model depth

Answer

The 6ND estimate assumes FP8 training; FP32 requires 24ND FLOPs

Question 17

What is perplexity as a metric in LLM pretraining and what are its limitations?

Accepted Answer

Perplexity measures token prediction quality but cannot be compared across tokenizers or models

Answer

Perplexity is comparable across models with different tokenizers

Answer

Lower perplexity always means better instruction following capability

Answer

Perplexity is the only reliable metric for pretraining evaluation

Question 18

What is a curriculum learning strategy in LLM pretraining?

Accepted Answer

Curriculum learning trains easier examples first, gradually increasing difficulty or sequence length

Answer

Curriculum learning trains on the hardest examples first for maximum gradient signal

Answer

Curriculum learning is only applicable to vision models, not LLMs

Answer

Curriculum learning always significantly outperforms random ordering in final benchmark scores

Question 19

What is embedding dimension and why does it not always scale with model depth?

Accepted Answer

d_model and depth both scale with compute budget but with an empirically tuned aspect ratio

Answer

d_model always equals the number of layers in a well-designed LLM

Answer

Increasing d_model while keeping depth fixed always improves benchmark performance

Answer

d_model must be a power of 2 for Flash Attention to work correctly

Question 20

What is the purpose of the MTP (Multi-Token Prediction) pretraining objective?

Accepted Answer

MTP predicts N future tokens simultaneously with parallel heads, improving planning and enabling speculative decoding

Answer

MTP trains multiple separate models on different token prediction tasks

Answer

MTP is equivalent to masked language modeling with N% mask rate

Answer

MTP is incompatible with causal language modeling architectures

Question 21

What is the Common Crawl dataset and how is it processed for LLM pretraining?

Accepted Answer

Common Crawl is processed via text extraction, language ID, quality filtering, and deduplication before use

Answer

Common Crawl data is used directly without any preprocessing for LLM training

Answer

Common Crawl only contains English web pages crawled after 2020

Answer

Common Crawl is a proprietary dataset available only to large AI labs

Question 22

What is the muP (Maximal Update Parametrization) and why does it matter for LLM pretraining?

Accepted Answer

muP ensures optimal hyperparameters transfer from small proxy models to large models via principled scaling

Answer

muP requires retuning hyperparameters at each model scale

Answer

muP only applies to optimizer choice, not weight initialization

Answer

muP is only compatible with SGD optimizers

Question 23

What is the role of the temperature parameter in token sampling during LLM evaluation?

Accepted Answer

Temperature < 1 sharpens the distribution (more greedy); > 1 flattens it (more random)

Answer

Higher temperature makes LLM outputs more deterministic and less creative

Answer

Temperature scaling is applied after nucleus sampling in all implementations

Answer

Temperature only affects the first token generated, not subsequent tokens

Question 24

What is instruction following capability and how is it introduced during pretraining vs post-training?

Accepted Answer

Instruction following is primarily introduced via SFT post-pretraining; pretraining produces a base completion model

Answer

Instruction following is learned entirely during pretraining from web data

Answer

All pretrained LLMs can follow instructions without any fine-tuning

Answer

Instruction following requires encoder-decoder architecture during pretraining

Question 25

What is the purpose of dropout in transformer pretraining and why is it often disabled in modern LLMs?

Accepted Answer

Modern large LLMs often disable dropout since massive data volume prevents overfitting and dropout hurts MFU

Answer

Dropout is mandatory in all LLM pretraining for preventing memorization

Answer

Dropout is only applied to the final linear layer in modern transformers

Answer

Dropout rates above 0.5 are used in LLMs to improve generalization

Question 26

What is the difference between teacher forcing and free-running generation in LLM training?

Accepted Answer

Teacher forcing feeds ground-truth tokens during training; free-running uses model predictions; gap is exposure bias

Answer

Teacher forcing uses model predictions as future inputs during training

Answer

Free-running training and teacher forcing produce identical loss values

Answer

Scheduled sampling is only used during inference, not training

Test yourself