Transformers quiz

Question 1

What is the Transformer architecture and why was it revolutionary?

Accepted Answer

Replaced recurrence with parallel self-attention, enabling O(1) path length between any positions

Answer

Added CNN layers to RNN

Answer

Only for machine translation

Answer

Requires sequential computation like RNN

Question 2

What is the encoder-decoder Transformer and how is it used for translation?

Accepted Answer

Encoder: bidirectional context; decoder: masked self-attention + cross-attention for generation

Answer

Only encoder is needed for translation

Answer

Decoder has no cross-attention

Answer

Encoder processes target sequence

Question 3

What is BERT and how is it pre-trained?

Accepted Answer

Pre-trained with masked language modeling (predict masked tokens) and next sentence prediction

Answer

Pre-trained with left-to-right language modeling

Answer

Uses decoder-only architecture

Answer

Requires labeled data for pre-training

Question 4

What is fine-tuning a pre-trained Transformer?

Accepted Answer

Load pre-trained weights, add task head, train on labeled data with small LR

Answer

Train from scratch on task data

Answer

Only update the last layer

Answer

Fine-tuning never changes pre-trained weights

Question 5

What is the [CLS] and [SEP] token in BERT?

Accepted Answer

[CLS] aggregates sequence-level info for classification; [SEP] separates sentences

Answer

[CLS] is a random initialization

Answer

[SEP] predicts the label

Answer

Only [SEP] is used for classification

Question 6

What is T5 (Text-to-Text Transfer Transformer)?

Accepted Answer

Frames all NLP tasks as text-to-text: encode text prompt, decode text answer

Answer

Uses decoder-only architecture

Answer

Cannot handle classification

Answer

Requires task-specific output heads

Question 7

What is few-shot and zero-shot prompting with GPT?

Accepted Answer

Zero-shot: task description only; few-shot: 2-10 examples in prompt; no weight update

Answer

Few-shot requires fine-tuning weights

Answer

Zero-shot requires examples

Answer

Only works for classification

Question 8

What is instruction fine-tuning (IFT)?

Accepted Answer

Fine-tuning on (instruction, response) pairs to follow human instructions across diverse tasks

Answer

Only fine-tunes on classification tasks

Answer

Replaces pre-training

Answer

Doesn't change model weights

Question 9

What is GELU activation and why is it preferred in Transformers?

Accepted Answer

x·Φ(x): smooth probabilistic gating, differentiable everywhere, outperforms ReLU in Transformers

Answer

Same as ReLU but shifted

Answer

Uses step function

Answer

Only defined for positive inputs

Question 10

What are embedding layers in a Transformer?

Accepted Answer

Three summed embeddings: token + positional + segment, all learned jointly

Answer

Only token embeddings, no positional

Answer

Fixed random embeddings

Answer

Output head always has separate weights

Question 11

What is chain-of-thought (CoT) prompting?

Accepted Answer

Few-shot examples with explicit reasoning steps improve arithmetic and logical reasoning

Answer

Always requires 10+ examples

Answer

Only works for math problems

Answer

Reduces model accuracy

Question 12

What is attention head pruning?

Accepted Answer

Remove redundant attention heads (low importance score) to reduce FLOPs without quality loss

Answer

Always requires fine-tuning after pruning

Answer

Removes entire layers not heads

Answer

Only weight-level pruning

Question 13

What is the position of layer normalization in pre-LN vs post-LN Transformers?

Accepted Answer

Pre-LN: LN before sublayer (stable, no warmup); post-LN: after residual (needs warmup)

Answer

Post-LN is always better

Answer

RMSNorm replaces attention

Answer

Pre-LN applies LN after the residual

Question 14

What is BART and how does it differ from BERT and GPT?

Accepted Answer

Denoising pre-training: corrupt input, decode to original; encoder-decoder combining BERT+GPT strengths

Answer

Encoder-only like BERT

Answer

Pre-trained with causal LM only

Answer

Cannot do generation

Question 15

What is the Transformer's inductive bias and how does it compare to CNN/RNN?

Accepted Answer

Transformers: permutation equivariant, global attention, less local/sequential prior than CNN/RNN

Answer

Transformers have strong locality bias

Answer

CNNs outperform Transformers at scale

Answer

Transformers require sequential data

Question 16

What is the difference between causal and non-causal language model pre-training?

Accepted Answer

Causal LM predicts next token (all tokens trained); MLM predicts masked tokens (~15% per step)

Answer

Both train on all tokens equally

Answer

Causal LM is more data efficient

Answer

MLM is used for generation

Question 17

What is Retrieval-Augmented Generation (RAG)?

Accepted Answer

Retrieve relevant documents at query time, concatenate with input for grounded generation

Answer

Only used for classification

Answer

Requires full model retraining to update knowledge

Answer

Dense retrieval replaces the Transformer encoder

Question 18

What is AlphaFold 2 and the role of Transformers in it?

Accepted Answer

Evoformer (Transformer-like) processes MSA + pair representations for protein structure prediction

Answer

Standard BERT applied to protein sequences

Answer

CNN-only architecture

Answer

Uses LSTM not Transformer

Question 19

What is the role of pre-training data in Transformer quality?

Accepted Answer

Data scale, deduplication, and careful mixture (web+books+code) are the main quality drivers

Answer

Model architecture alone determines quality

Answer

Only labeled data matters

Answer

Pre-training data doesn't affect fine-tuning quality

Question 20

What is perplexity and how is it used to evaluate language models?

Accepted Answer

exp(average negative log-likelihood); lower = better prediction; measures uncertainty per token

Answer

Cross-entropy loss directly

Answer

Only for classification tasks

Answer

Higher perplexity = better model

Question 21

What is the Transformer's encoder output and how is it used?

Accepted Answer

Sequence of contextual embeddings used for classification or fed to decoder via cross-attention

Answer

Single vector always

Answer

Sequence of token IDs

Answer

Only used for BERT classification

Question 22

What is XLNet and how does it improve on BERT?

Accepted Answer

Permutation-based autoregressive LM: bidirectional context without masking artifacts

Answer

Uses masked attention like BERT

Answer

Only works for text generation

Answer

Same architecture as GPT

Question 23

What is the SwiGLU activation used in modern LLMs?

Accepted Answer

Gated FFN: (xW_1 ⊙ Swish(xW_3))W_2; outperforms GELU/ReLU on language modeling

Answer

Same as GELU

Answer

Only used in image models

Answer

Requires fewer matrices than standard FFN

Question 24

What is the token prediction head in an LLM?

Accepted Answer

Maps hidden state to vocab logits via W_E^T; probability = softmax; trained with cross-entropy

Answer

Uses separate weight matrix from embeddings always

Answer

Only used at the first token

Answer

Outputs character probabilities not token probabilities

Question 25

What is the role of the softmax in the Transformer output?

Accepted Answer

Converts logits to probability distribution summing to 1; used in output prediction and attention weights

Answer

Converts probabilities to logits

Answer

Applied only during training

Answer

Outputs binary 0/1

Question 26

What is beam search in Transformer generation?

Accepted Answer

Keeps top-B most probable partial sequences at each step for more coherent outputs vs greedy

Answer

Same as greedy decoding

Answer

Random sampling strategy

Answer

Requires B=1 always

Test yourself