Transformers and attention deep dive quiz

Question 1

In scaled dot-product attention, why is the dot product scaled by 1/sqrt(d_k) before the softmax?

Accepted Answer

To keep score variance near 1 and avoid softmax saturation / vanishing gradients

Answer

To normalize scores to a probability distribution summing to 1

Answer

To reduce the computational cost of the matrix multiplication

Answer

To enforce orthogonality between query and key vectors

Question 2

What is the time and memory complexity of vanilla self-attention with respect to sequence length n, and why does it become a bottleneck?

Accepted Answer

O(n^2 * d) time and O(n^2) memory

Answer

O(n log n) time, O(n) memory

Answer

O(n * d^2) time and O(n) memory

Answer

O(n^3) time and O(n^2) memory

Question 3

How does FlashAttention achieve exact attention with sub-quadratic memory while remaining mathematically identical to standard attention?

Accepted Answer

It uses block tiling plus online softmax to avoid materializing the n x n matrix, giving exact results

Answer

It approximates softmax with a low-rank kernel to skip most computations

Answer

It quantizes the attention matrix to int8 to fit it in memory

Answer

It prunes low-probability attention entries before the softmax

Question 4

Why do Transformers need positional encodings, and how do learned absolute, sinusoidal, and rotary (RoPE) approaches differ?

Accepted Answer

Attention is permutation-equivariant; RoPE injects relative position via rotation in the QK dot product

Answer

Attention is order-sensitive by default, so encodings only help with very long sequences

Answer

Positional encodings are only needed in the decoder for causal masking

Answer

Learned absolute encodings extrapolate to arbitrary lengths better than RoPE

Question 5

What is the key advantage of multi-head attention over a single attention head of equivalent total dimension?

Accepted Answer

It allows attending to multiple representation subspaces/relation types in parallel

Answer

It strictly reduces parameter count compared to a single head

Answer

It removes the need for the feed-forward sublayer

Answer

It guarantees orthogonal attention patterns across heads

Question 6

In Grouped-Query Attention (GQA) and Multi-Query Attention (MQA), what is shared and what problem does this solve?

Accepted Answer

They share key/value heads to shrink the KV cache and inference memory bandwidth

Answer

They share query projections to reduce training compute

Answer

They share feed-forward weights across layers

Answer

They eliminate positional encodings to save memory

Question 7

What is the KV cache in autoregressive Transformer decoding, and why is it used?

Accepted Answer

Stored keys/values of past tokens reused to avoid recomputation during generation

Answer

A cache of softmax probabilities reused across batches

Answer

A cache of the model's weights kept in GPU SRAM

Answer

Precomputed positional encodings stored for all positions

Question 8

Explain the difference between pre-layer-norm (Pre-LN) and post-layer-norm (Post-LN) Transformer architectures and their training implications.

Accepted Answer

Pre-LN normalizes inside the residual branch, improving gradient flow and stability for deep models

Answer

Pre-LN normalizes after the residual, requiring warmup; Post-LN does not

Answer

They are mathematically equivalent and differ only in implementation

Answer

Post-LN removes the residual connection entirely

Question 9

Why is the causal (look-ahead) mask necessary in decoder self-attention, and how is it implemented?

Accepted Answer

It prevents attending to future positions by setting their scores to -inf before softmax

Answer

It pads short sequences to equal length

Answer

It normalizes attention scores across heads

Answer

It masks out padding tokens only

Question 10

What does it mean that attention is 'permutation-equivariant', and why does this matter for sequence modeling?

Accepted Answer

Permuting inputs permutes outputs identically, so order info must be added separately

Answer

Permuting inputs leaves outputs unchanged (invariance)

Answer

Attention cannot process sequences of variable length

Answer

It means attention is invariant to scaling of the inputs

Question 11

In the standard Transformer, what is the typical dimensionality and role of the position-wise feed-forward network (FFN)?

Accepted Answer

Two linear layers (expand to ~4x then project back) applied per position with a nonlinearity

Answer

A recurrent layer applied across positions

Answer

A convolution mixing information across all positions

Answer

An attention layer over the hidden dimension

Question 12

How does RoPE (Rotary Position Embedding) encode relative position, and why does this aid length generalization?

Accepted Answer

It rotates Q and K so their dot product depends only on relative offset m-n

Answer

It adds a learned bias per relative distance to the scores

Answer

It concatenates relative-distance one-hot vectors to embeddings

Answer

It scales attention logits by the inverse of the distance

Question 13

What is an attention sink, and why does it appear in autoregressive LLMs?

Accepted Answer

Heads dumping excess softmax mass onto initial token(s) as a no-op anchor

Answer

A bug where attention weights fail to sum to 1

Answer

A method to merge attention heads to save memory

Answer

The decay of attention to distant tokens over depth

Question 14

Why can't you simply drop the softmax to make attention linear, and what do linear attention methods do instead?

Accepted Answer

Softmax normalization couples keys; linear attention uses a kernel feature map and associativity for O(n)

Answer

Softmax is only for numerical stability and can be removed freely

Answer

Dropping softmax makes attention non-differentiable

Answer

Linear attention requires recurrence and cannot be parallelized

Question 15

What is the purpose of residual connections in Transformer blocks, and how do they interact with depth?

Accepted Answer

They provide an identity gradient path enabling stable training of deep stacks

Answer

They reduce parameter count by sharing weights

Answer

They replace the need for attention in deep models

Answer

They enforce that each layer learns orthogonal features

Question 16

In encoder-decoder Transformers, what are the queries, keys, and values in cross-attention?

Accepted Answer

Q from decoder, K and V from encoder output

Answer

Q, K, V all from the encoder

Answer

Q and K from decoder, V from encoder

Answer

Q from encoder, K and V from decoder

Question 17

What problem does Sliding Window Attention (as in Longformer/Mistral) solve, and what is its complexity?

Accepted Answer

It limits attention to a local window of size w, giving O(n*w) complexity

Answer

It compresses embeddings to lower dimension; O(n*d)

Answer

It removes positional encodings to handle long context

Answer

It caches all keys/values to avoid recomputation

Question 18

Why does the original Transformer scale token embeddings by sqrt(d_model) before adding positional encodings?

Accepted Answer

To balance embedding magnitude against unit-scale positional encodings

Answer

To normalize embeddings to unit norm

Answer

To match the attention scaling factor 1/sqrt(d_k)

Answer

To reduce overfitting via implicit regularization

Question 19

What is the difference between BERT-style (encoder-only, MLM) and GPT-style (decoder-only, causal LM) attention, and what tasks suit each?

Accepted Answer

BERT is bidirectional (MLM, understanding); GPT is causal (next-token, generation)

Answer

Both use causal masking; they differ only in tokenizer

Answer

BERT is decoder-only and GPT is encoder-only

Answer

They use identical attention but differ in positional encoding only

Question 20

How does the temperature parameter in the softmax of attention (or sampling) affect the resulting distribution?

Accepted Answer

Lower T sharpens (more peaked); higher T flattens toward uniform

Answer

Higher T sharpens the distribution toward argmax

Answer

Temperature only affects gradients, not the distribution

Answer

Temperature scales the values V rather than the logits

Question 21

What is the intuition behind 'induction heads' and why are they significant for in-context learning?

Accepted Answer

Heads implementing prefix-matching copy (A B ... A -> B), linked to in-context learning

Answer

Heads that normalize attention across the batch

Answer

Heads that prune redundant tokens for efficiency

Answer

Heads dedicated only to positional encoding

Question 22

Why is LayerNorm preferred over BatchNorm in Transformers?

Accepted Answer

LayerNorm normalizes per-token over features, independent of batch/sequence statistics

Answer

LayerNorm is faster to compute on GPUs

Answer

BatchNorm cannot be backpropagated through sequences

Answer

LayerNorm requires no learnable parameters

Question 23

What is RMSNorm and why is it used in many modern LLMs (e.g., LLaMA) instead of LayerNorm?

Accepted Answer

It scales by RMS without mean-centering, cheaper than LayerNorm

Answer

It normalizes using batch statistics for speed

Answer

It applies normalization only at inference time

Answer

It replaces softmax in the attention layer

Question 24

In multi-head attention, if d_model=512 and there are 8 heads, what is d_k per head and how are head outputs combined?

Accepted Answer

d_k=64; outputs are concatenated then projected by W_O

Answer

d_k=512; outputs are summed

Answer

d_k=64; outputs are averaged across heads

Answer

d_k=4096; outputs are multiplied elementwise

Question 25

What is the difference between absolute positional bias and ALiBi (Attention with Linear Biases)?

Accepted Answer

ALiBi adds a distance-proportional bias to attention scores, enabling length extrapolation

Answer

ALiBi learns absolute position embeddings per layer

Answer

ALiBi rotates queries and keys like RoPE

Answer

ALiBi removes the softmax to handle long sequences

Question 26

Why are attention weights generally a poor explanation of model 'importance' or causality?

Accepted Answer

Attention weights need not be causal/faithful; info also flows via residuals and values

Answer

Attention weights are always a faithful, causal explanation

Answer

Attention weights cannot be visualized

Answer

Attention weights only matter in the final layer

Question 27

What is the online (streaming) softmax used in FlashAttention, and what running statistics does it maintain?

Accepted Answer

It maintains a running max and running exp-sum, rescaling the accumulator per block

Answer

It maintains only the sum of values V

Answer

It stores the full logit matrix in registers

Answer

It approximates softmax with a Taylor expansion

Question 28

How does Mixture-of-Experts (MoE) change the feed-forward layer, and what is the trade-off?

Accepted Answer

It uses a router to send tokens to top-k expert FFNs, decoupling capacity from per-token FLOPs

Answer

It replaces attention with multiple experts

Answer

It averages outputs of all FFNs for every token

Answer

It shares one FFN across all layers to save memory

Question 29

What is the role of the output projection matrix W_O after concatenating attention heads?

Accepted Answer

It linearly mixes concatenated head outputs back into the d_model residual stream

Answer

It applies softmax across heads

Answer

It computes the keys for the next layer

Answer

It normalizes the attention scores

Question 30

Explain why context-length extension via positional interpolation (e.g., for RoPE) often works better than naive extrapolation.

Accepted Answer

Interpolation keeps rotation phases within the trained range, avoiding OOD high frequencies

Answer

Interpolation adds new learned parameters per position

Answer

Extrapolation is better but slower to compute

Answer

They are equivalent in effect on model quality

Test yourself