Attention Mechanisms quiz

Question 1

What is the core idea behind attention mechanisms in neural networks?

Accepted Answer

Weighted sum of values based on query-key relevance scores

Answer

Fixed linear combination of all inputs

Answer

Convolutional feature extraction

Answer

Recurrent state update

Question 2

Why do we scale by √d_k in dot-product attention?

Accepted Answer

Prevents softmax saturation from large dot products in high dimensions

Answer

Increases computational speed

Answer

Normalizes the value vectors

Answer

Removes the need for layer normalization

Question 3

What is self-attention vs cross-attention?

Accepted Answer

Self-attention: Q,K,V from same sequence; cross-attention: Q from one, K,V from another

Answer

They are identical

Answer

Cross-attention has no keys

Answer

Self-attention is only used in decoders

Question 4

What is the computational complexity of self-attention?

Accepted Answer

O(n²·d) time and O(n²) memory due to pairwise Q·K^T scores

Answer

O(n·d) — same as RNN

Answer

O(n log n) always

Answer

O(d²) regardless of sequence length

Question 5

What is Luong (multiplicative) attention?

Accepted Answer

Dot product or bilinear score between decoder and encoder states

Answer

Additive score using a feedforward network

Answer

Only works for images

Answer

Uses convolutional alignment

Question 6

What is the difference between hard and soft attention?

Accepted Answer

Soft: differentiable weighted sum; hard: stochastic single selection

Answer

Soft attention selects one position

Answer

Hard attention uses sigmoid

Answer

They are identical in implementation

Question 7

What is the attention mask and how is it used in Transformers?

Accepted Answer

Prevents attending to padding tokens or future positions by adding -inf before softmax

Answer

Multiplies attention weights by 0

Answer

Skips certain layers

Answer

Removes rows from the attention matrix

Question 8

How does attention enable the Transformer to replace recurrence?

Accepted Answer

Computes all pairwise relations in parallel in O(1) steps vs O(n) sequential RNN steps

Answer

Attention requires sequential computation too

Answer

Transformers still use RNN for long sequences

Answer

Self-attention has O(n) path length between positions

Question 9

What is relative positional encoding?

Accepted Answer

Injects relative distance (i-j) into attention scores for better length generalization

Answer

Always absolute position only

Answer

Only used in decoders

Answer

Replaces positional encoding with dropout

Question 10

What is the role of Q, K, V projections in multi-head attention?

Accepted Answer

Separate learned projections for query, key, value roles in attention

Answer

Shared weights for efficiency

Answer

Only K and V are projected

Answer

Learned only during fine-tuning

Question 11

What is sparse attention?

Accepted Answer

Restricts each token to attend to a fixed subset of positions to reduce O(n²) cost

Answer

Always reduces attention to 1 position

Answer

Only used for image attention

Answer

No performance trade-off vs full attention

Question 12

What is the feed-forward network in a Transformer block?

Accepted Answer

2-layer MLP applied identically to each position after attention

Answer

Recurrent network applied sequentially

Answer

Convolutional layer replacing attention

Answer

Only used during pre-training

Question 13

What is the intuition behind using Query-Key similarity for attention?

Accepted Answer

Q = what I need, K = what's available, V = the content; attention = soft database lookup

Answer

Q and K are always identical

Answer

V is the query in self-attention

Answer

Scores are computed by cross-entropy

Question 14

What is attention rollout for visualization?

Accepted Answer

Aggregates attention across layers by multiplying matrices, accounting for residual connections

Answer

Averages per-head attention weights

Answer

Only shows last-layer attention

Answer

Requires gradient computation

Question 15

What is attention temperature?

Accepted Answer

Low temperature → spiky concentration; high temperature → uniform distribution

Answer

Higher temperature always improves quality

Answer

Temperature only affects training, not inference

Answer

Temperature replaces scaling by √d_k

Question 16

What is BERT's attention and how does it differ from GPT's?

Accepted Answer

BERT: bidirectional (all-to-all); GPT: unidirectional (left-to-right causal)

Answer

Both are bidirectional

Answer

BERT is autoregressive

Answer

GPT attends to all positions simultaneously

Question 17

What is the role of layer normalization in attention blocks?

Accepted Answer

Normalizes across features per sample for stable residual stream

Answer

Normalizes across the batch dimension

Answer

Replaces attention weights

Answer

Only applied at the output layer

Question 18

What is the key-value cache in autoregressive Transformer inference?

Accepted Answer

Caches past K and V to avoid recomputation at each decoding step

Answer

Caches the full attention matrix

Answer

Only stores the query vectors

Answer

Only used during training

Question 19

What is Rotary Positional Embedding (RoPE)?

Accepted Answer

Rotates Q and K vectors by position-dependent angles so Q_m·K_n depends on relative position m-n

Answer

Adds sinusoidal vectors to token embeddings

Answer

Only works for short sequences

Answer

Replaces attention with position embeddings

Question 20

How does attention contribute to interpretability?

Accepted Answer

Attention heat maps show token relevance but may not be faithful; gradient weighting improves reliability

Answer

Attention weights are always perfectly faithful

Answer

Attention cannot be visualized

Answer

Only cross-attention is interpretable

Question 21

What is attention entropy and what does it measure?

Accepted Answer

Measures distribution spread: low entropy = concentrated; high entropy = diffuse

Answer

Measures model perplexity

Answer

Entropy of token embeddings

Answer

Loss function for attention training

Question 22

What is a causal language model and how does masked attention enable it?

Accepted Answer

Masked attention lets model train on all positions in parallel while enforcing left-to-right causality

Answer

Requires sequential forward passes during training

Answer

Uses cross-attention not self-attention

Answer

Can see future tokens during training

Question 23

What is the sentence representation from BERT using CLS token?

Accepted Answer

[CLS] final hidden state aggregates full sequence for classification

Answer

[CLS] only attends to itself

Answer

[CLS] is the last token in BERT

Answer

BERT uses [SEP] for classification

Question 24

What is the 'induction head' in attention mechanism research?

Accepted Answer

Pairs of heads implementing A→B pattern completion, key to in-context learning

Answer

Heads that attend only to the first token

Answer

Heads with lowest entropy

Answer

Heads that compute self-loops

Question 25

What is context length in Transformers and what limits it?

Accepted Answer

Limited by O(n²) memory/compute and positional encoding generalization

Answer

Only limited by RAM

Answer

Transformers have no practical context limit

Answer

Context length only affects training

Question 26

What is the 'attention sink' phenomenon?

Accepted Answer

First tokens receive disproportionate attention as 'dump' positions; removing them hurts long-context performance

Answer

First tokens are the most important semantically

Answer

Attention sinks are a training bug

Answer

Attention sinks only appear in vision models

Question 27

What is the difference between additive and scaled dot-product attention?

Accepted Answer

Additive: feedforward score; dot-product: q·k/√d_k — simpler, faster, GPU-efficient

Answer

Additive is always more accurate

Answer

Dot-product requires more parameters

Answer

They use the same formula

Test yourself