Attention mechanism quiz

Question 1

What is the core intuition behind the attention mechanism in neural networks?

Accepted Answer

It lets the model focus on relevant parts of the input for each output step

Answer

It forces the model to read input tokens strictly one by one

Answer

It compresses the entire input into a single fixed vector

Answer

It replaces all hidden states with a lookup table

Question 2

In the attention mechanism, what roles do Queries (Q), Keys (K), and Values (V) play?

Accepted Answer

Q is what you look for, K is what tokens advertise, V is the content retrieved

Answer

Q is the input, K is the output, V is the hidden state

Answer

Q is the value to retrieve, K is the attention weight, V is the query

Answer

Q is a bias term, K is a weight matrix, V is an activation function

Question 3

Why was the attention mechanism introduced in sequence-to-sequence (encoder-decoder) models?

Accepted Answer

To overcome the bottleneck of compressing long inputs into one fixed vector

Answer

To speed up training by skipping some input tokens entirely

Answer

To reduce the number of parameters in the model significantly

Answer

To handle only short sequences more efficiently than LSTMs

Question 4

What role does the softmax function play in the attention mechanism?

Accepted Answer

It converts raw dot-product scores into a probability distribution summing to 1

Answer

It normalizes input embeddings to unit length before attention

Answer

It selects only the single most relevant token (top-1)

Answer

It applies non-linearity directly to the value vectors

Question 5

What is self-attention, and how does it differ from attention in encoder-decoder models?

Accepted Answer

Self-attention computes attention between tokens within the same sequence

Answer

Self-attention attends to an external memory bank rather than input tokens

Answer

Self-attention only looks at adjacent tokens like a sliding window

Answer

Self-attention applies attention only between encoder and decoder layers

Question 6

What is the formula for scaled dot-product attention, and what does each term do?

Accepted Answer

softmax(Q·Kᵀ / √d_k) · V

Answer

softmax(Q·Vᵀ / d_k) · K

Answer

sigmoid(Q·Kᵀ) · V

Answer

softmax(K·Vᵀ / √d_k) · Q

Question 7

Why do we divide the dot-product scores by √d_k in scaled dot-product attention?

Accepted Answer

To prevent large dot products from saturating the softmax and killing gradients

Answer

To normalize key vectors to unit length before computing attention

Answer

To make the computation faster by reducing numerical values

Answer

To ensure the output has the same dimension as the input embedding

Question 8

What is multi-head attention, and why is it more powerful than single-head attention?

Accepted Answer

Multi-head attention runs parallel attention operations, each learning different relationships

Answer

Multi-head attention applies attention multiple times to the same Q, K, V without any projection

Answer

Multi-head attention uses multiple softmax temperatures on the same attention scores

Answer

Multi-head attention is dropout applied several times to attention weights

Question 9

Why do Transformers need positional encoding, and how is it typically added?

Accepted Answer

Positional encoding adds position information because self-attention is order-agnostic

Answer

Positional encoding replaces word embeddings with position-based features entirely

Answer

Positional encoding is used only during inference, not during training

Answer

Positional encoding reduces the embedding dimension to save GPU memory

Question 10

How does attention handle long-range dependencies better than RNNs?

Accepted Answer

Attention directly connects any two tokens in one step, bypassing sequential propagation

Answer

Attention reads the sequence backwards so early tokens are closer to the output

Answer

Attention uses LSTM-style gating to preserve long-range information

Answer

Attention splits long sequences into chunks and processes each independently

Question 11

What does the attention score between token i and token j represent intuitively?

Accepted Answer

How much one token should attend to another, measured by query-key dot product

Answer

The Euclidean distance between the two token embedding vectors

Answer

The frequency with which both tokens co-occur in the training corpus

Answer

The probability that both tokens belong to the same named entity

Question 12

What is the context vector in an attention-based encoder-decoder model?

Accepted Answer

A weighted sum of encoder states reflecting relevance to the current decoder step

Answer

The final hidden state of the encoder, passed once to initialize the decoder

Answer

The embedding of the most recently generated output token

Answer

A fixed compressed summary of the entire input sequence

Question 13

What is the difference between hard attention and soft attention?

Accepted Answer

Soft attention uses differentiable weighted averages; hard attention selects discrete positions

Answer

Hard attention uses large weight matrices; soft attention uses smaller ones

Answer

Hard attention uses softmax; soft attention uses sigmoid for normalization

Answer

Soft attention attends to the whole sequence; hard attention only uses the first token

Question 14

What is the difference between cross-attention and self-attention in the Transformer architecture?

Accepted Answer

In cross-attention, Q comes from one sequence and K, V come from another sequence

Answer

Cross-attention uses a different scaling factor (√d_v) instead of √d_k

Answer

Cross-attention is attention applied after the feed-forward layer in each block

Answer

Self-attention uses learnable weights; cross-attention uses fixed cosine similarity

Question 15

What is the time and space complexity of self-attention with respect to sequence length n?

Accepted Answer

O(n²) in both time and space, due to pairwise token comparisons

Answer

O(n log n) in time and O(n) in space

Answer

O(n) in time and O(n²) in space

Answer

O(n³) in time because three separate matrix multiplications are needed

Question 16

What is masked self-attention in the Transformer decoder, and why is it necessary?

Accepted Answer

It prevents decoder tokens from attending to future positions they should not yet see

Answer

It masks padding tokens so the model does not attend to empty positions

Answer

It randomly drops attention weights during training as a form of regularization

Answer

It masks encoder states that are not linguistically relevant to the decoder

Question 17

What is the purpose of a padding mask in the attention mechanism, and how is it applied?

Accepted Answer

It sets attention scores for padding positions to near-negative-infinity before softmax

Answer

It prevents the model from generating padding tokens in the output sequence

Answer

It doubles the attention weights on the first and last real (non-pad) tokens

Answer

It removes padding tokens from the input tensor before attention is computed

Question 18

What is Bahdanau attention, and how does it compute attention scores?

Accepted Answer

It uses a small neural network (tanh of summed projections) to compute attention scores

Answer

It computes scores by directly taking the dot product of encoder and decoder states

Answer

It uses cosine similarity between query and key vectors

Answer

It assigns a single learned scalar weight to each encoder hidden state

Question 19

How does Luong attention differ from Bahdanau attention in its score computation?

Accepted Answer

Luong attention uses multiplicative (dot-product-based) scoring without a tanh nonlinearity

Answer

Luong attention uses additive scoring with a tanh non-linearity like Bahdanau

Answer

Luong attention only ever attends to the final encoder hidden state

Answer

Luong attention replaces dot products with convolution over encoder states

Question 20

What can visualizing attention weight matrices reveal about what a language model has learned?

Accepted Answer

Attention matrices show which tokens are attended to most, revealing learned linguistic patterns

Answer

Attention matrices reveal the gradient magnitudes during backpropagation

Answer

Attention matrices represent learned word embeddings in a 2D projection

Answer

Attention matrices store the weights of the feed-forward sublayers

Question 21

If multi-head attention uses h=8 heads each with dimension d_model/h, how does this compare to single-head attention of dimension d_model in terms of capacity?

Accepted Answer

Each head specializes in different relationships, yielding richer multi-perspective representations

Answer

It reduces total computation by h times since each head only uses d_model/h dimensions

Answer

It allows the model to use a different softmax temperature per head for diversity

Answer

It eliminates the need for positional encoding by distributing positional info across heads

Question 22

In the following Python snippet implementing self-attention, what does the variable `weights` represent after the manual softmax?

```python
import numpy as np
def self_attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)
    weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True)
    return weights @ V
```

Accepted Answer

The normalized attention weights showing how much each token attends to each other

Answer

The raw dot-product similarity scores before any normalization

Answer

The final output context vectors after mixing the value matrix

Answer

The gradient of the loss with respect to the query matrix Q

Question 23

The Transformer uses positional encoding PE(pos, 2i) = sin(pos / 10000^(2i/d_model)). Why are different frequencies used for different embedding dimensions i?

Accepted Answer

Different frequencies let the model represent position at multiple scales simultaneously

Answer

Different frequencies prevent gradient vanishing in very deep attention stacks

Answer

Different frequencies ensure each embedding dimension has unit variance after encoding

Answer

Different frequencies make positional encodings orthogonal to word embeddings

Question 24

In a standard Transformer, how does attention work differently in the encoder compared to the decoder?

Accepted Answer

Encoder uses bidirectional self-attention; decoder uses masked self-attention plus cross-attention

Answer

The encoder uses cross-attention and the decoder uses bidirectional self-attention

Answer

Both encoder and decoder use identical masked self-attention layers

Answer

The encoder uses positional attention and the decoder uses semantic attention

Question 25

Why must causal masking be applied during training of a language model, not just at inference time?

Accepted Answer

Without it during training, the model sees future tokens it is predicting, making learning trivial

Answer

Causal masking is applied only at inference to prevent reading the full output

Answer

Causal masking speeds up training by reducing the effective sequence length processed

Answer

Causal masking prevents encoder states from influencing early decoder positions

Question 26

What is relative positional encoding, and why might it generalize better than absolute positional encoding?

Accepted Answer

Relative encoding captures token-to-token distances, generalizing better to unseen lengths

Answer

Relative encoding uses learnable parameters; absolute encoding uses fixed sine/cosine waves

Answer

Relative encoding is added after the attention layer rather than to the input embeddings

Answer

Relative encoding dedicates one attention head solely to positional information

Question 27

How does self-attention enable full parallelism during training, whereas RNNs cannot be fully parallelized?

Accepted Answer

Self-attention processes all token pairs simultaneously via matrix operations, unlike sequential RNNs

Answer

Self-attention uses fewer parameters than RNNs, reducing communication overhead between GPUs

Answer

Self-attention breaks sequences into non-overlapping chunks processed in parallel

Answer

Self-attention avoids backpropagation entirely, which is the main bottleneck in RNNs

Question 28

What is sparse attention, and in what situations is it preferred over full self-attention?

Accepted Answer

Sparse attention limits each token to a subset of positions, reducing O(n²) memory and compute

Answer

Sparse attention randomly drops some attention heads during training as regularization

Answer

Sparse attention is identical to hard attention — it attends to exactly one token

Answer

Sparse attention uses fewer key dimensions per head to lower memory usage

Question 29

What is key-value (KV) caching in Transformer inference, and why does it drastically speed up text generation?

Accepted Answer

KV caching stores and reuses key and value vectors of past tokens to avoid recomputation

Answer

KV caching stores previously computed attention weights to skip softmax for repeated queries

Answer

KV caching compresses all model weights into GPU L2 cache for faster matrix multiply

Answer

KV caching merges multiple tokens into one key-value pair to shorten the sequence

Question 30

Empirical studies of trained BERT models show that different attention heads specialize in different linguistic phenomena. Give two concrete examples of such specialization.

Accepted Answer

Heads can specialize in syntactic dependencies, coreference, or positional patterns without explicit supervision

Answer

One head always processes nouns and another always processes verbs, by explicit design

Answer

All heads learn identical patterns but with different random seeds for diversity

Answer

Each head specializes in exactly one language of the multilingual pretraining data

Test yourself