Attention and embeddings quiz

Question 1

What is a word embedding, and why is it more useful than representing words as one-hot vectors?

Accepted Answer

A dense vector that places similar words close together in space

Answer

A sparse one-hot vector with a single 1

Answer

A dictionary mapping words to their definitions

Answer

A list of grammar rules for a word

Question 2

In an embedding space, what does it usually mean geometrically when two word vectors have a high cosine similarity?

Accepted Answer

The words are likely semantically related

Answer

The words are antonyms

Answer

The words have the same length in letters

Answer

The words never appear together

Question 3

The famous analogy 'king - man + woman ≈ queen' demonstrates what property of learned word embeddings?

Accepted Answer

Linear/analogical structure encoded as vector offsets

Answer

Random noise with no meaning

Answer

Exact arithmetic on letter counts

Answer

Compression of images into text

Question 4

In one sentence, what core idea underlies an attention mechanism in neural networks?

Accepted Answer

Dynamically weighting relevant parts of the input

Answer

Sorting inputs alphabetically

Answer

Deleting unimportant words permanently

Answer

Converting text to images

Question 5

In the attention mechanism, what are the roles of the Query, Key, and Value vectors?

Accepted Answer

Query asks, Keys advertise relevance, Values carry content

Answer

Query and Key are images, Value is text

Answer

All three are identical copies of the input

Answer

Query stores output, Key stores loss, Value stores gradient

Question 6

Write the formula for scaled dot-product attention and explain why the scaling factor is used.

Accepted Answer

softmax(QKᵀ / √dₖ) · V; scaling stabilizes gradients

Answer

Q + K + V; scaling speeds up I/O

Answer

softmax(Q) · K · V; scaling increases magnitude

Answer

Q · Kᵀ only; no scaling is ever needed

Question 7

Why is the softmax function applied to attention scores before they are used to combine the value vectors?

Accepted Answer

To convert scores into weights that are non-negative and sum to 1

Answer

To make all weights exactly equal

Answer

To remove the value vectors

Answer

To convert numbers into words

Question 8

What is the difference between self-attention and cross-attention?

Accepted Answer

Self uses one sequence for Q,K,V; cross uses two different sequences

Answer

They are identical mechanisms

Answer

Self-attention has no Query vector

Answer

Cross-attention only works on images

Question 9

What is multi-head attention, and what advantage does it provide over a single attention head?

Accepted Answer

Multiple parallel attention heads capture different relationships

Answer

A single head repeated with no change

Answer

A way to delete heads to save memory

Answer

Attention applied only once at the end

Question 10

Why do Transformer models need positional encodings added to their input embeddings?

Accepted Answer

Attention alone is order-agnostic, so position info must be added

Answer

Embeddings are too small without them

Answer

To make the model slower on purpose

Answer

To convert words into numbers

Question 11

How does the Word2Vec skip-gram model learn word embeddings?

Accepted Answer

By predicting context words from a given center word

Answer

By memorizing a dictionary of definitions

Answer

By translating sentences into another language

Answer

By counting letters in each word

Question 12

What is the distributional hypothesis, and how does it relate to embeddings?

Accepted Answer

Words in similar contexts have similar meanings

Answer

Longer words are more important

Answer

Word meaning depends only on spelling

Answer

Words have fixed meanings independent of usage

Question 13

What is the key difference between static embeddings (like Word2Vec) and contextual embeddings (like those from BERT)?

Accepted Answer

Static gives one fixed vector per word; contextual varies with sentence

Answer

Contextual embeddings ignore context entirely

Answer

Static embeddings change every sentence

Answer

They are exactly the same approach

Question 14

In attention, what is an 'attention weight' and what is its typical range of values?

Accepted Answer

A value between 0 and 1 indicating how much a position contributes

Answer

Any real number from negative to positive infinity

Answer

Always exactly 0.5

Answer

An integer count of words

Question 15

Why is the dot product a natural choice for measuring similarity between a Query and a Key in attention?

Accepted Answer

It measures directional alignment and is computed efficiently

Answer

It always returns a value between 0 and 1

Answer

It ignores the direction of vectors

Answer

It only works on one-dimensional inputs

Question 16

What is a causal (masked) self-attention mask, and why is it used in language-generation models?

Accepted Answer

A mask blocking attention to future tokens during generation

Answer

A mask that hides all input tokens

Answer

A mask that doubles the attention weights

Answer

A mask that converts words to embeddings

Question 17

What does the embedding dimension (e.g., 300 or 768) represent, and what is the trade-off in choosing it?

Accepted Answer

The vector length; bigger holds more info but costs more

Answer

The number of words in the vocabulary

Answer

The number of sentences in the corpus

Answer

The accuracy of the model in percent

Question 18

Why is cosine similarity often preferred over Euclidean distance when comparing word embeddings?

Accepted Answer

It ignores magnitude and focuses on direction (meaning)

Answer

It is always smaller than Euclidean distance

Answer

It requires integer vectors

Answer

It cannot be computed with a dot product

Question 19

If the attention scores for three positions are [2.0, 1.0, 0.0], roughly what attention weights does softmax produce?

Accepted Answer

About [0.67, 0.24, 0.09]

Answer

About [0.50, 0.30, 0.20]

Answer

Exactly [2.0, 1.0, 0.0]

Answer

About [0.33, 0.33, 0.33]

Question 20

What is an embedding layer in a neural network, and how are its values typically obtained?

Accepted Answer

A learnable lookup table mapping token IDs to vectors

Answer

A fixed table that never changes

Answer

A layer that outputs probabilities

Answer

A function that counts word frequencies

Question 21

How did attention mechanisms help solve the long-range dependency problem that plagued earlier RNNs?

Accepted Answer

It gives direct connections between distant positions

Answer

It makes sequences shorter

Answer

It removes the need for any context

Answer

It only works on very short sentences

Question 22

What does it mean that self-attention is 'permutation equivariant', and why does this matter?

Accepted Answer

Reordering inputs reorders outputs the same way, ignoring order

Answer

Attention always sorts inputs

Answer

The output never changes when inputs change

Answer

It doubles the computation each time

Question 23

What is the approximate computational complexity of standard self-attention with respect to sequence length n, and why?

Accepted Answer

O(n²) because each position attends to every other

Answer

O(n) because it processes tokens once

Answer

O(log n) due to a tree structure

Answer

O(1) regardless of length

Question 24

In the original 'Attention Is All You Need' Transformer, why was attention preferred over recurrence and convolution?

Accepted Answer

It enables parallelism and direct long-range connections

Answer

It uses far fewer parameters always

Answer

It removes the need for any training data

Answer

It only works for image tasks

Question 25

What is the difference between token embeddings and sentence/document embeddings?

Accepted Answer

Token embeddings represent words; sentence embeddings represent whole texts

Answer

They are identical in all cases

Answer

Sentence embeddings represent single letters

Answer

Token embeddings cover entire paragraphs

Question 26

Why might attention weights NOT always be a reliable explanation of a model's reasoning?

Accepted Answer

High weights don't guarantee causal influence on the output

Answer

Attention weights are always random

Answer

They perfectly explain every decision

Answer

They are never computed during inference

Question 27

What is subword tokenization (e.g., Byte-Pair Encoding), and how does it relate to embeddings?

Accepted Answer

Splitting words into subunits, each with its own embedding

Answer

Treating every sentence as one token

Answer

Removing all rare words from the text

Answer

Embedding only the first letter of each word

Question 28

In cross-attention within an encoder-decoder Transformer, where do the Query, Key, and Value come from?

Accepted Answer

Query from decoder; Keys and Values from encoder output

Answer

All three from the encoder

Answer

All three from the decoder

Answer

Query from encoder; Keys/Values from decoder

Question 29

Why are word embeddings often visualized in 2D using techniques like t-SNE or PCA?

Accepted Answer

To project high-dimensional vectors into a viewable 2D/3D space

Answer

Because embeddings are originally 2D

Answer

To increase the number of dimensions

Answer

To delete unimportant words

Question 30

What problem can arise if a word embedding model is trained on biased text, and why is this important?

Accepted Answer

Embeddings can absorb and amplify societal biases in the data

Answer

Embeddings become too accurate to use

Answer

Bias only affects image models

Answer

Training data has no effect on embeddings

Test yourself