Sequence-to-sequence models quiz

Question 1

What is a sequence-to-sequence (seq2seq) model?

Accepted Answer

A neural network that maps an input sequence to an output sequence of possibly different length

Answer

A model that classifies a single input image into one of many categories

Answer

A model that always produces a single fixed-length output vector

Answer

A convolutional network architecture used for object detection

Question 2

What are the two main components of a classic seq2seq model?

Accepted Answer

Encoder and decoder

Answer

Tokenizer and embedder

Answer

Generator and discriminator

Answer

Classifier and regressor

Question 3

What is the role of the encoder in a seq2seq model?

Accepted Answer

To process the input sequence and compress it into a context vector

Answer

To generate the output sequence one word at a time

Answer

To convert output tokens back into readable text using a vocabulary lookup

Answer

To compute the cross-entropy loss during backpropagation

Question 4

What is the context vector in a basic seq2seq model, and what is its key limitation?

Accepted Answer

A fixed-size summary vector from the encoder representing the entire input, limited by its fixed size

Answer

A one-hot encoding of each input token concatenated into one long vector

Answer

The embedding of the very first token in the input sequence

Answer

The output probability distribution over the decoder vocabulary at the last step

Question 5

What does the decoder do during inference in a seq2seq model?

Accepted Answer

Generates the output sequence step by step using the context vector and previously generated tokens

Answer

Converts the input sequence into a fixed-size embedding for downstream tasks

Answer

Computes alignment scores between all pairs of input tokens

Answer

Normalises the input sequence before it is passed to the encoder

Question 6

What is teacher forcing, and what training problem does it introduce?

Accepted Answer

Feeding the ground-truth target token as the decoder input each step instead of the model's own prediction

Answer

Using a larger pre-trained model to supervise a smaller student model at every step

Answer

Adding a length penalty to the cross-entropy loss to prevent very short outputs

Answer

Forcing the output sequence to be the same length as the input sequence

Question 7

What core problem does the attention mechanism solve in seq2seq models?

Accepted Answer

The information bottleneck of compressing all input into a single fixed-size context vector

Answer

The vanishing gradient problem that makes very deep networks hard to train

Answer

The inability of RNNs to process variable-length input sequences

Answer

The slow inference speed of recurrent architectures on modern GPUs

Question 8

How does the attention mechanism compute its weighted context vector at a decoder step?

Accepted Answer

By comparing the decoder hidden state to each encoder hidden state, applying softmax, then computing a weighted sum of encoder states

Answer

By computing the cosine distance between input token embeddings at each step

Answer

By applying a fixed sliding window of size 3 over the encoder outputs

Answer

By selecting the single encoder hidden state with the highest L2 norm

Question 9

Why are LSTMs preferred over vanilla RNNs as the building block for seq2seq models?

Accepted Answer

LSTMs use gating mechanisms that mitigate vanishing gradients and capture long-range dependencies

Answer

LSTMs are faster to compute per step than vanilla RNNs

Answer

LSTMs require significantly less training data than vanilla RNNs

Answer

LSTMs can process the entire input sequence in parallel unlike vanilla RNNs

Question 10

Which of the following tasks is NOT naturally framed as a seq2seq problem?

Accepted Answer

Classifying a photograph as a cat or a dog

Answer

Machine translation from English to French

Answer

Automatic text summarisation of news articles

Answer

Speech recognition converting audio to text

Question 11

What is beam search, and why is it preferred over greedy decoding for seq2seq generation?

Accepted Answer

A decoding strategy that maintains the top-k most probable partial sequences at each step to find better final outputs

Answer

A training method that penalises the model for generating sequences longer than the target

Answer

A technique for pruning low-frequency words from the vocabulary before decoding

Answer

A regularisation method applied to the encoder hidden states during backpropagation

Question 12

How does greedy decoding differ from beam search, and what is the trade-off?

Accepted Answer

Greedy decoding picks the highest-probability token each step; beam search tracks k candidate sequences for better overall quality at higher cost

Answer

Greedy decoding uses attention while beam search does not

Answer

Beam search always selects the single best token; greedy decoding tracks k candidates

Answer

Beam search is used only during training; greedy decoding is used only at inference

Question 13

What is the BLEU score and what does it measure?

Accepted Answer

A metric that quantifies n-gram overlap between model output and human reference translations

Answer

A training loss function used in place of cross-entropy for seq2seq models

Answer

An attention variant that uses bidirectional encoders with learnable embeddings

Answer

A method for initialising word embeddings using co-occurrence statistics

Question 14

What is the vanishing gradient problem, and why does it specifically hurt seq2seq RNN encoders?

Accepted Answer

Gradients shrink exponentially during backpropagation through many time steps, preventing the encoder from learning early-word dependencies

Answer

Gradients grow too large and cause the loss to diverge to infinity

Answer

The model forgets to include rare tokens in its learnable vocabulary

Answer

Attention weights fail to sum to 1 due to floating-point precision errors

Question 15

What is a bidirectional encoder, and why does it improve seq2seq models?

Accepted Answer

An encoder that uses two separate RNNs — one left-to-right and one right-to-left — and concatenates their hidden states at each position

Answer

An encoder that processes the same input twice using identical weights to produce a more stable representation

Answer

An encoder capable of handling both audio waveforms and text tokens simultaneously

Answer

A multi-layer encoder where each layer has a progressively larger hidden dimension

Question 16

What role do word embeddings play at the input of a seq2seq model?

Accepted Answer

They convert discrete token IDs into dense continuous vectors that the neural network can process

Answer

They compute the BLEU score between the model output and the reference translation

Answer

They store the attention weights computed at each decoder step for visualisation

Answer

They determine the beam width used during inference

Question 17

What is the <EOS> token, and why is it essential in seq2seq decoding?

Accepted Answer

A special symbol appended to target sequences that signals the decoder to stop generating

Answer

A token used to replace unknown words that are outside the model vocabulary

Answer

A separator token placed between the encoder input and decoder input in the data

Answer

A padding token used to make all sequences in a batch the same length

Question 18

How is cross-entropy loss used to train a seq2seq decoder?

Accepted Answer

At each decoder step it measures how far the predicted probability distribution is from the true next token, summed over all output positions

Answer

Cross-entropy computes the cosine similarity between predicted and reference token embeddings

Answer

It calculates the BLEU score for each training example and uses it as a reward signal via REINFORCE

Answer

It penalises the model for producing output sequences longer than the reference

Question 19

How does the GRU differ from the LSTM, and when might you prefer it for seq2seq tasks?

Accepted Answer

A GRU is a simpler recurrent unit with fewer gates and parameters than LSTM, achieving comparable performance at lower computational cost

Answer

A GRU has more gates than an LSTM and is therefore more expressive

Answer

A GRU replaces recurrence entirely with self-attention mechanisms

Answer

A GRU can only be used as the encoder component, not as the decoder, in a seq2seq model

Question 20

What is cross-entropy loss computed on a padding token, and how do we handle it?

Accepted Answer

We apply a padding mask to zero out the loss at pad positions so only real tokens contribute to training

Answer

Padding tokens carry no gradients because they are not part of the computation graph

Answer

The model automatically ignores padding tokens because they have a zero embedding

Answer

Padding positions are excluded by reducing the batch size dynamically during training

Question 21

What is a stacked (deep) encoder in a seq2seq model, and what does depth buy us?

Accepted Answer

Using multiple RNN or Transformer layers where each layer's output serves as the next layer's input, enabling hierarchical feature learning

Answer

Using two separate encoders simultaneously to handle both audio and text inputs

Answer

Applying the same encoder weights multiple times in a loop to refine the context vector

Answer

A technique that stacks output sequences from multiple decoder runs to select the best one

Question 22

How does text summarisation differ from machine translation as a seq2seq task?

Accepted Answer

Translation preserves all information in a new language; summarisation compresses input to its most important points, possibly rephrasing

Answer

Translation uses the attention mechanism while summarisation uses only the plain context vector

Answer

Summarisation uses a bidirectional decoder; translation uses a unidirectional decoder

Answer

Summarisation is a classification task while translation is a sequence generation task

Question 23

What is the Transformer architecture, and how does it fundamentally differ from RNN-based seq2seq models?

Accepted Answer

A seq2seq architecture that replaces recurrence entirely with self-attention, allowing all tokens to be processed in parallel

Answer

A seq2seq model that replaces RNNs with convolutional layers for faster local feature extraction

Answer

A very deep bidirectional RNN trained on extremely large multilingual datasets

Answer

A seq2seq model that uses beam search with a beam width in the thousands to maximise output quality

Question 24

What is self-attention, and how does it differ from encoder–decoder attention?

Accepted Answer

Self-attention lets each token in a sequence attend to all other tokens in the same sequence; cross-attention lets decoder positions attend to encoder positions

Answer

Self-attention is the mechanism where the decoder attends to the encoder's hidden states

Answer

Self-attention is a regularisation technique that randomly drops attention connections during training

Answer

Self-attention reduces the sequence length by pooling adjacent tokens together

Question 25

Why do Transformer models need positional encodings, and how are they constructed?

Accepted Answer

Vectors added to token embeddings to inject word-order information since the Transformer processes tokens in parallel

Answer

Learnable embeddings that encode the part-of-speech tag of each token

Answer

Scaling factors applied to attention scores to suppress the influence of distant tokens

Answer

A layer normalisation applied to token embeddings to ensure zero mean and unit variance

Question 26

What is multi-head attention, and what advantage does it have over single-head attention?

Accepted Answer

Running h attention operations in parallel, each with different learned projections, allowing the model to jointly attend to different types of relationships

Answer

Applying a single attention operation multiple times in sequence to progressively refine representations

Answer

A beam search variant that maintains multiple hypothesis sequences at each decoding step

Answer

Using attention windows of varying sizes to capture both local and global context simultaneously

Question 27

What is the scaled dot-product attention formula, and why is the scaling factor needed?

Accepted Answer

Attention(Q, K, V) = softmax(QKᵀ / √d_k) × V

Answer

Attention(Q, K, V) = softmax(QVᵀ) × K

Answer

Attention(Q, K, V) = sigmoid(QKᵀ) + V

Answer

Attention(Q, K, V) = mean(Q + K) × V

Question 28

What is exposure bias in seq2seq training, and how can scheduled sampling reduce it?

Accepted Answer

A mismatch between training (ground-truth decoder inputs) and inference (model's own predictions) that causes errors to compound at generation time

Answer

The model over-represents common words because they appear more frequently in the training corpus

Answer

The training dataset is biased toward certain source languages or domains

Answer

The model memorises the training set instead of learning generalisable translation patterns

Question 29

What is the copy mechanism (pointer network) in seq2seq models, and what problem does it address?

Accepted Answer

A mechanism giving the decoder the option to directly copy tokens from the input, solving out-of-vocabulary and rare-word generation problems

Answer

A technique to copy pre-trained encoder weights into the decoder's initial parameters

Answer

A data augmentation method that duplicates rare training examples to balance the dataset

Answer

A beam search variant that copies the top-scoring partial hypothesis to each new beam position

Question 30

What is the difference between extractive and abstractive summarisation, and which does seq2seq perform?

Accepted Answer

Extractive summarisation selects existing sentences from the source; abstractive summarisation generates new text that may paraphrase or condense the source

Answer

Extractive summarisation uses seq2seq models; abstractive summarisation uses rule-based extraction

Answer

Abstractive summarisation copies sentences verbatim; extractive summarisation generates new text

Answer

The two approaches are architecturally identical and differ only in the evaluation metric used

Test yourself