Sequence-to-sequence models

A seq2seq model is a neural network architecture that transforms an input sequence into an output sequence that may have a completely different length. It was designed for tasks like machine translation, where a sentence in English must be converted to a sentence in French. The model learns a flexible mapping between two sequence domains rather than between a fixed input and a fixed label.

The two main components are the encoder and the decoder. The encoder reads the entire input sequence and compresses it into a fixed-size context vector representing the input's meaning. The decoder then uses that context vector to generate the output sequence one token at a time.

The encoder reads the input sequence token by token, updating its hidden state at each step using an RNN, LSTM, or Transformer. After processing all tokens, it produces a context vector — usually the final hidden state — that summarizes the entire input. This compact representation is then handed off to the decoder to begin generating the output.

The context vector is a fixed-size numerical representation — typically the encoder's final hidden state — that summarises the meaning of the entire input sequence. The decoder uses it as its starting point to produce the output. Its key limitation is that all information from an arbitrarily long input must be squeezed into this one vector, which causes quality to degrade for long sequences.

During inference the decoder generates the output sequence one token at a time. At each step it takes the context vector (or attention-weighted encoder states) plus the previously generated token, and outputs a probability distribution over the vocabulary via softmax to predict the next token. Generation continues until the model produces a special end-of-sequence token.

Teacher forcing is a training strategy where the ground-truth target token from the previous time step is fed as the decoder's next input, rather than the token the model actually predicted. This stabilises training and accelerates convergence. However, it creates an exposure bias: at inference the model must rely on its own (potentially wrong) outputs, a situation it never encountered during training.

The basic seq2seq model compresses the entire input into a single fixed-size context vector, which loses information for long inputs — often called the information bottleneck. Attention solves this by letting the decoder dynamically focus on different encoder hidden states at each generation step, rather than relying on one compressed summary. This dramatically improves performance on longer sequences such as paragraphs.

At each decoder step, a score is computed between the decoder's current hidden state and every encoder hidden state using a scoring function such as dot product or a small feed-forward network. These scores are converted to weights by softmax so they sum to 1. A weighted sum of all encoder hidden states is then calculated; this weighted sum is the context vector that the decoder uses for that particular step.

Vanilla RNNs suffer from the vanishing gradient problem: when backpropagating through many time steps, gradients shrink exponentially, making it nearly impossible to learn dependencies between distant tokens. LSTMs address this with a dedicated cell state and three learnable gates (input, forget, output) that control what information is retained or discarded. This allows LSTMs to remember relevant context across much longer sequences.

Seq2seq models are suited for tasks where both input and output are sequences of variable length, such as machine translation, text summarisation, speech recognition, and chatbots. Classifying a single image into a category (image classification) maps one fixed-size input to one label — it involves no output sequence — so it is not a seq2seq task.

Beam search is a decoding algorithm that keeps track of the top-k most probable partial sequences (called the beam) at every step instead of committing to just the single best token. By exploring multiple candidate sequences simultaneously, it avoids local optima that greedy decoding falls into. The beam width k controls the trade-off between output quality and computational cost.

Greedy decoding selects the single highest-probability token at each step — it is fast but can lock the model into a suboptimal path early on. Beam search maintains k candidate sequences and ultimately selects the best complete sequence, producing higher-quality outputs at the cost of k times more computation per step. Greedy decoding is equivalent to beam search with k = 1.

BLEU (Bilingual Evaluation Understudy) is an automated metric that measures how closely a machine-generated translation matches one or more human reference translations by comparing n-gram overlaps (unigrams, bigrams, trigrams, 4-grams). A perfect match scores 1.0 and no overlap scores 0.0. It is the standard benchmark for machine translation, though it penalises paraphrases that differ in word choice even when semantically correct.

During backpropagation through time (BPTT), gradients are multiplied by the same weight matrix at every time step; if the weights are smaller than 1, the gradients shrink exponentially and effectively vanish for early time steps. For a seq2seq encoder processing a long sentence, this means information about early words barely influences the learned weights. Gated units like LSTMs and GRUs were designed to maintain gradient flow across many steps.

A bidirectional encoder runs two separate RNNs over the input: one processes tokens left-to-right and the other right-to-left. At each position, the two hidden states are concatenated, giving the encoder access to both the past and future context of that token. This is important because translation or summarisation often depends on words that appear later in the source, which a left-to-right encoder cannot yet see.

Word embeddings convert discrete token IDs (integers representing vocabulary entries) into dense continuous vectors that the neural network can compute with. Unlike one-hot encodings that are extremely high-dimensional and sparse, embeddings (e.g., 256-dimensional vectors) capture semantic similarity: words with similar meanings end up close together in embedding space. Both the encoder and decoder start with an embedding layer to represent their respective input tokens.

The <EOS> (End of Sequence) token is a special symbol appended to every target sequence during training to mark where the output ends. During inference, the decoder generates tokens one by one until it predicts <EOS>, at which point generation stops. Without this learned stopping signal, the decoder would have no mechanism to terminate and would generate tokens indefinitely.

At each decoder time step, the model produces a probability distribution over the entire vocabulary using a softmax layer. Cross-entropy loss measures the negative log-probability of the true next token under this distribution. The total training loss is the sum (or average) of cross-entropy losses across all positions in all target sequences in the batch, and the model parameters are updated via backpropagation to minimise this loss.

The GRU (Gated Recurrent Unit) uses only two gates — a reset gate and an update gate — compared to the LSTM's three gates plus a separate cell state. This makes GRUs simpler and faster to train with fewer parameters. In practice GRUs and LSTMs achieve similar accuracy on many seq2seq benchmarks, so GRUs are preferred when training speed and model size are constraints.

When sequences in a batch are padded to the same length, the padding positions do not correspond to real tokens. Computing loss on these positions would incorrectly penalise or reward the model for pad-token predictions. We solve this by applying a padding mask: loss at pad positions is set to zero and excluded from the average, so only genuine tokens contribute to the gradient update.

A stacked encoder passes the hidden state outputs of one RNN (or Transformer) layer as the inputs to the next, creating multiple layers of processing. Lower layers typically learn surface-level features like local word order, while higher layers capture more abstract semantic patterns. Depth allows the model to build hierarchical representations, which is why state-of-the-art models like BERT and GPT use many stacked layers.

In machine translation the output must faithfully convey every piece of information from the input, just in a different language, so output length is roughly proportional to input length. In abstractive summarisation the output is much shorter and must distil only the most important information, potentially rephrasing it in ways not present verbatim in the source. Both use the same seq2seq architecture but differ in what constitutes a good output.

Introduced in the 2017 paper 'Attention Is All You Need', the Transformer replaces recurrent layers entirely with self-attention mechanisms. Because there is no sequential dependency between time steps, all input tokens can be processed in parallel on a GPU, making training dramatically faster. The Transformer also handles long-range dependencies more directly than RNNs and scales to much larger datasets and model sizes.

Self-attention lets each token in a sequence attend to every other token in the same sequence to build a richer contextual representation. Encoder–decoder attention (cross-attention) lets each decoder position attend to all encoder positions — the queries come from the decoder and the keys and values come from the encoder. Self-attention is used inside both the encoder and decoder stacks to process their respective sequences, while cross-attention is the bridge between encoder and decoder.

Because the Transformer processes all tokens in parallel rather than sequentially, it has no built-in notion of word order. Positional encodings are fixed or learned vectors added to the token embeddings before they enter the model to supply positional information. The original Transformer used sinusoidal functions of different frequencies — sin and cos at varying wavelengths — so that each position has a unique and smoothly varying encoding that generalises to sequence lengths unseen during training.

Multi-head attention runs h separate attention operations (heads) in parallel, each with its own learned linear projections for queries, keys, and values. Each head can specialise in capturing a different type of relationship — for example, one head might focus on syntactic structure while another captures semantic similarity. The outputs of all heads are concatenated and linearly projected, giving the model a much richer, multi-perspective view of the token relationships than a single attention head can provide.

The formula is Attention(Q, K, V) = softmax(QKᵀ / √d_k) V, where Q, K, V are the query, key, and value matrices and d_k is the dimension of the key vectors. Without the scaling factor √d_k, the dot products grow large in magnitude as d_k increases, pushing the softmax into regions with extremely small gradients (near-zero or near-one), making learning very slow. Dividing by √d_k keeps the dot products in a range where the softmax gradient is healthy.

Exposure bias arises because teacher forcing feeds ground-truth tokens to the decoder during training, but at inference the decoder must use its own predictions — a mismatch that causes errors to compound. Scheduled sampling gradually replaces ground-truth tokens with the model's own predictions during training, mixing both strategies on a schedule (e.g., starting at 100% ground-truth and decaying toward 100% model predictions). This bridges the gap between training and inference conditions.

The copy mechanism allows a seq2seq decoder to directly copy tokens from the source input sequence at each step instead of always generating from the fixed vocabulary. At each step the model learns a soft switch between generating a word from the vocabulary and pointing to a word in the input. This is crucial for rare words, named entities, and numbers that the model may never have seen during training and therefore cannot generate reliably.

Extractive summarisation selects and concatenates existing sentences or phrases directly from the source document without modification, guaranteeing factual faithfulness. Abstractive summarisation, which is what seq2seq models perform, generates entirely new text that paraphrases and condenses the source, similar to how a human writer would summarise. Abstractive outputs can be more fluent and concise but risk introducing inaccuracies if the model hallucinates facts.

After the multi-head self-attention sublayer, each Transformer encoder layer applies a position-wise feed-forward network: two linear transformations with a ReLU activation in between, applied identically and independently to each token's representation — FFN(x) = max(0, xW₁ + b₁)W₂ + b₂. The FFN introduces non-linear transformations that the linear attention mechanism alone cannot capture, effectively acting as a per-token reasoning step. Both sublayers are followed by layer normalisation and a residual connection.

Label smoothing replaces the hard one-hot target (probability 1 for the correct token, 0 for all others) with a soft distribution that assigns a small probability ε (e.g., 0.1) to all tokens and (1 − ε) to the correct one. This prevents the model from becoming overconfident and forces it to keep some probability mass on other tokens, acting as a form of regularisation. The original Transformer paper showed that label smoothing improved translation quality despite slightly increasing training loss.

The Embedding layer converts integer token IDs into dense vectors of size embed_dim. The LSTM processes these vectors sequentially and returns outputs (all hidden states) and the final hidden and cell states. We return hidden and cell as the context vector to initialise the decoder's LSTM state. Setting batch_first=True means the input tensor shape is (batch_size, seq_len) rather than (seq_len, batch_size).

import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True)

    def forward(self, x):
        # x: (batch_size, seq_len) — integer token IDs
        embedded = self.embedding(x)          # → (batch, seq_len, embed_dim)
        outputs, (hidden, cell) = self.rnn(embedded)
        # hidden, cell: (1, batch, hidden_dim) — context vector for decoder
        return hidden, cell

In the Transformer, each sublayer (self-attention and FFN) wraps its output with a residual connection: the sublayer's output is added to its input before layer normalisation, i.e., output = LayerNorm(x + Sublayer(x)). This allows gradients to flow directly through the network without passing through potentially saturated non-linearities, solving a key problem for very deep networks. Without residual connections, training dozens of Transformer layers would be extremely difficult due to vanishing or unstable gradients.

In text-to-text seq2seq (e.g., translation), the encoder input is a sequence of discrete token embeddings looked up from a vocabulary. In speech recognition the encoder input is a sequence of continuous acoustic feature vectors such as Mel-frequency cepstral coefficients (MFCCs) or log-Mel spectrograms — one vector per short time frame (typically every 10 ms). There is no discrete vocabulary on the input side; instead, convolutional or linear projection layers convert the raw feature frames into the hidden states that the encoder processes.