Recurrent networks and sequences

An RNN has a recurrent connection that feeds the hidden state from one time step back into the network at the next time step, giving it a form of memory. This lets it process sequences of arbitrary length and use information from earlier inputs to influence later outputs, unlike a feedforward network that treats each input independently.

Sequential data has order and dependencies where the meaning of one element depends on what came before. RNNs share parameters across time steps and maintain a hidden state, so they can capture temporal patterns and context regardless of where in the sequence they occur.

The hidden state is a vector that summarizes information about the inputs seen so far in the sequence. At each step it is updated using the current input and the previous hidden state, acting as the network's memory that is passed forward to influence future predictions.

At each time step t the new hidden state is computed as h_t = tanh(W_xh · x_t + W_hh · h_{t-1} + b), where x_t is the current input, h_{t-1} is the previous hidden state, W_xh and W_hh are weight matrices, and b is a bias. The output can then be derived from h_t.

import numpy as np
h = np.zeros(hidden_size)
for x_t in sequence:
    h = np.tanh(W_xh @ x_t + W_hh @ h + b)

Parameter sharing means the same weight matrices are reused at every time step instead of having separate weights per position. This lets the RNN handle sequences of varying length, drastically reduces the number of parameters, and lets it recognize a pattern no matter where it appears in the sequence.

BPTT is the training algorithm for RNNs where the network is 'unrolled' across all time steps and standard backpropagation is applied through this unrolled graph. Gradients flow backward through every time step, and because the same weights are shared, the gradients from all steps are summed to update each parameter.

During BPTT, gradients are repeatedly multiplied by weight factors as they flow back through many time steps. When these factors are less than one, the gradients shrink exponentially toward zero, so the network can barely learn long-range dependencies between distant elements in a sequence.

Exploding gradients occur when gradients grow exponentially large during BPTT, causing unstable updates and NaN values. A common fix is gradient clipping, which rescales the gradient vector so its norm does not exceed a chosen threshold before applying the update.

import torch
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

LSTMs (Long Short-Term Memory) are designed to capture long-range dependencies that vanilla RNNs fail at due to vanishing gradients. They use a cell state and gating mechanisms that allow information to flow across many time steps almost unchanged, preserving important context over long sequences.

An LSTM has a forget gate, an input gate, and an output gate. The forget gate decides what to remove from the cell state, the input gate decides what new information to add, and the output gate decides what part of the cell state to expose as the hidden state.

The cell state is a long-term memory channel that runs straight through the LSTM with only minor linear interactions controlled by the gates. Because information can flow along it with little modification, gradients pass through more easily, which helps the network retain information over long sequences.

A GRU (Gated Recurrent Unit) is a simplified gated RNN that uses only two gates—an update gate and a reset gate—and merges the cell state and hidden state into one. This gives it fewer parameters and lets it run faster than an LSTM, often with comparable performance.

A GRU has an update gate and a reset gate. The update gate decides how much of the previous hidden state to keep versus replace with new information, while the reset gate decides how much past information to forget when computing the new candidate state.

A bidirectional RNN runs two RNNs over the sequence—one forward and one backward—and combines their hidden states so each position has context from both past and future. It is useful when the entire sequence is available at once, such as in text classification or named-entity recognition, but not for real-time generation.

One-to-many takes a single input and produces a sequence (e.g., image captioning); many-to-one takes a sequence and produces one output (e.g., sentiment classification); and many-to-many maps a sequence to another sequence (e.g., machine translation or per-token tagging). The choice depends on the shape of the input and the desired output.

The encoder reads the input sequence and compresses it into a context (a fixed-size vector or set of states) that summarizes its meaning. The decoder then takes this context and generates the output sequence one element at a time, commonly used in machine translation and summarization.

In a basic seq2seq model the entire input must be squeezed into one fixed-size context vector, which loses information for long sequences. Attention solves this by letting the decoder look back at all the encoder hidden states and weight them at each output step, focusing on the most relevant parts of the input.

tanh squashes activations to the range (-1, 1), keeping the hidden state bounded and centered around zero as it is repeatedly fed back through time, which helps stabilize the recurrence. Unbounded ReLU outputs can accumulate and explode across many time steps in the recurrent loop.

Teacher forcing is a training technique where, instead of feeding the model's own predicted output back as the next input, the true (ground-truth) previous token is fed in. This speeds up and stabilizes training, though it can cause a mismatch at inference time when the model must rely on its own (possibly wrong) predictions.

To process multiple sequences as a single tensor, shorter sequences are padded with a dummy value so all have equal length. A sequence mask marks which positions are real versus padding, so the model and loss function can ignore the padded positions and not learn from them.

A word embedding maps each token to a dense, low-dimensional vector of learned real numbers that captures semantic similarity, so related words lie close together. This is preferred over sparse, high-dimensional one-hot vectors because it is more compact, generalizes better, and encodes meaning rather than treating every word as unrelated.

import torch.nn as nn
embedding = nn.Embedding(num_embeddings=10000, embedding_dim=128)

At each time step the RNN predicts a probability distribution over the vocabulary for the next token, given all the tokens seen so far. Training minimizes the cross-entropy between this predicted distribution and the actual next token in the sequence.

Unrolling means expanding the recurrent loop into an equivalent feedforward-like graph with one copy of the cell per time step, all sharing the same weights. This explicit representation makes it possible to visualize the computation and to apply backpropagation through time.

Truncated BPTT splits a very long sequence into smaller chunks and backpropagates gradients only within each chunk (or a limited number of steps), while carrying the hidden state forward. It is used to reduce memory usage and computation that would be impractical if gradients had to flow through thousands of time steps.

Stacking RNN layers feeds the hidden-state sequence of one layer as the input sequence to the next, letting the network learn hierarchical representations. Lower layers can capture short-term, local patterns while higher layers capture more abstract, longer-range structure.

Transformers use self-attention to relate all positions in a sequence directly and in parallel, avoiding the step-by-step recurrence that makes RNNs slow and prone to forgetting distant information. This parallelism speeds up training on modern hardware and captures long-range dependencies more effectively.

Softmax converts the raw output scores (logits) over the vocabulary into a probability distribution that sums to one. This allows the model to assign a normalized probability to each candidate next token and to be trained with cross-entropy loss.

Greedy decoding picks the single highest-probability token at each step, which is fast but can miss better overall sequences. Beam search keeps the top-k partial sequences (beams) at each step and expands them, exploring more options to find a sequence with higher total probability at the cost of more computation.

The forget gate outputs a value between 0 and 1 for each element of the cell state. A value near 1 means 'keep' that piece of memory almost entirely, while a value near 0 means 'discard' it, allowing the LSTM to selectively retain or erase information over time.

Gates need to act like soft switches that decide how much information to let through, so an output in the range 0 to 1 is ideal. The sigmoid function naturally maps any input to (0, 1), where 0 blocks and 1 fully passes information, which is exactly what a gate requires.

This is overfitting, where the model memorizes training-specific patterns rather than learning generalizable structure. Remedies include applying dropout (often recurrent dropout), reducing model size, using more training data, or early stopping based on validation loss.

The RNN initializes a hidden state (often zeros), then loops over the sequence: at each step it combines the current input with the previous hidden state to produce a new hidden state and optionally an output. After the final step, the last hidden state encodes a summary of the whole sequence.

h = init_hidden()
outputs = []
for x_t in sequence:
    h = rnn_cell(x_t, h)
    outputs.append(h)

The cell state is the long-term memory that flows through the LSTM with minimal modification, while the hidden state is the short-term, working output exposed at each step (a filtered view of the cell state via the output gate). The hidden state is what is passed to the next layer or used for predictions.

Temperature is a parameter that scales the logits before softmax to control randomness in sampling. A higher temperature flattens the distribution, making the output more diverse and surprising, while a lower temperature sharpens it toward the most likely tokens, producing more conservative, repetitive text.

import numpy as np
def sample(logits, temperature=1.0):
    scaled = logits / temperature
    scaled -= scaled.max()  # numerical stability
    probs = np.exp(scaled)
    probs /= probs.sum()
    return np.random.choice(len(probs), p=probs)