Attention Mechanisms

Attention allows a model to focus on the most relevant parts of the input when producing each output element, rather than compressing everything into a single fixed vector. Given a query q and a set of key-value pairs (K,V), attention computes a weighted sum of values: output = Σ α_i v_i, where weights α_i = softmax(score(q, k_i)). The score function measures relevance between query and each key. This gives the model dynamic, input-dependent focus — crucial for long sequences where a single context vector loses information.

Scaled dot-product attention computes: Attention(Q,K,V) = softmax(Q K^T / √d_k) V. Q (queries), K (keys), V (values) are matrices; d_k is the key dimension. The scaling by √d_k prevents the dot products from growing large in high dimensions (which would push softmax into near-zero gradient regions). This is the core building block of Transformers. The output is a weighted combination of values, where weights reflect how much each key matches the query. It runs efficiently as matrix multiplication.

import torch.nn.functional as F
import math
def scaled_dot_product(Q, K, V):
    d_k = Q.size(-1)
    scores = Q @ K.transpose(-2,-1) / math.sqrt(d_k)
    return F.softmax(scores, dim=-1) @ V

In high dimensions d_k, dot products Q·K^T have variance proportional to d_k (assuming unit-variance Q, K). Large dot products push softmax into saturation — all probability mass concentrates on one key, gradients become near zero. Dividing by √d_k normalizes variance back to ~1 regardless of dimension, keeping softmax in a useful, smooth regime. This preserves stable gradients and allows the model to produce soft, distributed attention weights rather than near-hard max, which is essential for effective learning across all attention heads.

Multi-head attention runs h parallel attention functions on different linear projections of Q, K, V: head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V). Outputs are concatenated and projected: MultiHead(Q,K,V) = Concat(head_1,...,head_h) W^O. Each head can attend to different aspects of the sequence simultaneously (e.g., syntactic vs. semantic relations). Using h=8 heads with d_k = d_model/h maintains total computation cost. Multi-head attention gives richer representations than single-head attention of the same dimension.

Self-attention: Q, K, V all come from the same sequence (e.g., encoder attending to itself). Each position attends to all others in the same sequence, capturing intra-sequence dependencies — how each word relates to every other word. Cross-attention: Q comes from one sequence (e.g., decoder), K and V come from another (e.g., encoder output). This is how the decoder in Transformer seq2seq attends to the input — it queries encoder representations to pull relevant context. Self-attention enables parallel contextual encoding; cross-attention enables inter-sequence alignment.

Self-attention has O(n²·d) time and O(n²) memory complexity, where n is sequence length and d is model dimension. This comes from computing all pairwise Q K^T scores — an n×n attention matrix. For long sequences, this quadratic scaling becomes prohibitive: for n=4096 tokens, n²=16M scores. This has motivated efficient attention variants: sparse attention (attend to local windows), linear attention (O(n) approximations via kernel tricks), Longformer, BigBird, and Flash Attention (memory-efficient exact attention). The quadratic bottleneck is a major challenge for long-document processing.

# n=sequence length, d=dimension
# attention matrix is n x n
scores = Q @ K.T  # shape: (n, n) — quadratic!

Bahdanau attention (2015) was introduced for seq2seq machine translation. For decoder step t with state s_t, and encoder hidden states h_i, it computes: score(s_t, h_i) = v^T tanh(W_1 s_t + W_2 h_i). Alignment weights α_{ti} = softmax(scores); context c_t = Σ α_{ti} h_i. This additive scoring uses a small feed-forward network to compute relevance. It allows the decoder to focus on different encoder positions at each step, solving the bottleneck of fixed-length encoder vectors. Bahdanau attention was the first major practical attention mechanism in NLP.

Luong attention (2015) computes similarity between decoder state h_t and encoder states h_s using three variants: (1) dot: h_t^T h_s; (2) general: h_t^T W h_s; (3) concat (same as Bahdanau). The dot and general forms are simpler and faster than Bahdanau's additive scoring. Context vector c_t = Σ α_{ts} h_s, then combined with decoder state for prediction. Luong attention is usually applied after the RNN computation (input-feeding), while Bahdanau applies before. Both served as motivation for Transformer's scaled dot-product attention.

Self-attention is permutation-invariant — it treats the sequence as a set, not an ordered sequence. Shuffling input tokens gives identical attention scores. Positional encoding injects position information as an additive signal: PE(pos, 2i) = sin(pos/10000^{2i/d}), PE(pos, 2i+1) = cos(pos/10000^{2i/d}). Sinusoidal encodings allow the model to extrapolate to longer sequences. Learned positional encodings (BERT, GPT) simply embed each position. Without positional encoding, the Transformer cannot distinguish 'dog bites man' from 'man bites dog'.

import numpy as np
def sinusoidal_pe(seq_len, d):
    pos = np.arange(seq_len)[:,None]
    i = np.arange(d)[None,:]
    angle = pos / 10000**(2*(i//2)/d)
    pe = np.where(i%2==0, np.sin(angle), np.cos(angle))
    return pe

Soft attention (standard) computes a weighted sum over all values: output = Σ α_i v_i, where α_i are continuous weights from softmax. Differentiable end-to-end, trainable via backprop. Hard attention selects exactly one position (stochastic): output = v_i where i ~ Categorical(α). Hard attention is not differentiable; training requires REINFORCE or Gumbel-Softmax approximations. Soft attention is used in Transformers. Hard attention was used in 'Show, Attend and Tell' (image captioning) to force the model to focus on exactly one image region, providing interpretable glimpses.

Global attention attends to every position in the full sequence — O(n²) cost. Local attention restricts each query to attend only within a local window of size 2D+1 around it — O(n·D) cost. Local attention captures local patterns efficiently but misses long-range dependencies. Hybrid approaches combine both: Longformer uses local sliding window + global tokens for special positions (CLS). BigBird uses local + global + random patterns. Local attention is critical for processing long documents (legal, genomics) where full attention is computationally infeasible.

Attention masks prevent positions from attending to certain other positions. Two main types: (1) Padding mask — prevents attending to padding tokens (variable-length sequences in a batch are padded; masked positions set to −∞ before softmax → weight ≈ 0). (2) Causal (autoregressive) mask — upper triangular mask that prevents position t from attending to future positions t+1,...,n. This ensures autoregressive generation (GPT) where each token only sees past context. Masks are added to scores before softmax: score_{ij} += −∞ if masked(i,j).

import torch
# Causal mask for autoregressive attention
mask = torch.triu(torch.ones(n, n), diagonal=1).bool()
scores.masked_fill_(mask, float('-inf'))

Flash Attention (Dao et al. 2022) is an IO-aware exact attention algorithm that avoids materializing the full n×n attention matrix in HBM (GPU main memory). It tiles Q, K, V and computes attention in blocks within fast SRAM, using the online softmax trick to accumulate results. This reduces memory from O(n²) to O(n) and achieves 2-4× speedup. Flash Attention is exact (not approximate) and enables training Transformers on much longer sequences. It's now the default attention implementation in most modern Transformer libraries (PyTorch, HuggingFace).

RNNs process sequences step-by-step, limiting parallelism and suffering from vanishing gradients over long distances. Self-attention computes all pairwise relationships in parallel in O(1) sequential operations: any two positions interact directly regardless of distance. This eliminates the 'information highway' problem and allows full parallelism over sequence positions during training (though autoregressive decoding is still sequential). Path length between any two positions is O(1) in attention vs O(n) in RNN. The Transformer's parallelism makes it scale far better on modern hardware.

Linear attention approximates softmax attention by factorizing the kernel: Attention(Q,K,V) ≈ φ(Q)(φ(K)^T V), where φ is a feature map. By associativity of matrix multiplication, compute K^T V first (d×d matrix, O(n·d²)), then Q(K^T V) — reducing complexity from O(n²·d) to O(n·d²). Popular kernels: ELU+1 (Katharopoulos et al.), random Fourier features (Performer), Favor+. Linear attention approximates full attention but sacrifices some expressiveness. It enables processing very long sequences (n=100k+) that quadratic attention cannot handle.

Standard positional encoding adds absolute position embeddings to token embeddings, so the model knows position 5. Relative positional encoding instead injects the relative distance between positions into attention scores: score(i,j) += embedding(i−j). This allows the model to generalize to sequence lengths not seen during training (length extrapolation), and captures local vs. distant relationships more naturally. Examples: Transformer-XL, T5 (bucketed relative bias), RoPE (Rotary Position Embedding, used in LLaMA/GPT-Neo). RoPE multiplies Q and K by rotation matrices dependent on position.

The attention weight α_{ij} = softmax_j(score(q_i, k_j)) represents how much query position i attends to key position j. α_{ij} ∈ [0,1] and Σ_j α_{ij} = 1. They can be visualized as a heat map over the input sequence — showing where the model focuses for each output position. In NLP, attention weights sometimes align with linguistic intuitions (e.g., verb attending to its subject). However, research shows attention weights don't always reflect importance reliably and can be manipulated, so interpreting them causally requires caution.

Q, K, V projections are learned linear transformations applied before attention: Q = X W^Q, K = X W^K, V = X W^V (for self-attention; X is the input). These projections map the same input into three different roles: queries encode 'what I'm looking for', keys encode 'what I offer to match', values encode 'what I send if matched'. Learning separate projections allows each head to attend using different representation subspaces. The output projection W^O combines head outputs. These 4 weight matrices (W^Q, W^K, W^V, W^O) are the trainable parameters of one multi-head attention block.

Vision Transformer (ViT) applies Transformer self-attention to images by splitting the image into non-overlapping 16×16 patches, linearly embedding each patch as a token, prepending a [CLS] token, adding position embeddings, and processing with standard Transformer encoder blocks. Attention between patches captures global image relationships that convolutions miss (which have fixed local receptive fields). ViT achieves state-of-the-art on image classification when pre-trained on large datasets. DINO/DINOv2 show ViT attention maps align with semantically meaningful image regions.

Sparse attention restricts each token to attend to a fixed subset of positions (not all n), reducing complexity from O(n²) to O(n√n) or O(n log n). Patterns include: local window (fixed size neighborhood), strided (every k-th token), global (special tokens attend everywhere), random (random subset). Longformer combines local + global; BigBird uses local + global + random. Sparse attention is crucial for long documents (books, code, genomic sequences) but may miss dependencies outside the predefined patterns. The sparse pattern should be designed to cover expected dependency structures.

In encoder-decoder Transformers (original Transformer, T5, BART), the decoder has cross-attention layers between its masked self-attention and feed-forward layers. Cross-attention uses decoder states as Q, encoder output as K and V: it lets each decoder position query the entire encoder output. This is how the decoder 'reads' the encoded input while generating the output. For translation, cross-attention aligns target words with relevant source words. The cross-attention weights are often the most linguistically interpretable weights in seq2seq models.

Each Transformer block has two sub-layers: multi-head attention + a position-wise feed-forward network (FFN). The FFN is: FFN(x) = max(0, x W_1 + b_1) W_2 + b_2 — a 2-layer MLP with ReLU/GELU, applied identically to each position. d_ff is usually 4× the model dimension (e.g., d_model=512, d_ff=2048). FFN adds non-linearity and capacity beyond what attention provides. It's sometimes called the 'memory' of the Transformer — research suggests it stores factual knowledge. FFN parameters account for ~2/3 of Transformer parameters in large models.

class FFN(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.l1 = nn.Linear(d_model, d_ff)
        self.l2 = nn.Linear(d_ff, d_model)
    def forward(self, x):
        return self.l2(F.gelu(self.l1(x)))

Think of attention as a soft database lookup. The Query encodes 'what information I need now'. Keys encode 'what information each position contains'. Values encode the actual content. The dot product Q·K measures relevance between need and content — a high dot product means position j is relevant to query i. Softmax normalizes these into a probability distribution over positions (how much to weight each). The weighted sum of Values retrieves the relevant information. This is analogous to a search engine: query matches against an index (keys) to retrieve relevant documents (values).

GPT uses a causally masked (autoregressive) self-attention: token at position t can only attend to positions 1...t (not t+1...n). This is implemented by setting attention scores for future positions to −∞ before softmax, making those weights ≈ 0. This constraint ensures that during training (where full sequences are given), the model cannot 'cheat' by looking at future tokens. At inference, generation proceeds left-to-right, so no mask is needed then. Masked self-attention is the key architectural feature distinguishing decoder-only (GPT) from encoder-only (BERT) Transformers.

Attention rollout (Abnar & Zuidema 2020) aggregates attention across Transformer layers by recursively multiplying attention matrices: Rollout_l = A_l · Rollout_{l-1}, with Rollout_0 = I. Since attention is applied to residual-connected outputs, both attending and non-attending paths carry information — rollout adds 0.5·I to each layer's attention before multiplying. The result approximates how much information from each input token flows to each output position through all layers. Attention rollout is more faithful than per-layer attention maps for explaining Transformer predictions.

Different attention heads specialize in different types of relationships. In BERT, analyzed heads show: some heads attend to previous/next tokens (local positional relations), some attend to syntactic dependencies (subject-verb), some attend to long-range coreference, and some spread uniformly (low-specificity). Each head operates in a d_k = d_model/h subspace, learning different projections W_i^Q, W_i^K, W_i^V. Not all heads are equally important — pruning heads that are redundant can compress models with minimal quality loss, suggesting significant redundancy in large Transformer heads.

The temperature τ controls attention sharpness: α = softmax(scores/τ). Low τ makes attention spiky (concentrated on one or few positions — approaches hard max). High τ spreads attention uniformly (flat distribution — approaches uniform random). The scale factor √d_k in scaled dot-product attention acts as an implicit temperature: d_k normalization keeps temperature ~1. At inference, sampling temperature in language models similarly controls output diversity. Some models expose this as a hyperparameter for controlling how deterministic vs. diverse generated text is.

def attention_with_temperature(Q, K, V, tau=1.0):
    d_k = Q.size(-1)
    scores = Q @ K.T / (tau * math.sqrt(d_k))
    return F.softmax(scores, -1) @ V

In encoder-decoder RNNs without attention, the entire input sequence is compressed into a single fixed-size context vector (the last encoder hidden state). This becomes a bottleneck for long sequences — all information must fit in one vector. Attention solves this by maintaining all encoder hidden states and allowing the decoder to dynamically select which ones to focus on at each step. This was demonstrated dramatically in machine translation: Bahdanau et al. showed attention-based models outperform non-attention models especially on long sentences (>20 words).

BERT uses bidirectional self-attention — each token attends to all others (left and right) simultaneously. This makes BERT powerful for understanding tasks (classification, NER, QA). GPT uses unidirectional (causal/left-to-right) masked attention for autoregressive text generation. BERT is pre-trained with Masked Language Modeling (predict masked tokens using bidirectional context); GPT with standard language modeling (predict next token). BERT cannot generate text left-to-right coherently; GPT cannot see future context. T5 and BART use full bidirectional encoder + causal decoder.

Longformer uses a sliding window of size w: each token attends only to w/2 neighbors on each side. This gives O(n·w) attention complexity instead of O(n²), enabling processing documents up to n=4096 tokens. For tasks requiring global context (e.g., classification), special tokens (like [CLS]) use global attention — they attend to and are attended by all positions. Longformer combines local sliding window + global tokens, achieving near-full-attention quality on long-document tasks at a fraction of the cost. It extends BERT to books, patents, and scientific articles.

Layer normalization (LN) normalizes activations across the feature dimension (not batch): LN(x) = (x − μ) / σ · γ + β, where μ,σ are per-sample statistics. In Transformer blocks, LN stabilizes the residual stream: output = LN(x + Attention(x)) or output = x + Attention(LN(x)) (pre-LN). Pre-LN (used in GPT-2+) is more training-stable without warm-up. LN is preferred over batch norm in Transformers because it doesn't depend on batch size and works naturally for variable-length sequences.

import torch.nn as nn
ln = nn.LayerNorm(d_model)
# Pre-LN
out = x + attention(ln(x))  # more stable

Xu et al. (2015) applied attention to image captioning. A CNN encodes the image into a 14×14 grid of features (one per region). At each decoder step, attention weights over the 196 regions determine which part of the image to focus on when generating the next word. Soft attention is differentiable; hard attention stochastically selects one region. Attention weight visualization shows the model looking at relevant image regions: attending to a dog when generating 'dog', to a ball when generating 'ball'. This was one of the first interpretable applications of attention in computer vision.

In autoregressive decoding (GPT), at each new token the model needs K,V for all previous tokens. Recomputing them from scratch is wasteful. The KV cache stores K and V matrices for all past positions: when generating token t, only compute Q_t (new query), then attend using cached K_{1:t-1} and V_{1:t-1}. This reduces per-step computation from O(t·d) to O(d). Without KV cache, generating n tokens is O(n²·d); with cache it's O(n·d). KV cache is the key optimization making LLM inference practical, but requires O(n·d) memory.

Standard MHA has h heads, each with its own K and V — large KV cache. Multi-Query Attention (MQA) shares a single K, V across all h query heads, shrinking KV cache h× but degrading quality. Grouped Query Attention (GQA) is a compromise: g groups share K, V (h/g queries per K,V pair). GQA is used in Llama 2, Mistral, Gemma. GQA achieves MQA-like inference speed with near-MHA quality. As model context windows grow (128k tokens), reducing KV cache size becomes critical for memory-efficient long-context inference.

RoPE (Su et al. 2021) encodes position by rotating query and key vectors in 2D subspaces: Q_m = R_m Q, K_n = R_n K, where R_m is a block-diagonal rotation matrix with angle mθ. The dot product Q_m · K_n then depends on (m−n) — the relative position — not absolute positions. This gives RoPE the benefits of relative encoding while being as simple to implement as absolute encoding. RoPE generalizes well to longer sequences than seen during training. It's used in LLaMA, GPT-NeoX, PaLM, and most modern open LLMs.

Deformable attention (DETR variant) attends to a sparse set of learned sampling points around a reference point, rather than all positions. For each query, it learns offset positions (Δp_k) and amplitudes (A_k), computing: output = Σ_k A_k · V(p + Δp_k). This gives O(n·K) complexity (K = number of sampling points, typically 4–8) vs O(n²). Deformable attention captures multi-scale, spatially adaptive dependencies in vision tasks. It enables efficient attention for high-resolution feature maps in object detection and segmentation (Deformable DETR, DINO).

Attention weights can be visualized as heat maps showing which input tokens most influence each output. In machine translation, cross-attention often shows diagonal alignment (source and target words align). In BERT, certain heads detect syntactic dependencies. Gradient-weighted attention (multiply α by gradient) gives more faithful importance estimates. However, Jain & Wallace (2019) showed attention weights are not always faithful explanations — they can be swapped with uniform distributions while maintaining predictions. Combining attention with gradient-based methods (Attention×Gradient) gives more reliable attribution.

In text-to-image diffusion models (DALL-E 2, Stable Diffusion), cross-attention layers in the U-Net inject text conditioning. The U-Net's spatial features are Q; text token embeddings from CLIP/T5 are K and V. Cross-attention: output = softmax(Q K^T / √d) V fuses text semantics with image features at each spatial location. Prompt-to-Prompt (Hertz et al. 2022) showed that manipulating cross-attention maps (swapping attention between prompts) enables semantic image editing without retraining. Cross-attention is the key mechanism linking text and image in modern generative models.

Attention entropy H(α) = −Σ_j α_j log α_j measures how spread out attention is. Low entropy → concentrated (attending to 1–2 positions, sharp). High entropy → spread (attending uniformly over many positions, diffuse). High-entropy heads are often less informative; low-entropy heads capture sharp dependencies. In BERT, entropy varies dramatically across layers: early layers have high entropy (global patterns), middle layers have low entropy (specific dependencies), later layers again higher. Monitoring entropy helps identify redundant or collapsed heads during training and pruning.

import torch
def attention_entropy(attn_weights):
    # attn_weights: [head, seq, seq]
    return -(attn_weights * attn_weights.clamp(1e-8).log()).sum(-1).mean()

ALBERT reduces parameters by sharing weights across Transformer layers. All 12 (or more) layers use the same attention and FFN parameters (cross-layer parameter sharing). Additionally, ALBERT uses factorized embedding parameterization: E = d_embed × d_model projected, where d_embed can be small (e.g., 128). This shrinks parameters 18× vs BERT while maintaining most performance on GLUE. The intuition: deep layers learn progressively refined representations of the same semantic content, so sharing is reasonable. ALBERT achieves BERT-level performance with far fewer parameters, enabling lighter deployment.

A causal language model predicts the next token given all previous tokens: P(x_t | x_{<t}). GPT-style models use masked self-attention to enforce this: the attention mask ensures token t only attends to tokens 1,...,t. This allows the model to be trained on all positions in parallel (teacher forcing) — the same forward pass computes predictions for all positions simultaneously using the mask. At inference, generation proceeds auto-regressively one token at a time. The causal mask is what makes a standard Transformer suitable for language modeling without any architectural changes.

Residual connections (skip connections) add the input directly to the output of each sub-layer: output = x + SubLayer(x). This ensures that even if attention weights collapse, the original signal passes through. Residuals also create gradient highways enabling training of deep networks. In Transformers, each block has two residuals: one around multi-head attention, one around FFN. LayerNorm is applied either after (Post-LN: original Transformer) or before (Pre-LN: GPT-2+) the residual. Pre-LN has been shown to train more stably without learning rate warm-up.

# Pre-LN Transformer block
def block(x, attention, ffn, ln1, ln2):
    x = x + attention(ln1(x))
    x = x + ffn(ln2(x))
    return x

BERT prepends a special [CLS] token to every input sequence. After passing through all Transformer layers, the [CLS] token's final hidden state is used as an aggregate sequence representation for classification tasks. Because self-attention is global, [CLS] can attend to all tokens and accumulate sentence-level information. During fine-tuning, a linear classifier is added on top of [CLS]. However, research shows that pooling over all token embeddings (mean/max) sometimes gives better sentence representations than [CLS] alone, especially for semantic similarity tasks (e.g., Sentence-BERT).

Perceiver IO handles arbitrary-size inputs by first cross-attending a fixed-size latent array (e.g., 512 latent vectors) to the input (which can be millions of elements). Queries come from latents; K,V from the input. This reduces the dominant computation to O(M×N) where M is latent size (small, fixed) and N is input size. Subsequent self-attention operates only on the M latents — O(M²). Perceiver IO generalizes to images, audio, video, point clouds, and multimodal inputs without architectural changes, making it a universal architecture for any modality.

Induction heads (discovered via mechanistic interpretability) are pairs of attention heads that together implement a pattern-completion algorithm: if the sequence contains '...A B ... A', predict B. One head copies the previous token (A→B pair), another head attends back to match A and retrieves B. Induction heads appear in 2-layer Transformers and are believed to be key to in-context learning — the model's ability to follow new patterns from examples in the prompt. They illustrate how attention heads compose to implement algorithms, a key insight in mechanistic interpretability.

Early layers of Transformers attend to local, syntactic patterns — neighboring tokens, punctuation, special tokens (BOS, CLS). Middle layers capture longer-range syntactic dependencies (subject-verb agreement, coreference). Later layers attend to task-specific, semantic patterns relevant to the prediction target. This layered progression mirrors hierarchical linguistic analysis. BERT probing studies confirm: simple morphological features are extractable from early layers; complex syntactic/semantic features from middle and late layers. This layer-specialization insight informs fine-tuning strategies (often only fine-tuning later layers).

Context length (also context window) is the maximum number of tokens a Transformer can process in one forward pass. Limits: (1) Memory — self-attention requires O(n²) memory for the attention matrix. (2) Positional encoding — absolute positions not seen during training may degrade quality. (3) Computational cost — O(n²) FLOPs. GPT-3: 4k tokens; GPT-4: 128k tokens; Gemini 1.5: 1M tokens. Techniques to extend: sparse attention, FlashAttention (memory), RoPE+YaRN (position extrapolation), LongLoRA, and retrieval-augmented generation (bypass context limit entirely).

Sparse Mixture of Experts replaces the FFN in each Transformer block with multiple expert FFNs, routing each token to a small subset (typically 2 out of N experts) via a learned router. This scales model capacity (N experts × FFN size) without proportionally increasing compute per token. Attention remains dense. GShard, Switch Transformer, and Mixtral use this approach. Mixtral 8×7B has 8 experts per layer but activates only 2 per token — effectively a 12.9B active-parameter model with 46.7B total. MoE enables very large models with practical inference cost.

In autoregressive LLMs with long contexts, attention weights concentrate on the first few tokens (especially the initial token) even when they're semantically irrelevant — these are called 'attention sinks'. The model uses these positions to dump attention weight when no position is strongly relevant (like a /dev/null). StreamingLLM exploits this: by keeping attention sinks + a recent window in the KV cache, LLMs can process unlimited-length sequences with constant memory. This reveals that the first token acts as a special anchor, and removing it collapses the model's ability to handle long contexts.

Additive (Bahdanau) attention computes scores using a feedforward network: score(q,k) = v^T tanh(W_q q + W_k k), with parameters v, W_q, W_k. Scaled dot-product attention computes: score(q,k) = q^T k / √d_k — just a dot product then scale. Both produce a scalar relevance score. Scaled dot-product is faster (matrix multiplication, GPU-optimized) while additive can capture more complex interactions. In practice, scaled dot-product works better at scale and is universally used in Transformers. Additive attention was important historically (LSTM seq2seq era) but is largely replaced by the simpler, faster dot-product variant.