Transformers

The Transformer (Vaswani et al. 2017, 'Attention Is All You Need') replaced recurrence and convolution with self-attention for sequence modeling. Key innovation: all positions attend to all others in parallel — no sequential dependency. This enables full parallelism during training (vs. RNN's sequential O(n) steps), direct O(1) path between any two positions (vs. RNN's O(n)), and better modeling of long-range dependencies. The Transformer became the foundation of BERT, GPT, T5, and all modern large language models, and scaled far better than RNNs with more data and compute.

A Transformer encoder block has two sub-layers: (1) Multi-Head Self-Attention: each token attends to all others to gather context. (2) Position-wise Feed-Forward Network: two linear layers with GELU/ReLU in between, applied identically to each position. Each sub-layer is wrapped with a residual connection and layer normalization: output = LayerNorm(x + SubLayer(x)). The encoder stack has N such blocks (typically N=6 to 12). Encoder output is a rich contextual representation of the entire input sequence, used by the decoder via cross-attention or directly for classification (BERT).

class EncoderBlock(nn.Module):
    def forward(self, x, mask):
        x = ln1(x + attn(x, x, x, mask))
        x = ln2(x + ffn(x))
        return x

The encoder-decoder Transformer processes source sequence with the encoder (bidirectional self-attention), then generates the target sequence with the decoder. The decoder has three sub-layers: (1) Masked self-attention (causal — can only see past generated tokens). (2) Cross-attention (decoder queries, encoder output as keys/values — reads source context). (3) Feed-forward network. At each decoder step, the model generates the next token using previous outputs + full encoder context. This architecture is used for MT (English→French), summarization (long→short), and any seq2seq task. T5, BART, mT5 use encoder-decoder.

Tokenization converts raw text into discrete tokens (integers) that the Transformer can process. Word tokenization: splits on spaces (large vocab, many OOV words). Character tokenization: tiny vocab but very long sequences. Subword tokenization: BPE (Byte-Pair Encoding, used in GPT) and WordPiece (BERT) balance vocabulary size and sequence length by merging frequent character pairs. BPE starts with characters, iteratively merges most frequent pairs: 'lo', 'w' → 'low'. Sentencepiece is language-agnostic. Good tokenization: unknown words are split into known subwords ('tokenization' → 'token', '##ization'). Vocabulary size is a key hyperparameter (30k–100k).

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained('bert-base-uncased')
tokens = tok('Hello world', return_tensors='pt')
print(tokens['input_ids'])

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al. 2018) is a Transformer encoder pre-trained on two tasks: (1) Masked Language Modeling (MLM): randomly mask 15% of tokens, predict them using bidirectional context — forces deep contextual understanding. (2) Next Sentence Prediction (NSP): predict if sentence B follows sentence A — later shown less important. BERT uses bidirectional self-attention, enabling richer representations than left-to-right GPT. Fine-tuning adds a task-specific head on [CLS] or token embeddings. BERT revolutionized NLP, achieving SOTA on GLUE, SQuAD, and NER.

GPT (Generative Pre-trained Transformer, OpenAI) uses a decoder-only Transformer with causal (left-to-right) masked self-attention, pre-trained with standard language modeling: predict next token given all previous. GPT generates text autoregressively. BERT uses encoder-only with bidirectional attention, pre-trained with MLM, better for understanding/classification. GPT: generation, zero/few-shot prompting, instruction following. BERT: classification, NER, QA (reading comprehension). GPT-2, GPT-3, GPT-4 scaled the architecture to 1.5B, 175B, and ~1T+ parameters respectively, discovering emergent capabilities at scale.

Pre-trained Transformers (BERT, GPT) learn general language representations from massive text. Fine-tuning adapts them to a specific task with a small labeled dataset and a task head. Process: (1) Load pre-trained weights. (2) Add a task-specific head (e.g., linear classifier on [CLS] for classification, token-level classifier for NER). (3) Train all weights (or just the head) on task data with small learning rate (2e-5 to 5e-5) for 3–10 epochs. Fine-tuning leverages pre-trained knowledge, requiring far less labeled data than training from scratch. The 'pre-train once, fine-tune many' paradigm transformed NLP.

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
# optimizer with small LR ~2e-5

The inner dimension of the Transformer FFN is typically 4× the model dimension: FFN(x) = max(0, x W_1 + b_1) W_2 + b_2, where W_1 ∈ R^{d_model × d_ff}, d_ff = 4 × d_model. For BERT-base: d_model=768, d_ff=3072. The 4× ratio gives the model ample capacity for feature transformation while keeping computation manageable. FFN parameters dominate large models: in a 12-layer BERT, FFN accounts for ~2/3 of all parameters. Mixture of Experts (MoE) replaces this FFN with multiple experts to scale capacity without scaling FLOPs proportionally.

d_model, d_ff = 768, 3072  # BERT-base
ffn = nn.Sequential(
    nn.Linear(d_model, d_ff),
    nn.GELU(),
    nn.Linear(d_ff, d_model)
)

BERT prepends [CLS] (classification) and inserts [SEP] (separator) tokens. [CLS] aggregates sequence-level information via self-attention — its final embedding is used for classification. For two-sentence tasks (QA, NLI), [SEP] separates them: [CLS] sentence_A [SEP] sentence_B [SEP]. BERT also uses segment embeddings (0 for sentence A, 1 for B) to distinguish them. At fine-tuning, a linear layer maps [CLS] embedding to class logits. Research shows [CLS] is not always the best sentence representation — mean pooling over all tokens often works better for semantic similarity.

BERT-base: 12 layers, 768 hidden dim, 12 attention heads, 110M parameters. BERT-large: 24 layers, 1024 hidden dim, 16 attention heads, 340M parameters. BERT-large achieves better performance on most benchmarks but requires significantly more memory (2–3× GPU memory) and compute. For many tasks, BERT-base is competitive and preferable due to speed/memory trade-offs. The scaling trend showed more layers + heads + dimensions consistently improve performance — motivating GPT-2 (1.5B), GPT-3 (175B), and beyond. BERT's authors showed all three dimensions (depth, width, heads) contribute to quality.

T5 (Raffel et al. 2020) frames every NLP task as text-to-text: input and output are both text strings. Translation: 'translate English to French: Hello' → 'Bonjour'. Classification: 'sentiment: I love this movie' → 'positive'. QA: 'question: What is...? context: ...' → 'answer'. T5 uses an encoder-decoder Transformer with relative position encodings. Pre-trained on C4 (Colossal Clean Crawled Corpus). T5 unified all tasks under one framework, enabling multi-task training and zero-shot generalization. The text-to-text paradigm influenced later models like FLAN-T5 and enables intuitive prompting without task-specific heads.

from transformers import T5ForConditionalGeneration, T5Tokenizer
tok = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small')
inputs = tok('translate English to French: Hello', return_tensors='pt')
out = model.generate(**inputs)

Scaling laws (Kaplan et al. 2020, Chinchilla 2022) describe power-law relationships between model performance and scale: L(N) ≈ N^{-0.076}, L(D) ≈ D^{-0.095}, where N=parameters, D=data tokens. Compute-optimal training (Chinchilla): for a given compute budget C, allocate equal scaling to N and D (N ∝ C^0.5, D ∝ C^0.5). GPT-3 was undertrained; Chinchilla (70B params, 1.4T tokens) matched Gopher (280B params) with same compute. This implies larger models need proportionally more data. Scaling laws guide practical decisions about model size vs training tokens given a compute budget.

GPT-3 demonstrated remarkable in-context learning: without weight updates, it performs tasks from examples in the prompt. Zero-shot: describe task in text, no examples — 'Translate to French: Hello'. Few-shot: provide 2–10 input-output examples before the query — the model infers the pattern. One-shot: exactly 1 example. This emerged at scale (~100B+ parameters). The model treats examples as context, not training signal — no gradient update. Few-shot performance often rivals task-specific fine-tuned models on BERT scale. Prompting is now a discipline: chain-of-thought, role prompting, and instruction following extend few-shot capabilities.

Autoregressive (decoder-only, GPT-style): causal mask — each token sees only past. Pre-trained with next-token prediction. Best for generation. Masked (encoder-only, BERT-style): bidirectional attention — each token sees all others. Pre-trained with MLM. Best for understanding/classification. Encoder-decoder (T5, BART): encoder is bidirectional, decoder is causal with cross-attention. Best for seq2seq. Recent trend: decoder-only (LLaMA, Claude, GPT-4) dominates because sufficiently large autoregressive models can also understand, and the generation capability is more general. BERT-style is still preferred for classification tasks in resource-constrained settings.

Instruction fine-tuning (IFT) trains a base language model on diverse (instruction, response) pairs to follow human instructions across many tasks. Examples: 'Summarize this text: ...', 'Write a Python function that...', 'Explain quantum entanglement simply'. IFT transforms a 'next token predictor' into an 'instruction follower'. Key datasets: FLAN (1000+ tasks via templates), InstructGPT (human-written demonstrations), Alpaca (GPT-4 generated instructions). IFT dramatically improves zero-shot task performance and makes models like ChatGPT, Claude, and Gemini safe and usable. Combined with RLHF, IFT is the main technique for LLM alignment.

LoRA fine-tunes large Transformers by decomposing weight updates into low-rank matrices: ΔW = BA, where B ∈ R^{d×r}, A ∈ R^{r×k}, r << min(d,k). Original weights W are frozen; only A,B are trained. For r=4 vs d=4096: 4096×4096=16M parameters → 2×4096×4=32k parameters (500× fewer). Merge at inference: W' = W + ΔW (no extra latency). LoRA typically achieves near full fine-tuning quality at 1% of parameters. QLoRA adds 4-bit quantization of base model, enabling fine-tuning of 65B models on a single A100. LoRA is now the standard efficient fine-tuning method.

from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=32, target_modules=['q_proj','v_proj'])
model = get_peft_model(base_model, config)
model.print_trainable_parameters()  # ~1% of total

GELU (Gaussian Error Linear Unit) is defined as: GELU(x) = x · Φ(x), where Φ is the Gaussian CDF — approximately x · σ(1.702x). It smoothly gates inputs by their probability of being positive under a Gaussian, unlike ReLU which hard-gates at 0. GELU is differentiable everywhere (no kink at 0), enabling smoother optimization. Empirically, GELU outperforms ReLU in Transformers. Used in BERT, GPT-2+, ViT. Variants: Swish (x·σ(x)), SiLU (same as Swish), SwiGLU (Swish-gated linear unit, used in LLaMA). Modern LLMs almost universally use GELU or SwiGLU.

import torch
import torch.nn.functional as F
gelu_approx = lambda x: x * torch.sigmoid(1.702 * x)

RoBERTa (Liu et al. 2019) found BERT was significantly undertrained and made key improvements: (1) Remove NSP (Next Sentence Prediction) — hurts downstream performance. (2) Train on 10× more data (160GB vs 16GB). (3) Train with larger batches (8192 vs 256). (4) Use dynamic masking (different masks each epoch). (5) Train longer (100k→500k steps). With these changes and same architecture as BERT-large, RoBERTa significantly outperformed BERT on GLUE, SQuAD. Lesson: data quality, training compute, and careful choices of training objective matter as much as architecture.

Input embeddings convert token IDs to dense vectors: E ∈ R^{V×d_model} maps each of V vocabulary tokens to a d-dimensional vector. BERT has three embedding components summed: (1) Token embeddings (learned per token). (2) Positional embeddings (learned absolute positions). (3) Segment embeddings (sentence A/B indicator). Initialized randomly, these are learned jointly with all other parameters. Output embeddings (logits) typically share weights with input token embeddings (weight tying), reducing parameters and improving generalization. Embeddings are the first and most fundamental component of any Transformer.

embed = nn.Embedding(vocab_size, d_model)  # token
pos_embed = nn.Embedding(max_len, d_model)  # positional
x = embed(tokens) + pos_embed(positions)

BLEU (BiLingual Evaluation Understudy) measures translation quality by n-gram precision between hypothesis and references: BLEU = BP · exp(Σ w_n log p_n), where p_n is precision of n-grams (n=1..4), w_n=1/4, and BP is brevity penalty for short translations. Range: 0–100 (higher = better). Limitations: doesn't capture meaning (syntactic re-orderings hurt even if semantically correct), poor correlation with human judgments for high-quality models. Alternatives: ROUGE (recall-based, used for summarization), METEOR, BERTScore (embedding similarity). BLEU is still widely reported for historical comparability in MT papers.

from sacrebleu.metrics import BLEU
bleu = BLEU()
result = bleu.corpus_score(hypotheses, [references])
print(result.score)

Chain-of-thought prompting (Wei et al. 2022) adds intermediate reasoning steps in few-shot examples: instead of 'Q: What is 15+27? A: 42', it shows 'Q: What is 15+27? A: 15+27 = (10+5)+(20+7) = 30+12 = 42'. The model learns to generate step-by-step reasoning before the final answer, dramatically improving arithmetic, commonsense, and symbolic reasoning. Zero-shot CoT: append 'Let's think step by step.' to any prompt — surprisingly effective. CoT emerges at ~50–100B parameters. Tree-of-thought extends CoT with branching exploration. CoT is central to modern LLM reasoning capabilities.

Weight tying shares the input embedding matrix E ∈ R^{V×d} with the output projection matrix for computing logits: logits = H E^T, where H is the final hidden state. Since input embeddings and output logits both map between token ID space and model space, sharing makes sense semantically. Benefits: reduces parameters by V×d (e.g., 50k×768 = 38M for BERT), improves generalization, and aligns embedding space between input and output. Used in GPT-2, T5, and most modern LLMs. The gradients from the language modeling loss flow to embeddings from both the input encoding and output projection paths.

# Weight tying
model.lm_head.weight = model.embed_tokens.weight

Not all attention heads are equally important — some are redundant. Michel et al. (2019) showed that 20–40% of BERT heads can be pruned without significant quality loss. Pruning: identify heads with low importance scores (e.g., gradient × activation magnitude), mask them out (multiply attention output by 0). Structured pruning removes entire heads (saves FLOPs); unstructured removes individual weights (harder to accelerate). Head pruning reduces inference cost while maintaining performance for specific tasks. In practice, fewer important heads remain for each task — different tasks use different subsets of heads.

DistilBERT (Sanh et al. 2019) distills BERT-base (110M params, 12 layers) into a smaller student (66M params, 6 layers). Distillation loss: L = α L_CE(student logits, labels) + β L_KL(student soft probs, teacher soft probs) + γ L_cosine(student, teacher hidden states). Soft targets from teacher carry more information than one-hot labels (temperature scaling T=2). DistilBERT retains 97% of BERT's performance at 40% fewer parameters and 60% faster inference. TinyBERT and MobileBERT further compress. Distillation is the standard technique for deploying large Transformers on mobile/edge devices.

Post-LN (original Transformer): LN applied after residual — output = LN(x + SubLayer(x)). Needs careful LR warm-up for stable training; gradients can be unstable in deep networks. Pre-LN (GPT-2, LLaMA): LN applied before — output = x + SubLayer(LN(x)). More training-stable, doesn't need warm-up, allows training deeper models reliably. The residual path in Pre-LN is always clean (no normalization), which preserves gradient magnitude. Most modern LLMs (GPT-2, GPT-3, LLaMA) use Pre-LN. Some use RMSNorm (Root Mean Square Norm) instead of LayerNorm for efficiency: RMSNorm(x) = x / RMS(x) · γ.

# Pre-LN
x = x + attention(layer_norm(x))
# Post-LN
x = layer_norm(x + attention(x))

Context window is the maximum sequence length a Transformer can process. Limits: O(n²) attention memory, positional encoding generalization. GPT-3: 4096 tokens; GPT-4: 128k; Gemini 1.5: 1M. Extension techniques: (1) Sparse attention (Longformer, BigBird). (2) FlashAttention — exact but memory-efficient. (3) RoPE scaling — extend positional embedding range via YaRN, NTK-aware interpolation. (4) LongLoRA — sparse attention during fine-tuning, full at inference. (5) Retrieval-Augmented Generation — don't put everything in context, retrieve relevant chunks. Extended context enables processing full books, codebases, and multi-day conversations.

BART (Lewis et al. 2020) is an encoder-decoder Transformer pre-trained with denoising: corrupt input text with various noise (token masking, deletion, permutation, text infilling) and train decoder to reconstruct the original. BART combines BERT's bidirectional encoder with GPT's autoregressive decoder. Excellent for generation tasks requiring understanding: summarization, translation, question generation. BART fine-tuned for summarization achieves SOTA ROUGE scores. Compared to BERT (encoder-only, no generation) and GPT (decoder-only, no bidirectional context), BART excels at both comprehension (encoder) and generation (decoder).

Code Transformers fine-tune or pre-train language models on code. Codex (OpenAI) fine-tuned GPT-3 on GitHub code, powering GitHub Copilot — achieves 28.8% pass@1 on HumanEval (function synthesis from docstring). CodeT5 (encoder-decoder, T5-based) supports multi-task code understanding and generation. AlphaCode (DeepMind) generates competitive programming solutions. Code-specific pre-training tokens include programming language syntax, comments, docstrings. Code LLMs must understand both natural language (specs, comments) and formal language (syntax, semantics). The structural regularity of code (syntax trees, indentation) differs from natural language and benefits from special tokenization.

CNN inductive biases: translation equivariance (same filter everywhere), local connectivity. RNN inductive biases: sequential processing, Markov-like state. Transformer inductive biases: permutation equivariance (with positional encoding added), global attention (any position to any). Transformers have less built-in bias — they must learn position and locality from data. This makes them more flexible but requiring more data and compute than CNNs/RNNs. With enough data and scale, Transformers outperform both. The lack of locality bias is why Transformers scale better — they don't impose any specific structural prior on the task.

Parameter-efficient fine-tuning alternatives to full fine-tuning. Prefix tuning (Li & Liang 2021): prepend a sequence of virtual tokens (prefix) to each layer's key and value, optimize only these prefix embeddings (~0.1% of params). The prefix acts as a soft continuous prompt. Prompt tuning (Lester et al. 2021): only add soft prompts at the input embedding layer. As model scale increases, prompt tuning approaches full fine-tuning quality. These methods: don't modify model weights, allow multiple tasks per model (different prefixes), and require minimal storage (just prefix embeddings). Compared to LoRA, prefix tuning is sometimes less stable.

Causal LM (GPT): P(x_1,...,x_T) = Π P(x_t | x_{<t}). Only sees past. Trains with cross-entropy at every position simultaneously (full sequence in one forward pass using causal mask). Used for generation. Non-causal / Masked LM (BERT): randomly masks tokens and predicts them using bidirectional context. Not a valid joint probability — cannot directly generate (must sample masked positions iteratively). Better for understanding tasks. Efficiency: CLM trains on 100% of tokens per forward pass; MLM on only ~15% (masked tokens). XLNet (autoregressive with permutation-based bidirectional context) tries to combine both.

Emergent abilities are capabilities absent in small models that appear unpredictably at larger scales, not as smooth extrapolations. Examples: GPT-3 performs few-shot arithmetic; smaller versions cannot. Chain-of-thought reasoning only works above ~50B parameters. Models spontaneously learn to program, solve math, translate between many languages — despite not being explicitly trained for these. The emergence appears sharp on metrics (near-zero → near-SOTA in one scale jump) but may be partly a metric artifact. Emergence motivates scaling: properties that don't extrapolate smoothly from small experiments may suddenly appear in large models.

RAG (Lewis et al. 2020) augments generation by retrieving relevant documents at inference time: (1) Encode query with a dense retriever (e.g., DPR). (2) Retrieve top-k documents from a vector database. (3) Concatenate documents with query in the Transformer context. (4) Generate answer conditioned on retrieved context. RAG updates knowledge by updating the retrieval database (no retraining) and reduces hallucination by grounding responses in retrieved facts. It effectively gives the model access to millions of documents without fitting them in context. Used in ChatGPT plugins, Bing, and enterprise AI systems.

Original Transformer (2017): encoder-decoder for MT, 65M params, learned absolute positional encoding, post-LN, ReLU activation. Modern LLMs (LLaMA 2, 2023): decoder-only, 7B–70B params, RoPE (rotary positional encoding), pre-LN with RMSNorm, SwiGLU activation, Grouped Query Attention (GQA), context 4096–128k tokens, trained on 2T tokens. Key improvements: RoPE for better generalization, GQA for faster inference (smaller KV cache), SwiGLU for better performance, pre-LN for stable training, much more data and compute. The core self-attention mechanism remains the same; everything around it has been refined.

AlphaFold 2 (DeepMind, 2021) solved the protein structure prediction problem with near-experimental accuracy. Its core component, Evoformer, is a Transformer-like block that processes both multiple sequence alignment (MSA — evolutionary information from related proteins) and pair representations simultaneously. Evoformer uses axial attention (row + column attention on the MSA matrix), triangle updates (pair distances satisfy triangle inequality), and alternates between sequence and structure reasoning. The structure module then predicts 3D coordinates. AlphaFold 2 represents Transformer's extension beyond text/vision to structural biology, showing their power for scientific discovery.

Temperature T scales logits before softmax: P(x_t = w) = softmax(logits_w / T). T=1: standard distribution. T < 1: sharper, more deterministic (top tokens dominate). T > 1: flatter, more random/creative. T=0: greedy decoding (always take the argmax). For factual Q&A, use low T (~0.2) for consistency. For creative writing, use high T (~0.8-1.0) for diversity. Related: top-p (nucleus) sampling — sample from smallest set of tokens whose cumulative probability exceeds p (e.g., p=0.9). Top-k sampling: restrict to top k tokens. Combining low T + top-p is the standard for controlled generation.

import torch.nn.functional as F
def sample_with_temperature(logits, T=1.0):
    probs = F.softmax(logits / T, dim=-1)
    return torch.multinomial(probs, 1)

Pre-training data quality and scale are the most important factors for LLM quality. Sources: Common Crawl (web), Wikipedia, books (BookCorpus, Pile-Books3), code (GitHub), academic papers (arXiv, PubMed). Data cleaning: deduplicate near-identical documents, filter low-quality/harmful content, apply language detection, remove boilerplate. Data mixture matters: more Wikipedia and books improve factual accuracy; code data improves reasoning. LLaMA 2 trained on 2T tokens; GPT-3 on 300B tokens. The Chinchilla paper showed: for fixed compute, train on more tokens with a smaller model rather than a huge model on fewer tokens.

FlashAttention (Dao et al. 2022) is an IO-aware implementation of exact attention. Standard attention materializes the full n×n attention matrix in GPU HBM (slow, O(n²) memory). FlashAttention tiles computation in SRAM: (1) Load Q,K,V chunks. (2) Compute attention scores and update output in SRAM using online softmax. (3) Never write the full attention matrix to HBM. Result: O(n) memory, 2–4× speedup, no approximation. Enabled training on sequences up to 16k+ tokens on standard hardware. Flash Attention 2 + 3 further optimize for H100 tensor cores. Now standard in HuggingFace Transformers, PyTorch 2.0, and all major frameworks.

# PyTorch 2.0 built-in FlashAttention
with torch.backends.cuda.sdp_kernel(enable_flash=True):
    out = F.scaled_dot_product_attention(q, k, v, is_causal=True)

Perplexity (PPL) measures how well a language model predicts a text: PPL = exp(−(1/T) Σ log P(x_t | x_{<t})). It's the exponentiated average negative log-likelihood — lower is better. A perplexity of 10 means the model is as uncertain as if choosing uniformly from 10 options at each step. WikiText-103 PPL: GPT-2-large ~20, GPT-3 ~10, GPT-4 ~3. PPL is easy to compute but doesn't capture all quality dimensions (a model with low PPL might be repetitive or factually wrong). PPL is used for model comparison, hyperparameter tuning, and as a training signal proxy.

import torch
def perplexity(model, tokens):
    with torch.no_grad():
        loss = model(tokens, labels=tokens).loss
    return torch.exp(loss).item()

The encoder processes the full input sequence through N stacked self-attention + FFN blocks, outputting a sequence of contextual embeddings H_enc ∈ R^{n×d_model} — one embedding per input token. These embeddings capture contextual relationships between all token pairs via self-attention. In encoder-decoder models (T5, BART), the decoder uses these via cross-attention at every decoder layer. In encoder-only models (BERT), the output embeddings are used directly: [CLS] for classification, token embeddings for NER/QA. The encoder does not generate — it only transforms the input into rich representations.

Teacher forcing trains auto-regressive decoders by feeding the ground-truth target tokens as inputs at each decoder step, rather than the model's own previous predictions. This allows computing all decoder positions in parallel using the causal mask — a single forward pass gives predictions for all positions simultaneously. Without teacher forcing, errors compound: early prediction mistakes corrupt all subsequent steps. At inference, teacher forcing is not used (ground-truth unavailable), leading to exposure bias — a train-test distribution mismatch. Scheduled sampling gradually mixes teacher and model tokens during training to reduce this gap.

XLNet (Yang et al. 2019) is an autoregressive model that uses permutation language modeling: for a random permutation of the sequence, predict each token using only the preceding tokens in that permutation order. This gives bidirectional context (unlike standard left-to-right LM) while maintaining autoregressive properties (no masking artifacts like BERT's [MASK] tokens). XLNet also uses Transformer-XL's recurrence mechanism for longer context. XLNet outperformed BERT on many benchmarks at release. Key insight: predicting unmasked tokens using all-order permutations avoids BERT's artificial [MASK] vs no-[MASK] discrepancy.

Decoder-only Transformers (GPT family, LLaMA, Claude, Gemini) use only the decoder component with causal masked self-attention — no encoder, no cross-attention. All tasks are framed as conditional text generation: prepend context, generate completion. Pre-trained with next-token prediction. At sufficient scale (>1B parameters), decoder-only models can perform understanding tasks competitively with BERT-style models. Advantages: unified architecture for all tasks, easy few-shot prompting, natural extension to generation. Most modern production LLMs are decoder-only. The simplicity of one forward pass per token enables efficient batch generation.

SwiGLU (Shazeer 2020) is a gated activation used in LLaMA, PaLM, and other modern LLMs as a replacement for FFN(x) = max(0, xW_1)W_2. SwiGLU: FFN(x) = (xW_1 ⊙ Swish(xW_3)) W_2, where Swish(x) = x·σ(x), ⊙ is elementwise product. The gate xW_3 controls how much of xW_1 passes through. This adds a multiplicative interaction that improves expressiveness with minimal extra parameters. SwiGLU consistently outperforms ReLU and GELU FFNs on language modeling benchmarks, especially combined with pre-LN and RMSNorm. It requires 3 weight matrices instead of 2, so d_ff is reduced to 2/3 × 4d to maintain parameter count.

def swiglu(x, W1, W3, W2):
    return (x @ W1 * torch.sigmoid(x @ W3) * (x @ W3)) @ W2

MoE Transformers replace each FFN with N expert FFNs and a gating network. For each token, the router selects top-K experts (typically K=1 or 2): gate(x) = softmax(xW_g)[:K]. Only activated experts process the token — model has N×FFN parameters but each token uses K×FFN FLOPs. Mixtral 8×7B: 8 experts per layer, 2 active per token → 12.9B active params, 46.7B total. MoE enables large model capacity with manageable inference cost. Training challenge: load balancing (router must distribute tokens evenly across experts). Auxiliary loss penalizes unbalanced routing to prevent expert collapse.

The token prediction head maps the final hidden state h_t ∈ R^{d_model} to vocabulary logits: logits = h_t W_E^T ∈ R^V, where W_E is the token embedding matrix (weight tying). The probability of next token w is: P(w) = softmax(logits)[w]. During training: minimize cross-entropy loss against the true next token. During inference: sample from the distribution (temperature, top-k, top-p) or take argmax (greedy). The logits can be biased (repetition penalty, presence penalty) to control generation quality. The vocabulary size V (typically 32k–100k) and embedding dimension d_model determine this final projection's compute cost.

logits = hidden @ embed_weight.T  # (batch, seq, vocab)
loss = F.cross_entropy(logits.view(-1, V), targets.view(-1))

PEFT methods adapt large pre-trained models to downstream tasks by updating only a small number of parameters, avoiding the cost of full fine-tuning. Main methods: (1) LoRA — low-rank weight update ΔW=BA (most popular). (2) Prefix tuning — soft prompt prepended to K,V of each layer. (3) Prompt tuning — only input embeddings trained. (4) Adapter layers — small FFN bottlenecks inserted in each Transformer block. (5) IA³ — scale attention keys, values, and FFN activations by learned vectors. PEFT typically updates <1% of parameters, enabling fine-tuning on consumer hardware and rapid task switching by swapping only the small trained modules.

from peft import get_peft_model, LoraConfig
config = LoraConfig(r=16, target_modules=['q_proj','v_proj','k_proj','o_proj'])
model = get_peft_model(base_model, config)
model.print_trainable_parameters()

Softmax converts raw logits (unnormalized scores) to a valid probability distribution over the vocabulary: P(w) = exp(logit_w) / Σ_v exp(logit_v). Properties: outputs ∈ (0,1), sum to 1. In attention, softmax converts raw scores to attention weights. Cross-entropy loss: L = −log P(true_token). Softmax's exponential magnifies differences between logits — one dominant logit takes most probability (sharp distribution). Temperature scaling modifies logits before softmax: logits/T, making distribution sharper (T<1) or flatter (T>1). Numerical stability: subtract max logit before exp to prevent overflow.

import torch.nn.functional as F
probs = F.softmax(logits / temperature, dim=-1)
next_token = torch.multinomial(probs, 1)

Pre-training trains a model from scratch on massive datasets with self-supervised objectives (MLM, next-token prediction). Resources: thousands of GPU-days, terabytes of data, often done by large AI labs. The result is a general-purpose model that understands language/structure. Fine-tuning adapts this model to a specific task with small labeled datasets (100–100k examples) and small learning rate (2e-5 to 5e-5) for 3–10 epochs. Catastrophic forgetting is a risk: fine-tuning may overwrite pre-trained knowledge. Mitigation: use low LR, regularize toward pre-trained weights, or use PEFT. Fine-tuning typically takes minutes to hours on one GPU.

from transformers import TrainingArguments, Trainer
args = TrainingArguments(num_train_epochs=3, learning_rate=2e-5, per_device_train_batch_size=16)
trainer = Trainer(model=model, args=args, train_dataset=train_ds)
trainer.train()

Beam search generates text by keeping the top-B (beam width) most probable partial sequences at each step, instead of greedily picking the single best token. At each step: for each of B hypotheses, expand all vocab tokens and keep the B highest-probability sequences. Final output: best-scoring complete sequence. Beam search often produces more coherent, higher-probability text than greedy decoding, but can generate repetitive/generic outputs (tends to prefer common phrases). Beam width 4–8 is typical. For open-ended generation, sampling-based methods (top-p, temperature) produce more diverse, creative outputs than beam search.

from transformers import AutoModelForSeq2SeqLM
outputs = model.generate(inputs, num_beams=4, max_new_tokens=100)