Cheat sheet · LLMs & Agents

Transformers cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

The Transformer replaces recurrence and convolution with self-attention, enabling massively parallel sequence modeling and forming the backbone of modern LLMs. This sheet covers the attention mechanism, the full architecture, positional encodings, scaling/efficiency tricks, training, and the inference-time optimizations that dominate 2026-era practice.

Scaled Dot-Product & Multi-Head Attention

Attention maps a query against a set of keys to produce a weighted sum of values; weights are a softmax over query-key similarity. Q, K, V are linear projections of the input.
The 1/sqrt(d_k) scaling keeps dot products from growing with dimension. If q and k have unit-variance independent components, q·k has variance d_k; dividing by sqrt(d_k) restores unit variance and keeps softmax out of saturated, low-gradient regions.
Multi-head attention runs h attention ops in parallel on d/h-dim subspaces, then concatenates and applies an output projection W_O — letting different heads attend to different relations (syntax, position, coreference).
Complexity is O(n^2 * d) time and O(n^2) attention-matrix memory in sequence length n — the core scalability bottleneck.
Three attention modes: encoder self-attention (full bidirectional), decoder masked self-attention (causal, no peeking ahead), and cross-attention (decoder queries attend to encoder keys/values).
Causal masking sets upper-triangular (future) positions to -inf before softmax so position i attends only to positions <= i.

The Transformer Block & Full Architecture

Each block = attention sublayer + position-wise feed-forward network (FFN), each wrapped with a residual connection and a normalization layer.
FFN is two linear layers with a nonlinearity (GELU, or gated SwiGLU), typically expanding to ~4x the model dim (SwiGLU often uses ~8/3x to match params). It holds most non-attention parameters and acts like per-token key-value memory.
Pre-LN (norm before each sublayer) is now standard over the original Post-LN: it stabilizes gradients in deep stacks and reduces sensitivity to learning-rate warmup.
Encoder-only (BERT): bidirectional, for understanding/embeddings. Decoder-only (GPT/Llama): causal, for generation — dominant for LLMs. Encoder-decoder (T5): for seq2seq such as translation.
Modern LLMs use RMSNorm (cheaper than LayerNorm, no mean-centering), SwiGLU FFNs, and rotary position embeddings (RoPE).
The residual stream is the central communication channel — every sublayer reads from and writes to it additively.

Positional Encoding

Self-attention is permutation-equivariant, so position must be injected explicitly — otherwise the model treats input as a bag of tokens.
The original Transformer used fixed sinusoidal encodings added to embeddings; learned absolute embeddings (BERT/GPT-2) are an alternative but extrapolate poorly beyond the trained length.
RoPE (Rotary Position Embedding) rotates Q and K by an angle proportional to position, so the attention dot product depends on relative position — the de facto standard in modern LLMs (Llama, Qwen, etc.).
ALiBi adds a distance-based linear bias to attention scores instead of modifying embeddings; both RoPE and ALiBi support better length extrapolation.
Context-length extension (RoPE scaling: linear/NTK-aware/YaRN) lets a model trained at e.g. 4k generalize to 128k+ with light or no fine-tuning.

Efficient Attention & Long Context

FlashAttention computes EXACT attention without materializing the n*n matrix, using tiling + online (streaming) softmax in fast SRAM. It is IO-aware: memory is linear in n and wall-clock is much faster. FlashAttention-2/3 are standard on modern GPUs.
Sparse/windowed attention (Longformer; sliding-window in Mistral) restricts each token to a local window plus a few global tokens, cutting cost toward O(n).
Linear/kernel attention (Performer, linear attention) approximates softmax to reach O(n), trading some accuracy.
Mixture-of-Experts (MoE) replaces the dense FFN with many experts and routes each token to top-k — scaling total parameters while keeping per-token FLOPs roughly constant (Mixtral, DeepSeek-V3).
State-space models (Mamba/SSMs) and attention-SSM hybrids challenge attention on very long sequences via linear-time recurrence, though attention still dominates production LLMs in 2026.
Gradient checkpointing trades compute for memory during training by recomputing activations in the backward pass.

Inference Optimization (KV cache & decoding)

Autoregressive decoding caches past keys/values (KV cache) so each new token's attention costs O(n) instead of recomputing O(n^2). KV-cache memory grows with batch * layers * 2 * kv_heads * head_dim * seq_len and is the main serving bottleneck.
GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) share K/V across query heads, shrinking the KV cache with minimal quality loss — standard in Llama-2/3 and most modern models.
PagedAttention (vLLM) stores the KV cache in non-contiguous fixed-size pages, like OS virtual memory, reducing fragmentation and enabling high-throughput batched serving.
Decoding is memory-bandwidth bound, not compute bound; speculative decoding uses a small draft model to propose tokens that the large model verifies in parallel, cutting latency with identical output distribution.
Quantization (INT8/INT4, FP8, GPTQ/AWQ) and continuous (in-flight) batching are core levers for cheap, high-throughput inference.
Prefill (parallel over the prompt, compute-bound) vs decode (sequential, memory-bound) have very different profiles — disaggregating them across hardware is a common serving optimization.

Training & Practical Considerations

Pretraining objective is typically next-token prediction (causal LM) with cross-entropy; BERT-style models use masked LM. Quality is often reported as perplexity = exp(cross-entropy).
Optimizer is AdamW with cosine or warmup-stable-decay LR schedules; warmup matters most for Post-LN and large-batch training.
Mixed precision with bf16 is standard; bf16's wider exponent avoids the overflow/underflow issues of fp16, removing the need for loss scaling.
Chinchilla scaling: for a fixed compute budget, scale parameters and training tokens together (~20 tokens per parameter is a rough compute-optimal ratio). Much modern practice over-trains smaller models far past this for cheaper inference.
Alignment after pretraining: SFT (instruction tuning) then preference optimization via RLHF (PPO) or the simpler DPO; parameter-efficient tuning via LoRA/QLoRA is standard for adaptation.
Data quality/dedup, weight decay, and (sometimes) dropout drive generalization; large modern runs often use little or no dropout.

Common pitfalls

Forgetting the 1/sqrt(d_k) scaling — softmax saturates, gradients vanish, and training stalls.
Confusing the causal mask (decoder, prevents future leakage) with the padding mask (ignores pad tokens) — both are needed and serve different purposes.
Claiming attention is O(n) — vanilla attention is O(n^2) in FLOPs; FlashAttention cuts memory and wall-clock but the FLOPs stay quadratic.
Thinking FlashAttention is an approximation — it computes exact attention; it is an IO/memory optimization, not a sparse approximation.
Believing MoE makes the model faster overall — it scales parameters at roughly constant per-token FLOPs but increases total memory and adds routing/load-balancing complexity.
Adding extra absolute positional info to a model that already uses RoPE/ALiBi, or assuming learned absolute positions extrapolate to longer contexts (they do not).
Ignoring the KV cache when reasoning about inference memory/latency — it can dwarf model weights at long context and large batch.
Using Post-LN deep stacks without warmup and wondering why training diverges; Pre-LN is more robust.
Treating decode throughput as compute-bound — it is memory-bandwidth bound, which is why MQA/GQA and quantization help so much.
Conflating parameter count with number of heads, or assuming more heads is always better — head dim shrinks as heads grow, and total attention params stay fixed.

Interview tips

Be ready to write scaled dot-product attention from scratch and explain every term, including the mask and the scaling factor.
Explain why the scale is sqrt(d_k) specifically: the variance of the query-key dot product grows linearly with d_k.
Derive the time and memory complexity of attention and state exactly how FlashAttention changes the memory term but not the FLOPs.
Compare encoder-only vs decoder-only vs encoder-decoder and give a concrete use case for each.
Discuss how MQA/GQA and PagedAttention attack the KV-cache bottleneck — a strong signal of real serving experience.
Explain RoPE vs absolute positional encoding and how context windows are extended (NTK/YaRN scaling).
Contrast prefill vs decode phases and articulate why one is compute-bound and the other memory-bound.
Know the alignment pipeline (pretrain -> SFT -> RLHF/DPO) and where LoRA/QLoRA fit.
Reason about Chinchilla scaling and the inference-cost tradeoff of over-training smaller models.
Expect a follow-up on why Transformers beat RNNs: parallelism over sequence length and constant-length paths for long-range dependencies (vs O(n) in RNNs).

Read the full lesson Test yourself — 26 question quiz