LLMs and GenAI

For large d_k, the dot products grow large in magnitude, pushing the softmax into regions with extremely small gradients. Scaling by 1/sqrt(d_k) keeps the variance of the logits roughly constant (around 1), which stabilizes gradients and training.

import torch, torch.nn.functional as F
def attention(Q,K,V):
    dk = Q.size(-1)
    scores = Q @ K.transpose(-2,-1) / dk**0.5
    return F.softmax(scores, dim=-1) @ V

RAG keeps the model frozen and supplies relevant external context at inference time via retrieval, making knowledge easy to update and cite. Fine-tuning bakes knowledge/behavior into the weights, which is better for style/format and latency but costly to update and prone to stale or forgotten facts.

LoRA freezes the pretrained weights and learns a low-rank update ΔW = B·A (rank r << d) for selected layers, so only the small A and B matrices are trained. This drastically reduces trainable parameters and optimizer state, while the original weights stay fixed and the adapters can be merged at inference.

# ΔW = B @ A, with A:(r,d), B:(d,r), r << d
# W_eff = W_frozen + (alpha/r) * B @ A

RoPE encodes position by rotating query/key vectors so that attention depends on relative positions, which generalizes better to sequence lengths longer than those seen in training and naturally captures relative distance. Absolute learned embeddings are fixed to a max length and extrapolate poorly.

Temperature rescales logits before softmax to control overall randomness (lower = sharper). Top-k restricts sampling to the k highest-probability tokens; top-p restricts to the smallest set of tokens whose cumulative probability exceeds p, adapting the candidate count to the distribution's shape.

LLMs are trained to predict the most likely next token, not to assert truth, so when parametric knowledge is missing or ambiguous they generate fluent but unsupported content. Grounding via RAG with citations, lowering temperature, and asking the model to abstain when unsure are common mitigations.

During generation each new token attends to all previous tokens; caching the past keys and values avoids recomputing them every step, turning per-step attention cost from quadratic to linear in sequence length (and total decoding from O(n^3) to O(n^2)). The trade-off is memory that grows linearly with sequence length, batch size, layers, and heads, often becoming the inference bottleneck.

Encoder-only models use bidirectional attention and are trained with masked language modeling, ideal for understanding/embedding tasks. Decoder-only models use causal attention for autoregressive generation. Encoder-decoder models encode the input bidirectionally and decode autoregressively, suiting seq2seq tasks like translation.

RMSNorm normalizes only by the root-mean-square of activations without subtracting the mean or learning a bias, making it cheaper to compute while empirically matching LayerNorm's stabilizing effect. This reduces compute and memory at scale with negligible accuracy loss.

The reward model is trained on human preference comparisons to score outputs, providing the reward signal for policy optimization (e.g., PPO). A KL-divergence penalty against the frozen reference (SFT) model keeps the policy from drifting too far and reward-hacking, preserving fluency and general capability.

DPO reframes preference alignment as a single supervised classification-style loss directly on preferred vs. rejected pairs, implicitly optimizing the same KL-regularized reward objective without training a separate reward model or running online RL. This makes it simpler and more stable than PPO.

Embeddings encode semantic meaning as directions in a high-dimensional space, so cosine similarity measures the angle between vectors, capturing semantic closeness independent of magnitude. It is the standard relevance metric for retrieving documents whose meaning aligns with a query.

import numpy as np
def cosine(a,b):
    return a @ b / (np.linalg.norm(a)*np.linalg.norm(b))

Exact nearest-neighbor search over millions of high-dimensional vectors is O(N) per query and too slow at scale. ANN methods like HNSW build a navigable graph that finds near-optimal neighbors in roughly logarithmic time, trading a small recall loss for large latency gains.

CoT prompts the model to produce intermediate reasoning steps before the final answer, which improves performance on multi-step arithmetic, logic, and reasoning tasks. It helps most in larger models where reasoning ability has emerged and on problems requiring decomposition; it adds little on simple lookup tasks.

MHA gives each head its own queries, keys, and values. MQA shares a single key/value head across all query heads to shrink the KV cache and speed decoding, at some quality cost. GQA is the middle ground: query heads are split into groups that each share a K/V head, balancing speed and quality.

Chunks that are too large dilute relevance and waste context budget with off-topic text, hurting retrieval precision; chunks that are too small lose surrounding context and may split key information across boundaries. The right size (often with overlap) balances retrieval precision against semantic completeness.

An MoE layer has many expert FFNs but a router activates only the top-k per token, so total parameters can be huge while FLOPs per token stay small (sparse activation). This yields high capacity with the inference cost of a much smaller dense model, though it raises memory and load-balancing challenges.

BLEU/ROUGE measure n-gram overlap with references and miss semantically correct outputs phrased differently, and there is often no single correct answer for open-ended tasks. Alternatives include embedding-based metrics (BERTScore), LLM-as-a-judge, human preference evaluation, and task-specific functional checks.

Zero-shot gives only the task instruction; few-shot adds a handful of input-output examples in the prompt; both are forms of in-context learning, where the frozen model adapts its behavior from the prompt context without any weight updates. The model learns the pattern at inference time only.

Quantization stores/computes weights (and sometimes activations) in lower precision (e.g., INT8 or 4-bit) instead of FP16/32, cutting memory and increasing throughput. The trade-off is a usually small accuracy degradation; techniques like GPTQ, AWQ, and outlier handling minimize the quality loss.

Teacher forcing feeds the ground-truth previous token during training so the model learns next-token prediction efficiently and in parallel. At inference the model consumes its own predictions, creating exposure bias: errors compound because the model never trained on its own (possibly wrong) outputs.

Prompt injection is an attack where untrusted input (e.g., retrieved web content) contains instructions that override or hijack the system's intended behavior, especially dangerous in agentic/RAG systems. Jailbreaking specifically aims to bypass safety guardrails to elicit disallowed content; injection is the broader instruction-hijacking class.

The FFN projects up to a larger hidden dimension (commonly 4x d_model), applies a non-linearity, then projects back, giving the model capacity to mix and transform per-token features. Most of a transformer's parameters and a large share of compute live in these FFN layers.

Residual connections add a layer's input to its output, providing gradient highways that prevent vanishing gradients and enable very deep stacks. Pre-normalization (LayerNorm/RMSNorm before the sublayer) keeps activations well-scaled, making training of deep transformers more stable than post-norm.

Hybrid retrieval combines dense embedding similarity (semantic) with sparse lexical scoring like BM25, then fuses results (e.g., reciprocal rank fusion). Dense search captures meaning while lexical search catches exact terms, rare keywords, codes, and acronyms that embeddings may miss, improving overall recall.

ReAct interleaves reasoning traces (Thought) with actions (tool calls) and observations in a loop, so the model plans, acts, observes results, and re-plans iteratively. This grounds reasoning in external tool feedback, reducing hallucination and enabling multi-step task completion.

Reranking re-scores the top candidates from the initial retriever to put the most relevant passages first. A cross-encoder jointly encodes the query and each passage together, capturing fine-grained interactions for higher accuracy than the bi-encoder used for fast first-stage retrieval, at higher compute cost per pair.

The context window is the maximum number of tokens the model can attend to at once. Standard self-attention compares every token with every other token, so compute and memory scale quadratically (O(n^2)) with sequence length, which is why long contexts are expensive and motivate efficient-attention variants.

Catastrophic forgetting occurs when fine-tuning on a narrow dataset overwrites general capabilities learned in pretraining. Parameter-efficient methods (LoRA/adapters) keep base weights frozen, and mixing in replay/general data or using low learning rates preserves prior knowledge while learning the new task.

Subword tokenization keeps the vocabulary fixed and manageable while representing any word, including rare or unseen ones, as a sequence of known subword units, eliminating out-of-vocabulary failures. It balances sequence length against vocabulary size and handles morphology across many languages.

A small, fast draft model proposes several tokens ahead, and the large target model verifies them in a single parallel forward pass, accepting the longest correct prefix. Because verification preserves the target model's distribution, it yields identical-quality outputs with fewer expensive sequential steps.

Embedding models map text to fixed-dimensional vectors capturing semantic meaning for retrieval, clustering, or similarity. Generative LLMs autoregressively produce text token by token. They are optimized for different objectives and are often used together in RAG (embeddings for retrieval, LLM for generation).

Chinchilla showed that prior large models were undertrained: for a fixed compute budget, model size and training tokens should scale roughly in equal proportion (~20 tokens per parameter). A smaller model trained on more data can outperform a larger, data-starved one.

LLMs tend to use information best when it appears at the beginning or end of a long context and perform worse when the relevant content sits in the middle, producing a U-shaped accuracy curve. This motivates placing key context near the edges and careful reranking in RAG.

Prompting for JSON still allows the model to emit malformed or schema-violating output occasionally, breaking downstream parsers. Constrained decoding (grammar/FSM-based, e.g., JSON schema) masks invalid next tokens during generation, guaranteeing syntactically valid, schema-conformant output deterministically.

Extractive summarization selects and copies salient sentences verbatim from the source, while abstractive summarization generates new phrasing that paraphrases and condenses meaning. Generative LLMs naturally do abstractive summarization, which is more fluent but carries higher hallucination risk requiring grounding/faithfulness checks.

LLM-as-a-judge uses a strong model to score or compare outputs against a rubric, scaling evaluation cheaply. Known biases include position bias (favoring the first/second answer), verbosity bias (preferring longer answers), and self-preference (favoring the same model family); mitigations include randomizing order, pairwise comparison, and calibrated rubrics.

The system prompt sets persistent instructions, persona, constraints, and safety rules that should govern the whole conversation, given higher priority than user turns in the model's instruction hierarchy. Proper use reduces drift and injection risk, but it is not a hard security boundary on its own.

FlashAttention computes exact attention but tiles the computation and uses online softmax to avoid materializing the full N x N attention matrix in slow HBM, keeping intermediates in fast on-chip SRAM. It is IO-aware: by reducing memory reads/writes it cuts both runtime and memory, especially for long sequences.

Continued pretraining keeps doing next-token prediction on domain corpora to inject knowledge. Instruction tuning (SFT) trains on instruction-response pairs to teach the model to follow tasks/format. Preference alignment (RLHF/DPO) then optimizes for human-preferred, helpful/harmless behavior beyond what SFT demonstrations capture.