Cheat sheet · LLMs & Agents

RAG & Vector Databases cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Retrieval-Augmented Generation (RAG) grounds LLM outputs in external knowledge fetched at query time, reducing hallucination and enabling fresh, private, and citable answers. This sheet covers the pipeline, embeddings and vector indexes (HNSW/IVF/PQ), hybrid retrieval and reranking, evaluation, and production concerns (cost, security, freshness) relevant to 2026 systems.

RAG Fundamentals & Pipeline

RAG = Retrieve relevant context from a knowledge store, then Augment the prompt and Generate. It decouples knowledge (in the store) from reasoning (in the LLM), so you update facts without retraining.
Why RAG over fine-tuning: cheaper to update, supports citations/attribution, handles private/fresh data, and reduces hallucination. Fine-tuning teaches style/format/behavior; RAG injects knowledge. They are complementary, not mutually exclusive.
Two phases: (1) Indexing (offline) — load, chunk, embed, store vectors + metadata; (2) Inference (online) — embed query, retrieve top-k, rerank, build prompt, generate, optionally cite.
Core failure modes map to stages: retrieval miss (relevant chunk not returned), low precision (noisy context), context-stuffing ('lost in the middle'), and generation ignoring or contradicting the context.
2026 landscape: 'agentic RAG' (LLM plans multi-step retrieval, query decomposition, tool calls), GraphRAG (entity/relationship graphs for multi-hop), and long-context models complement but do not replace RAG — retrieval still wins on cost, freshness, access control, and citations at scale.
Variants: naive RAG, advanced RAG (pre/post-retrieval optimization), modular RAG, self-RAG (model decides when to retrieve and self-critiques), and corrective RAG (CRAG: grade retrievals, fall back to web search).

Chunking & Embeddings

Chunking strategies: fixed-size with overlap (simple baseline), recursive/structure-aware (split on headings/paragraphs/sentences), semantic chunking (split where embedding similarity drops), and document-layout-aware for PDFs/tables.
Typical chunk size 200-800 tokens with 10-20% overlap to preserve context across boundaries. Too large = diluted relevance + wasted context window; too small = fragmented meaning and more chunks to retrieve/merge.
Embeddings map text to dense vectors where semantic similarity ~= geometric proximity. Use the SAME model (and version) for indexing and querying; never mix embedding spaces.
Use cosine similarity for normalized embeddings (the default for most text models). Cosine = dot product on L2-normalized vectors; raw dot product favors magnitude (used by some models); Euclidean (L2) is rarely best for text.
Strong 2026 choices: OpenAI text-embedding-3-large, Voyage (voyage-3; Anthropic's recommended embedding provider — Anthropic has no first-party embeddings API), Cohere embed-v3, and open models (BGE, E5, GTE, Nomic, Jina). Many support Matryoshka (truncatable dims) to trade accuracy for storage/speed.
Advanced: store metadata (source, date, section) for filtering and citations; consider multi-vector (ColBERT-style late interaction) for higher recall at higher cost; prepend a document-level context summary to each chunk ('contextual retrieval', Anthropic) to fix ambiguous fragments.

Vector Indexes (ANN) & Databases

Exact (flat/brute-force) search is O(N*d) per query — accurate but slow at scale. Approximate Nearest Neighbor (ANN) trades a little recall for large latency/throughput gains.
HNSW (Hierarchical Navigable Small World): graph-based, excellent recall/latency, high memory. Key params: M (edges per node), ef_construction (build quality), ef_search (recall-vs-latency at query time). The most popular default.
IVF (Inverted File): cluster vectors into nlist centroids, search the nprobe nearest clusters. Lower memory, tunable; recall rises with nprobe. Often combined as IVF_PQ for scale.
PQ (Product Quantization) & SQ (Scalar Quantization) compress vectors to cut memory ~4-32x at some recall loss. Use at billion-scale and rerank survivors on full-precision vectors. Binary quantization + Hamming distance is increasingly common as a cheap first stage.
Recall metric: recall@k = (relevant items retrieved in top-k) / (total relevant). Tune index params on a labeled set to hit a recall target at acceptable p95 latency.
Databases: dedicated (Pinecone, Weaviate, Qdrant, Milvus) vs embedded libs (FAISS, Chroma, LanceDB) vs Postgres/pgvector (+ pgvectorscale, HNSW), plus search engines (Elasticsearch/OpenSearch). Choose on scale, filtering needs, hybrid support, and ops burden — pgvector is often enough below ~10M vectors.

Retrieval, Hybrid Search & Reranking

Dense (vector) retrieval captures semantics/synonyms but can miss exact terms (IDs, codes, rare names). Sparse/lexical retrieval (BM25, SPLADE) nails keywords. Hybrid combines both for robust recall.
Fuse hybrid results with Reciprocal Rank Fusion (RRF) — rank-based, so no score normalization is needed — or weighted score blending. RRF is the common default.
Two-stage retrieval: a cheap recall-oriented first stage (retrieve top 50-100), then a cross-encoder reranker (e.g., Cohere Rerank, BGE-reranker) scores query-doc pairs for precision. Big quality win for small latency cost.
Query transformations: HyDE (generate a hypothetical answer, embed it), multi-query expansion, query rewriting/decomposition for multi-hop, and step-back prompting. These improve recall on vague queries.
Always apply metadata filtering (date, tenant, ACLs) for security and relevance. Prefer filtering inside the ANN index (native filtered search) over post-filtering, which can return too few results.
Context assembly: dedupe, order by relevance, respect the token budget, and mitigate 'lost in the middle' by placing the strongest evidence at the start and end. Include source IDs to enable citations.

Evaluation & Observability

Evaluate retrieval and generation separately. Retrieval: recall@k, precision@k, MRR, NDCG, hit-rate. Generation: faithfulness/groundedness, answer relevance, completeness.
RAG triad (RAGAS / TruLens): Context Relevance (retrieved context fits the query), Faithfulness/Groundedness (answer supported by context, no hallucination), Answer Relevance (answer addresses the question).
Build a golden eval set (question, ground-truth answer, expected source chunks). Use LLM-as-judge for scale but calibrate against human labels; judges are biased (position, verbosity) and need rubrics + few-shot anchors.
Faithfulness ~= (answer claims supported by context) / (total answer claims). Surface citations so users and evaluators can verify.
Production observability: log queries, retrieved chunks + scores, per-stage latency, token cost, and user feedback. Track drift (new docs, changing queries) and re-tune chunking/index/k.
Regression-test the pipeline in CI against the golden set before shipping prompt/model/index changes — RAG quality is emergent and easily broken by small tweaks.

Production Concerns & Cost

Freshness: incremental indexing/upserts, change-data-capture from sources, TTL/versioning, and re-embedding on model upgrades (an expensive full reindex — plan a blue/green migration).
Security & multi-tenancy: enforce per-user ACLs at retrieval time (filter by permission), isolate tenants (namespaces/partitions), and defend against prompt injection via retrieved/untrusted content (treat context as data, sanitize, sandbox tools).
Latency budget = embed query + ANN search + rerank + generate. Cache aggressively: embedding cache, semantic cache for repeated questions, and prompt caching on the LLM side.
Cost levers: smaller/Matryoshka embeddings, quantized indexes, fewer/shorter chunks, reranking to send fewer tokens to the LLM, and cheaper generation models fed with strong context.
Scaling: HNSW memory grows with N and M; shard/partition, use disk-backed indexes (DiskANN) or quantization at billion-scale, and separate read replicas from the write/index path.
Robustness: handle no-good-result cases (abstain / 'I don't know'), add CRAG-style fallback (web search), monitor for stale/contradictory sources, and always show citations to build trust and aid debugging.

Common pitfalls

Using different embedding models (or versions) for indexing vs querying — silently destroys relevance because vectors live in different spaces.
Forgetting to normalize vectors when the index uses cosine/inner product, producing magnitude-biased rankings (and note FAISS IndexHNSWFlat defaults to L2 unless you pass METRIC_INNER_PRODUCT).
Treating bigger context as better: stuffing top-k chunks causes 'lost in the middle', higher cost, and more distraction. Rerank and trim instead.
Bad chunking (too large, too small, or splitting mid-sentence/mid-table) — the most common cause of poor RAG, yet often ignored in favor of model tweaks.
Skipping a reranker and trusting raw vector scores; recall without precision floods the prompt with noise.
Pure dense retrieval that misses exact keywords (product codes, names, numbers) — needs hybrid/lexical search.
Post-filtering metadata after ANN instead of native filtered ANN, which can return too few or zero results after the filter.
No evaluation harness: changes to prompt/model/k 'feel better' but silently regress; not separating retrieval vs generation metrics hides the real failure stage.
Trusting retrieved content blindly — prompt injection via poisoned documents can hijack the model. Treat context as untrusted data.
Assuming long-context models make RAG obsolete — they raise cost/latency, still hallucinate, and lack freshness, citations, and access control.
No 'I don't know' path: the model fabricates answers when retrieval fails instead of abstaining.
Ignoring re-embedding cost when upgrading the embedding model — it requires reindexing the entire corpus.

Interview tips

Be ready to draw the end-to-end pipeline (index-time vs query-time) and name what can fail at each stage and how you'd detect it.
Explain when to choose RAG vs fine-tuning vs long-context — and that they combine (RAG for knowledge, fine-tune for behavior/format).
Know HNSW vs IVF vs PQ tradeoffs and their tunable params (ef_search, nprobe, M) and how each moves the recall/latency/memory triangle.
Articulate cosine vs dot vs Euclidean and why normalization matters; expect to write recall@k and the cosine formula.
Justify hybrid search + RRF + cross-encoder reranking; interviewers probe whether you understand recall-then-precision two-stage design.
Discuss evaluation concretely: golden sets, the RAG triad (faithfulness / context relevance / answer relevance), LLM-as-judge calibration, and CI regression testing.
Cover production realities: ACL-aware retrieval, multi-tenancy, prompt-injection defense, incremental indexing, re-embedding migrations, caching, and cost levers.
Mention current directions (agentic RAG, GraphRAG for multi-hop, self-RAG/CRAG, contextual retrieval, Matryoshka/quantized embeddings) to show you're up to date.
Have a debugging story: given 'answers are wrong', how do you isolate whether it's retrieval, ranking, chunking, or generation?
Give sensible defaults with numbers (chunk ~300-500 tokens, ~15% overlap, retrieve 20-50, rerank to top 3-5) to signal hands-on experience.

Read the full lesson