Advanced Retrieval & RAG

BM25 is a sparse retrieval method based on exact term matching with TF-IDF-like term frequency and inverse document frequency weighting. It excels at keyword precision but fails on synonyms and paraphrases. Dense retrieval encodes queries and documents into continuous vector spaces using neural encoders (bi-encoders); similarity is measured by dot product or cosine similarity, capturing semantic meaning beyond lexical overlap. Dense retrieval generalizes better to paraphrased queries but requires GPU-intensive encoding and approximate nearest neighbor indexes. Hybrid systems combining both typically outperform either alone.

RAG combines a retrieval system with a generative LLM: given a query, relevant documents are retrieved from an external corpus and prepended to the LLM's context, grounding generation in retrieved evidence. Pure parametric LLMs store knowledge in weights—which become stale, may hallucinate, and cannot cite sources. RAG allows: (1) knowledge updates without retraining (replace the document corpus); (2) attribution by showing retrieved sources; (3) handling of long-tail facts unlikely to be memorized; (4) domain-specific knowledge injection. The RAG paper (Lewis et al., 2020) showed substantial gains on open-domain QA benchmarks over both pure retrieval and pure generation.

A bi-encoder encodes query and document independently into vectors, enabling pre-computation of document embeddings for fast ANN retrieval at O(1) inference over the index. A cross-encoder concatenates query and document and runs them jointly through a transformer, allowing full cross-attention between them for much higher accuracy but O(N) cost over all N documents. The standard pipeline uses bi-encoders for first-stage retrieval (recall), then re-ranks the top-k candidates with a cross-encoder (precision). Cross-encoders are used only on small candidate sets (50–200) to keep latency acceptable.

HNSW builds a multi-layer proximity graph where higher layers form sparse long-range connections (coarse navigation) and lower layers form dense local connections (fine-grained search). A query starts at the top layer, greedily navigating to the nearest neighbor, then descends layer by layer. This mimics a skip list and achieves O(log N) search complexity. HNSW provides excellent recall-latency tradeoffs with tunable parameters: M (max edges per node) and ef (exploration factor). Used in FAISS, Weaviate, Qdrant, and Chroma. Trade-off: high memory usage (O(M×N) edges) and slow incremental updates.

Hybrid search combines sparse (BM25) and dense (vector) retrieval results. RRF merges ranked lists without needing to normalize scores across different retrieval systems. For each document, its RRF score is: Σ 1/(k + rank_i), where rank_i is the document's position in retrieval system i and k is a constant (typically 60). Documents appearing high in multiple lists get high combined scores. RRF consistently outperforms linear score interpolation because it is scale-invariant—no tuning of inter-system score normalization is needed. Most vector databases (Weaviate, Elasticsearch, Qdrant) implement hybrid search with RRF.

Chunking splits documents into retrievable units. Naive fixed-size chunking (512 tokens, 50-token overlap) breaks semantic units arbitrarily. Better strategies include: (1) sentence/paragraph chunking respecting natural boundaries; (2) recursive character text splitting (LangChain's approach) that tries multiple separators in order; (3) semantic chunking—splitting at embedding similarity drops between sentences; (4) hierarchical chunking that creates parent-child relationships for context enrichment; (5) document-structure-aware chunking using markdown headers or HTML tags. Chunk size is a recall-precision tradeoff: smaller chunks have higher precision but lower context for the LLM; larger chunks provide more context but reduce precision.

ColBERT produces contextualized token-level embeddings for both query and document rather than a single pooled embedding. At retrieval time, relevance is computed via MaxSim: for each query token, the maximum cosine similarity to any document token is taken, then summed across all query tokens. This late interaction captures fine-grained token-level relevance while still allowing pre-computation of document token embeddings. ColBERT achieves cross-encoder-level quality at significantly faster inference by performing late interaction on compressed token vectors (128-dim). ColBERT-v2 adds residual compression for 6–10× index size reduction.

HyDE generates a hypothetical document that answers the query using an LLM (e.g., 'Write a passage that answers: What causes the Northern Lights?'), then uses the embedding of the generated passage rather than the original query for retrieval. This bridges the query-document vocabulary gap: queries are typically short and abstract while documents are long and specific. The generated hypothetical document uses document-like language patterns, making its embedding land closer to real relevant documents in the embedding space. HyDE works especially well for multi-hop and domain-specific queries where the query language differs significantly from document language.

Liu et al. (2023) showed that LLMs attend strongly to context at the beginning and end of the prompt, often missing information in the middle of long contexts. In RAG, if the most relevant chunk is placed in the middle of many retrieved documents, answer quality degrades significantly. Mitigations include: (1) re-ranking and placing the highest-ranked chunk first or last; (2) limiting retrieved context to 3–5 chunks rather than 20+; (3) using long-context models with improved middle attention (though the problem persists); (4) multi-hop retrieval to reduce individual context length. This finding argues for precision-first retrieval (fewer but better chunks).

Self-querying retrieval uses an LLM to parse a natural language query into both a semantic search query and structured metadata filters. For example, 'Python tutorials published after 2023 for beginners' is decomposed into: semantic query='Python tutorials for beginners', filter={year > 2023, difficulty='beginner'}. The retriever then applies both vector similarity search on the semantic part and metadata filtering on the structured part, implemented via vector database filter APIs (Pinecone's filter, Weaviate's where clause). This enables precise retrieval that neither pure BM25 nor pure semantic search supports efficiently.

FLARE (Jiang et al., 2023) generates text iteratively: the LLM generates a tentative next sentence; if any token in that sentence has low probability (below a threshold), FLARE treats this as an uncertainty signal and triggers retrieval using the tentative sentence as a query. Retrieved documents are prepended, and the sentence is regenerated with better grounding. This avoids retrieving at every step (wasteful) or only at the start (misses evolving information needs). FLARE implements active, need-based retrieval rather than static front-loaded retrieval, particularly useful for long-form generation tasks.

RAGAs (Retrieval Augmented Generation Assessment) evaluates four dimensions using LLM-as-judge: (1) Context Precision—are retrieved chunks relevant to the question? (2) Context Recall—do retrieved chunks cover all facts needed to answer? (3) Faithfulness—does the generated answer stay within retrieved context without hallucination? (4) Answer Relevancy—does the answer address the question? These are computed without human labels using a secondary LLM to judge. Additional metrics include Context Utilization and Answer Correctness (requires gold answers). RAGAs enables automated evaluation of retrieval and generation components separately.

Multi-vector retrieval generates multiple embeddings per document to capture different aspects. In ColBERT, each token has its own embedding. In ParentDocument retrieval, child chunks are embedded for retrieval while parent sections are returned for context. In Summary-based retrieval, a summary embedding is generated for each document alongside chunk embeddings. Multi-vector approaches improve recall for documents with multiple distinct topics and reduce the problem of single embeddings losing specifics of long documents. LangChain's MultiVectorRetriever stores child-to-parent mappings in a docstore for context enrichment.

Graph RAG (Microsoft Research, 2024) builds a knowledge graph from documents—extracting entities and relationships—and uses community detection (Leiden algorithm) to create hierarchical summaries of graph communities. At query time, relevant communities are retrieved and summarized hierarchically, enabling global reasoning across the entire document corpus rather than local chunk retrieval. Standard vector RAG retrieves top-k semantically similar chunks, which fails for questions requiring aggregation across many documents (e.g., 'What are the main themes across all emails?'). Graph RAG trades higher preprocessing cost for better global reasoning.

Anthropic's contextual retrieval (2024) prepends a short LLM-generated context summary to each chunk before embedding—the summary explains what the chunk covers relative to the whole document. For example, a financial report chunk about Q3 revenue might be prefixed with 'This chunk from Acme Corp's 2024 annual report describes Q3 revenue breakdown by region.' This context prevents retrieval failures caused by chunks being ambiguous without document-level context. Combined with BM25 hybrid retrieval and re-ranking, contextual retrieval reportedly reduces retrieval failure rates by ~67% vs. baseline naive chunking.

Query routing classifies incoming queries to select the appropriate retrieval source or strategy. A router (LLM-based or lightweight classifier) determines: which vector index to query (code, medical, legal), whether to use SQL for structured data vs. vector search for unstructured, or whether to answer from parametric memory vs. retrieval. LlamaIndex's RouterQueryEngine and LangChain's MultiRetrievalQAChain implement this. LLM-based routers classify via a prompt; embedding-based routers compare query embedding to index description embeddings. Routing reduces retrieval noise by querying only the relevant source.

Result caching stores exact query-response pairs and returns cached results only for identical or near-identical queries (exact string match or hash). Semantic caching stores query-response pairs with their embeddings and retrieves cached results for semantically similar queries above a cosine similarity threshold. GPTCache implements semantic caching: if a new query's embedding is within 0.85 cosine similarity to a cached query, the cached response is returned without LLM or retrieval calls. Semantic caching can cut API costs 30–60% for high-repetition workloads like customer support but risks stale or slightly incorrect responses for similar but distinct questions.

RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval, 2024) builds a tree of summaries over a document corpus. Leaf nodes are original text chunks; parent nodes are LLM-generated summaries of clusters of children (using Gaussian Mixture Model clustering in embedding space). Multiple levels of abstraction are built recursively. At query time, retrieval can happen at any tree level: detailed chunks for specific factual queries, high-level summaries for broad thematic queries. RAPTOR outperforms flat retrieval on multi-step and global reasoning questions on QASPER and QuALITY benchmarks.

Naive RAG retrieves once, passes context to the LLM, and generates. Agentic RAG uses an LLM agent that iteratively decides: should I retrieve more information, which query to use next, which source to query, and when the answer is sufficient. The agent uses tools (retriever, web search, calculator) and re-queries based on gaps in current knowledge. Frameworks include LangGraph, LlamaIndex's ReAct agent, and AutoGen. Self-RAG (Asai et al.) adds special reflection tokens to the LLM itself: [Retrieve], [Relevant], [Supported] tokens guide when and how to retrieve, intrinsically rather than via external orchestration.

Indexing pipeline optimization includes: (1) metadata extraction—attaching document-level metadata (title, author, date, section, page number, entity types) to each chunk for filtered search; (2) summary embedding—creating both detailed chunk embeddings and summary embeddings per document; (3) entity tagging—running NER to tag entities and create entity-based indexes; (4) relationship extraction for graph indexes; (5) de-duplication using MinHash LSH to remove near-duplicate chunks that waste index space and dilute retrieval. Rich metadata enables self-querying filters and reduces the search space, improving precision without sacrificing recall.

Embedding model quality determines semantic retrieval recall—the primary bottleneck in RAG. MTEB (Massive Text Embedding Benchmark) evaluates embedding models across 56+ tasks including retrieval, clustering, STS, and classification across multiple languages. Top performers in 2025 include text-embedding-3-large (OpenAI), e5-mistral-7b-instruct, GTE-Qwen2, and voyage-3 (Anthropic). Key factors: dimensionality (higher = better quality but more memory), max token length (longer = better for large chunks), language support, and MTEB Retrieval subset score. Domain-specific fine-tuning of embedding models can yield 5–20% gains on specialized corpora.

After bi-encoder retrieval returns top-k candidates (k=50–200), a cross-encoder scores each (query, passage) pair jointly via full cross-attention, providing highly accurate relevance scores for final re-ordering. The final k' results (k'=3–10) are fed to the LLM. Cross-encoder models: ms-marco-MiniLM-L6-v2 and ms-marco-MiniLM-L12-v2 (fast, small), bge-reranker-v2-m3 (multilingual, strong), Cohere Rerank 3.5 (API, top quality), Jina Reranker v2 (open-source, multilingual). Re-ranking typically improves RAG answer quality by 5–15% NDCG@10 at modest latency cost.

Context window stuffing (or 'full context RAG') loads as many retrieved chunks as possible into the LLM's full context window, relying on the LLM's attention to find relevant information. It works well for short answer tasks when the answer appears in one of the chunks and the model reliably finds it. It fails when: (1) the lost-in-the-middle effect hides the answer; (2) contradictory information across chunks confuses the model; (3) context is too long, causing latency and cost spikes; (4) the query requires synthesizing information from 20+ documents; (5) the model is prompted without re-ranking, loading noisy low-relevance chunks.

Multi-modal RAG uses joint embedding models (CLIP, ImageBind, voyage-multimodal) that embed images and text into the same vector space. Images are indexed alongside text chunks; retrieval uses cross-modal similarity. Strategies include: (1) late fusion—separate retrieval for text and image indexes, merged with RRF; (2) joint embedding retrieval—single query embedding retrieves from a unified index; (3) image captioning as proxy—generating text captions for images before indexing and embedding captions instead of raw images. The LLM then receives text chunks plus image file paths or base64-encoded images for visual context.

CRAG (Yan et al., 2024) adds a retrieval evaluator that scores retrieved chunks as Correct, Incorrect, or Ambiguous. If all retrieved chunks are marked Incorrect or Ambiguous, CRAG triggers web search to supplement or replace the retrieved context. If Correct, retrieved chunks are refined using knowledge decomposition (breaking chunks into fine-grained knowledge strips and filtering irrelevant strips). This corrective mechanism reduces the risk of generating answers from wrong retrieved context—a common failure mode. CRAG combines retrieval quality assessment with fallback web search for robust grounding.

Product quantization splits each d-dimensional vector into m sub-vectors of d/m dimensions, then quantizes each sub-vector to its nearest centroid in a trained codebook of k centroids. The full vector is represented as m centroid IDs, each requiring log₂(k) bits. For d=768, m=96, k=256: each vector uses 96 bytes instead of 768×4=3072 bytes—a 32× compression. Distance computation uses precomputed partial distance lookup tables for speed. FAISS implements PQ with variants (IVFPQ for combined clustering+quantization). PQ trades retrieval recall for memory efficiency, controlled by m and k.

MMR selects retrieved documents iteratively to maximize both relevance to the query and diversity among selected documents. The selection formula: MMRᵢ = arg max[λ·Sim(d, q) − (1−λ)·max_{s∈S} Sim(d, s)], where q is the query, S is the already-selected set, and λ balances relevance vs. diversity. MMR prevents returning multiple nearly-identical chunks (redundant context) which wastes context window and confuses the LLM. λ=0.5 is a common default. LangChain implements MMR via vector store search type parameter. Particularly valuable for long documents with repetitive content or multi-document corpora with near-duplicate sources.

Standard BM25 has a lower-bound issue: documents with zero occurrences of a query term receive a score of exactly zero for that term, making any term occurrence infinitely more valuable than none. BM25+ adds a small delta (typically δ=1) to the term frequency component, ensuring non-zero documents with rare query terms still receive proportionally positive scores. This improves retrieval for long documents where rare terms appear rarely but are still relevant. BM25+ outperforms BM25 particularly on verbose documents and specialized terminology queries.

Document-level embeddings represent the entire document as one vector (via mean pooling or CLS token), useful for routing or finding relevant documents broadly. Chunk-level embeddings represent fixed-size text segments, enabling precise localization of the relevant passage within a document. Most RAG systems use chunk-level because document-level embeddings fail to capture specific information in long documents—averaging over 50 paragraphs loses signal from any specific paragraph. Hybrid approaches store both: document embeddings for first-stage filtering, chunk embeddings for precise retrieval, parent documents for final LLM context.

For tabular data, standard text chunk retrieval is inadequate because table cells lack natural language context. Approaches: (1) Text-to-SQL—use an LLM to convert natural language queries to SQL, execute against the database, and return results; (2) Table-to-text conversion—serialize rows/aggregations as natural language before embedding; (3) Embedding table metadata (column names, descriptions) with examples for schema-aware retrieval; (4) Retrieval-augmented code execution—retrieve relevant tables and have the LLM generate pandas/SQL code. LlamaIndex's SQLTableRetriever and LangChain's SQLDatabaseChain implement text-to-SQL RAG.

Sentence window retrieval embeds individual sentences for high-precision retrieval (small, precise units are more likely to match specific queries) but returns an expanded window (±k surrounding sentences, typically k=2–3) to the LLM for sufficient context. The retrieval granularity and the context granularity are decoupled: sentences provide high-recall embeddings while windows provide readable context. LlamaIndex implements this via SentenceWindowNodeParser and MetadataReplacementPostProcessor. This approach outperforms fixed chunk retrieval on QA tasks where the answer is a single sentence but requires surrounding context for the LLM to formulate a response.

LLM-as-judge uses a powerful LLM (GPT-4, Claude) to score generated answers on criteria like factuality, completeness, and relevance, often compared against a gold answer or retrieved context. The judge is given a rubric and scores 1–5 or rates preferences. Failure modes include: (1) verbosity bias—longer answers score higher regardless of quality; (2) positivity bias—LLMs often prefer kindly worded answers; (3) self-preference—a model judging its own outputs gives higher scores; (4) format sensitivity—minor formatting changes affect scores; (5) insufficient calibration on domain-specific correctness. Mitigation: use chain-of-thought for reasoning, use multiple judges, calibrate with human labels.

KG-augmented RAG links entities in queries to a knowledge graph (Wikidata, domain KG) and retrieves structured facts about those entities alongside unstructured text chunks. Entity disambiguation maps surface forms ('Apple') to specific KG entities (Apple Inc. Q312 vs. apple fruit Q89) using entity linking (BLINK, ReFinED). Retrieved KG facts (Apple Inc. founded: 1976, CEO: Tim Cook) provide structured ground truth that complements text chunks. This reduces hallucination for factual queries about named entities and provides consistently formatted structured information that text chunks may lack.

Speculative RAG (2024) parallelizes drafting and retrieval: a small draft LLM generates a preliminary answer without retrieval while the retrieval pipeline runs concurrently. If the draft is high-confidence, it is used directly; otherwise the retriever's results are used to regenerate or verify the draft. Additionally, a small speculative retriever pre-fetches likely-needed documents during conversation turns before the user submits the final query, using predicted query embeddings. These approaches reduce the serial latency of retrieve-then-generate pipelines by overlapping computation.

Qdrant implements hybrid sparse-dense retrieval with native support for both vector types in a single collection. Dense vectors use HNSW indexing for approximate nearest neighbor search. Sparse vectors (supporting BM25, SPLADE, or custom sparse representations) use an inverted index on non-zero dimensions. For a hybrid query, both indexes are queried in parallel; results are merged using RRF or custom score weighting. SPLADE sparse vectors, trained to produce sparse BERT-like token activations, often outperform BM25 at lexical retrieval while remaining compatible with sparse vector infrastructure. Qdrant allows per-vector-field indexing configurations.

LLM prompt compression reduces the number of tokens sent to the LLM while preserving information relevant to the query. LLMLingua uses a small perplexity model to score and prune tokens from retrieved chunks based on their importance to the query context. LLMLingua-2 trains a compression model on distilled annotations. Selective Context compresses by removing less informative sentences. Compression ratios of 2–5× are achievable with minimal answer quality loss. This reduces both token costs (direct API savings) and latency (shorter prompts process faster), critical for high-throughput RAG deployments.

Conversational RAG augments retrieval with conversation history awareness. The main challenge: queries like 'What about their CEO?' are unresolvable without knowing 'their' refers to Apple from the previous turn. Solutions: (1) query rewriting—use an LLM to rewrite each turn into a standalone query incorporating chat history; (2) chat history embedding—encode the full conversation summary as part of the query embedding; (3) history-aware retriever—retrieve conditioned on both current query and chat history. LangChain's ConversationalRetrievalChain and LlamaIndex's CondensePlusContextChatEngine implement these patterns.

Freshness management strategies: (1) incremental indexing—detect changed or new documents via ETL pipelines (webhook triggers, CDC for databases, RSS feeds) and update only affected chunks; (2) TTL-based expiration—soft-delete chunks older than N days for rapidly changing domains; (3) timestamp metadata filtering—filter retrieval to documents within a recency window (e.g., past 30 days); (4) hybrid with web search—fall back to live web search for queries about recent events; (5) versioning—maintain document version history and filter by version. Pinecone's sparse updates and Weaviate's partial updates support efficient incremental reindexing.

SPLADE (Sparse Lexical and Expansion model) uses a BERT-based encoder with a log-RELU activation and regularization to produce sparse token-weight vectors. For a document, each vocabulary term receives a non-negative weight based on its relevance to the document's content. The model performs query/document expansion: terms not literally present but semantically related are assigned positive weights. This combines BM25's interpretability and inverted index compatibility with neural semantic expansion. SPLADE++ variants outperform BM25 by 10–15% on BEIR benchmarks while remaining compatible with traditional inverted index infrastructure.

Long document strategies: (1) hierarchical chunking with RAPTOR—build a tree of summaries; (2) semantic chunking—split at topic boundaries detected by embedding similarity drops; (3) document map-reduce—chunk, process each chunk independently with LLM, then aggregate; (4) structured extraction—use layout models (Unstructured.io, Azure Document Intelligence) to detect section headers, tables, and figures, preserving structure; (5) section-based retrieval—index at section granularity for routing, then at sentence level for precision; (6) late chunking (jina-embeddings-v3)—embed full document then pool sub-sequences for better contextual chunk embeddings.

The NIAH test inserts a specific fact ('the needle') at various positions within a long context window filled with unrelated text ('the haystack'), then asks the model to retrieve that specific fact. Results visualized as a 2D heatmap (context length × needle position) reveal where models fail. Most LLMs show degraded recall for needles placed in the middle of long contexts (the lost-in-the-middle effect), with performance also degrading as total context length increases. This benchmark motivates: (1) re-ranking with highest-relevance chunks at context edges; (2) using precision retrieval to reduce context length; (3) selecting models with stronger long-context attention patterns.

Late chunking (Günther et al., Jina AI 2024) runs the full document through a long-context embedding model to compute token embeddings with full document attention context, then pools these token embeddings within chunk boundaries to produce chunk embeddings. This contrasts with standard chunking which feeds each chunk to the model independently, losing cross-chunk context. Because token embeddings encode the full document context, chunk embeddings produced by pooling are contextually enriched—a sentence about 'the CEO' pools token embeddings that attended to the earlier mention of Apple, resolving coreference. Results: improved retrieval on multi-document benchmarks by 3–8%.

Production RAG SLAs include: (1) Retrieval latency P50/P95/P99—typically target <100ms for ANN search, <500ms with re-ranking; (2) End-to-end response latency—total time including LLM generation, target varies by use case (500ms for chat, 5s acceptable for research); (3) Retrieval recall@k—fraction of queries where the gold document is in top-k, measured offline; (4) Faithfulness/hallucination rate—sampled eval using RAGAs or custom judge; (5) Embedding throughput for indexing pipelines; (6) Vector index memory usage and QPS; (7) Cache hit rate for semantic caching. Tracing tools: LangSmith, Phoenix (Arize), Langfuse, Helicone.

Standard RAG concatenates all retrieved documents into one long prompt before the decoder, forcing the model to process all context jointly with quadratic attention cost. FiD (Izacard & Grave, 2021) processes each (question, retrieved document) pair independently through the encoder, producing separate encoder representations for each. All encoder outputs are then concatenated and cross-attended by a single decoder. This allows processing more documents (up to 100) than fits in a concatenated context, and the independent encoding makes computation linear in the number of documents. FiD outperforms RAG concatenation on Natural Questions and TriviaQA.

Namespace isolation partitions a vector index into logically separate spaces (namespaces in Pinecone, collections in Qdrant/Weaviate, namespaces in Weaviate multi-tenancy). Each tenant's documents are indexed in their own namespace; queries are scoped to the querying tenant's namespace, preventing cross-tenant data leakage. Physical isolation uses separate HNSW graphs per namespace (stronger but expensive); logical isolation uses metadata filtering on a shared index (cheaper but filtering must be enforced at application layer). For compliance-critical applications (healthcare, legal, finance), physical namespace isolation or separate collections per tenant is mandatory.

IRCoT (Trivedi et al., 2023) interleaves chain-of-thought reasoning with retrieval: the model generates one reasoning sentence, retrieves documents relevant to that sentence, adds them to context, generates the next reasoning sentence, retrieves again, and so on. This is superior to single-shot retrieval for multi-hop questions because: (1) intermediate reasoning steps generate more specific retrieval queries than the original broad question; (2) each retrieval step can discover documents that chain to the next sub-question; (3) the model builds a reasoning chain grounded in retrieved evidence at each step. IRCoT significantly outperforms one-shot RAG on MuSiQue and HotpotQA.

Code RAG requires specialized indexing: (1) AST-based chunking—split at function/class boundaries rather than token count; (2) code-specific embeddings—use code embedding models (code-embedding-ada-002, voyage-code-3, UniXcoder, CodeBERT) trained on code corpora; (3) symbol-level indexing—extract function signatures, docstrings, class hierarchies as metadata; (4) call graph enrichment—add function call relationships as graph edges; (5) BM25 over identifiers—code contains precise identifiers that benefit from exact matching. Tools like Sourcegraph Cody and GitHub Copilot Chat use AST-aware chunking with code-specific bi-encoders for repository-scale code search.

Query decomposition breaks complex queries into simpler sub-queries, retrieves for each sub-query independently, then combines results for the LLM. Applied when: (1) the query contains multiple distinct information needs; (2) multi-hop reasoning is required; (3) comparative questions need retrieving from multiple sources. Decomposition approaches: LLM-prompted decomposition (step-back prompting, least-to-most), ReAct-style decomposition, and Tree of Thoughts. Subquery results can be combined by: parallel retrieval + context concatenation, sequential retrieval where later queries are conditioned on earlier answers, or iterative like IRCoT. The overhead is higher latency and cost from multiple retrieval calls.

Streaming RAG requires careful pipeline design: (1) retrieve synchronously before starting generation—simplest, but adds retrieval latency to time-to-first-token; (2) speculative streaming—stream a retrieval-free draft while retrieval runs in parallel, then either continue with retrieved context if draft is good or regenerate; (3) two-phase streaming—send a 'Retrieving...' status token immediately, then stream LLM output after retrieval completes; (4) background retrieval with progressive enrichment—start generation from parametric knowledge while retrieval enriches later tokens. LangChain's streaming callbacks and LangGraph allow injecting retrieved context mid-generation for pipeline (4).

Open-domain QA (ODQA) answers questions from a large, unrestricted corpus (Wikipedia, the web) where the relevant document must be retrieved from millions of candidates—requiring high-recall retrieval. Closed-domain QA restricts answers to a specific, curated corpus (a company's documentation, a legal database) where the corpus is smaller but precision and domain coverage matter more. RAG systems for ODQA need scalable ANN indexes and aggressive recall optimization. Closed-domain RAG benefits from curated, high-quality chunking, metadata enrichment, and can afford slower re-ranking since the candidate set is smaller. Evaluation datasets: NQ, TriviaQA (open); SQuAD, CoQA (closed-ish).