RAG and vector databases

HNSW builds a multi-layer proximity graph that gives approximate nearest-neighbor search with sub-linear (roughly logarithmic) query complexity, versus O(N) for a flat scan. It trades a small amount of recall and higher memory/build cost for dramatically lower query latency at scale. Flat indexes remain exact and are preferable only for small datasets or when 100% recall is mandatory.

`ef_search` sets the size of the dynamic candidate list explored during search. Increasing it raises recall/accuracy but increases query latency; decreasing it speeds up queries at the cost of recall. It only affects query time, not the built graph, so it can be tuned per query without reindexing.

Cosine similarity normalizes for vector magnitude, comparing only direction, so documents are not unfairly favored merely because their embeddings have larger norms. Dot product conflates magnitude and direction, which can bias retrieval toward longer or higher-norm vectors. If embeddings are L2-normalized beforehand, dot product and cosine become equivalent (rank-wise).

IVF partitions vectors into clusters (Voronoi cells) so a query only searches the nearest few cells (controlled by `nprobe`), reducing the candidate set. PQ compresses each vector into compact codes, drastically cutting memory and speeding distance computation via lookup tables. Together (IVFPQ) they enable billion-scale search in RAM-constrained environments at the cost of some recall.

Hybrid search combines dense vector (semantic) retrieval with sparse lexical retrieval (e.g., BM25/keyword) and fuses the results, often via Reciprocal Rank Fusion. It captures both semantic meaning and exact term matches (names, codes, rare tokens) that pure embeddings can miss. This typically improves recall and robustness over either method alone.

RRF scores each document as the sum over result lists of 1/(k + rank), where rank is its position in each list and k is a constant (often 60). It needs no score normalization across heterogeneous retrievers, is robust to differing score scales, and rewards documents ranked highly by multiple systems. This makes it a simple, effective default for fusing hybrid search results.

def rrf(rank_lists, k=60):
    scores = {}
    for ranks in rank_lists:
        for r, doc in enumerate(ranks):
            scores[doc] = scores.get(doc, 0) + 1/(k + r + 1)
    return sorted(scores, key=scores.get, reverse=True)

A cross-encoder jointly encodes the query and document together, letting full attention model fine-grained interactions, which yields more accurate relevance scores. A bi-encoder encodes query and document independently into vectors, enabling fast precomputed ANN search but losing cross-term interaction. The cost is that cross-encoders cannot be precomputed and must run per query-document pair, so they are used only to rerank a small candidate set.

LLMs tend to attend most strongly to information at the beginning and end of their context window, with degraded recall of content placed in the middle. In RAG this means simply concatenating many retrieved chunks can bury the most relevant passage. Mitigations include reranking so the best chunks sit at the edges, limiting the number of chunks, and reordering retrieved passages.

Overlap (e.g., 10-20% of chunk size) ensures that a concept or sentence spanning a chunk boundary is not split such that neither chunk contains the full context. It improves recall for boundary-straddling information and preserves local coherence. The trade-off is increased storage and some duplicate content in retrieved results.

As dimensionality grows, distances between points concentrate, so the contrast between nearest and farthest neighbors shrinks, making exact NN less meaningful and harder to compute efficiently. This motivates ANN methods and dimensionality-aware index design. Modern embeddings (768-3072 dims) still work because they lie on lower-dimensional manifolds, but indexing must account for these effects.

MRL trains embeddings so that meaningful information is front-loaded into the leading dimensions, allowing a single embedding to be truncated to shorter lengths with graceful accuracy degradation. This lets you store/search shorter vectors for speed and memory savings, then optionally rerank with full-length vectors. It enables a single model to serve multiple dimensionality/cost trade-offs without retraining.

Quantization reduces each dimension's precision (e.g., float32 to int8 or 1-bit binary), shrinking memory and accelerating distance computation (binary enables Hamming distance). It can cut storage by 4x-32x with modest recall loss, and binary often pairs with a float rescoring step. The main risk is recall degradation if applied to embeddings not robust to precision loss, so rescoring/oversampling is commonly added.

Faithfulness measures whether the generated answer's claims are actually supported by the retrieved context, i.e., the absence of hallucination relative to the provided sources. It is distinct from answer correctness, which compares to a ground-truth answer. High faithfulness with low answer relevance means the model is grounded but off-topic; both must be evaluated.

Context recall measures whether all the information needed to answer (the ground-truth relevant content) was retrieved. Context precision measures whether the retrieved chunks are relevant and well-ranked, i.e., the signal-to-noise ratio of the context. You can have high recall but low precision (relevant info present but buried in noise), which harms generation.

HyDE uses an LLM to generate a hypothetical answer/document for the query, then embeds that hypothetical document to perform the vector search instead of (or alongside) the raw query. This bridges the lexical/semantic gap between short questions and the longer answer-style passages in the corpus, often improving zero-shot retrieval. The risk is propagating LLM hallucinations into retrieval, so it is used carefully.

The system uses an LLM to rewrite the original query into several diverse paraphrases or sub-questions, retrieves for each, and merges the results. This counters the brittleness of single-query embeddings to phrasing and covers multiple facets of a complex question. The trade-off is added LLM latency/cost and potential noise, usually managed with deduplication and reranking.

You embed and search over small, precise child chunks (good for matching), but return the larger parent chunk or surrounding context to the LLM for richer answers. This decouples retrieval granularity from generation context, improving both match precision and answer completeness. It avoids the dilemma of choosing one chunk size that is simultaneously precise enough to retrieve and large enough to answer.

DiskANN-style indexes keep most of the graph/vectors on SSD and only a compressed representation in RAM, enabling billion-scale search on a single machine at far lower memory cost. The trade-off is higher per-query latency due to disk I/O, mitigated by SSD-friendly access patterns and in-memory quantized vectors for candidate generation. It is chosen when datasets exceed economical RAM capacity.

They tell the index which distance metric to optimize for: cosine distance, Euclidean (L2) distance, and (negative) inner product respectively, matching the `<=>`, `<->`, and `<#>` operators. The index only accelerates queries that use the matching operator/metric. You must align the operator class with the metric your embedding model was trained for (often cosine or inner product).

CREATE INDEX ON items
USING hnsw (embedding vector_cosine_ops);
-- query with matching operator
SELECT id FROM items ORDER BY embedding <=> $1 LIMIT 10;

Metadata filtering restricts results by attributes (e.g., tenant, date, doc type) alongside vector similarity. Post-filtering runs ANN first then drops non-matching results, which can return too few items if the filter is selective; pre-filtering restricts the candidate set before/within the ANN search, preserving result count but is harder to implement efficiently on graph indexes. Modern engines use filtered-HNSW or hybrid approaches to balance correctness and speed.

Separate namespaces give hard data isolation, prevent cross-tenant leakage, allow per-tenant index tuning and deletion, and avoid the recall/performance pitfalls of highly selective metadata filters on a shared graph index. A single shared index with filtering scales better for huge numbers of tiny tenants but risks noisy-neighbor effects and filtering inefficiency. The choice depends on tenant count, size distribution, and isolation requirements.

`M` is the maximum number of bidirectional links (neighbors) per node in the graph layers. Higher `M` increases recall and connectivity but also raises memory consumption and build time; lower `M` is leaner but can hurt recall, especially in high dimensions. It is a build-time parameter, so changing it requires reindexing.

Semantic chunking splits text at points where the meaning shifts, typically by detecting drops in embedding similarity between consecutive sentences, so each chunk is topically coherent. Fixed-size chunking splits by token/character count regardless of meaning, which can cut mid-thought. Semantic chunking can improve retrieval relevance but is more expensive to compute and can produce variable chunk sizes.

Dimensionality controls representational capacity, but if the model was never trained on the domain's vocabulary/semantics (e.g., legal, biomedical, code), it maps domain terms poorly regardless of size. Retrieval quality depends on whether the model's learned semantic space separates relevant from irrelevant content for that domain. The fix is a domain-adapted or fine-tuned embedding model, not merely more dimensions.

With compressed (e.g., int8/binary) vectors, you retrieve more candidates than needed (oversample) using the fast approximate distance, then recompute exact full-precision distances on that shortlist to reorder. This recovers most of the recall lost to quantization while keeping the bulk of the search cheap. It is a standard pattern in binary quantization pipelines.

Vector similarity is only meaningful within a single shared embedding space; queries and documents embedded by different models (or versions) live in incompatible spaces, so distances become meaningless. Changing the model requires re-embedding the entire corpus. This is why embedding model upgrades are operationally costly and must be version-tracked.

Before embedding, each chunk is prepended with a short LLM-generated context describing how it fits within the whole document (e.g., section, entities, timeframe). This disambiguates chunks that are uninterpretable in isolation, improving both embedding and BM25 matching. Anthropic reported it substantially reduces retrieval failures, especially when combined with hybrid search and reranking.

A large top-k injects more irrelevant or marginally relevant chunks, increasing noise, token cost, and latency, and can trigger 'lost in the middle' or distract the model into hallucinating from off-topic context. It can also dilute precision even if recall improves. The right top-k balances recall against context-window noise, often paired with a reranker to filter.

ColBERT encodes each token of the query and document into separate vectors and computes relevance via a MaxSim operation (sum of max token-level similarities) at query time, capturing fine-grained term interactions. Single-vector dense retrieval compresses an entire passage into one vector, losing token-level detail. Late interaction improves accuracy but increases storage (many vectors per doc) and requires specialized indexing.

Embedding similarity scores are typically uncalibrated and model-specific; the same cosine value means different things across models and even across queries due to anisotropy in the embedding space. Absolute thresholds (e.g., 'accept if > 0.8') are brittle and don't transfer. Better practice is relative ranking, per-model calibration, or training a classifier/reranker for accept/reject decisions.

Anisotropy is when embeddings occupy a narrow cone in the vector space rather than being uniformly distributed, so even unrelated texts have high cosine similarity, reducing discriminative power. Mitigations include whitening/normalization, contrastive training objectives, and post-processing techniques; modern contrastively-trained embedding models exhibit much less anisotropy than early transformer mean-pooled embeddings. It matters because it compresses the usable similarity range.

It uses an LLM to parse a natural-language query into a structured query, extracting metadata filters (e.g., date ranges, categories) and the semantic search string, and/or to route the query to the appropriate index/data source. This enables combining structured constraints with semantic search and directing queries to the right corpus. It improves precision for queries that mix factual filters with semantic intent.

When source documents are updated or deleted, the vector index can serve outdated or orphaned embeddings, leading to wrong answers and citing nonexistent content. It's addressed with change-data-capture pipelines, upsert/delete by stable document IDs, versioned chunks, and periodic reconciliation between source-of-truth and index. Soft-deletes plus TTLs and re-embedding on content change are common patterns.

A single vector for a long document averages/compresses many distinct topics into one point, diluting any specific concept and hurting both retrieval precision (the vector matches everything weakly) and the ability to return focused context. Most embedding models also truncate beyond a max token limit, silently dropping content. Chunking into coherent units gives sharper, retrievable representations.

Retrieval recall@k is the fraction of queries for which a relevant document appears in the top-k retrieved results (or the fraction of relevant docs found within top-k). Classification recall is true positives over all actual positives at a decision threshold. Retrieval recall@k explicitly depends on the cutoff k and ranking, whereas classification recall depends on a threshold, not a ranked list size.

GraphRAG builds a knowledge graph (entities and relationships, often with community summaries) from the corpus and uses graph traversal/summarization to answer queries. It excels at global, multi-hop, or 'connect-the-dots' questions that require synthesizing information across many documents, which flat top-k vector retrieval handles poorly. The trade-off is higher indexing cost and complexity, so it complements rather than replaces vector RAG.

Vector data has no natural total order, and semantically similar vectors must be discoverable regardless of shard, so each query typically fans out to all shards and merges top-k results. Random/round-robin distribution balances load evenly across shards; range-based sharding on an arbitrary key gives no locality benefit and risks skew. The scatter-gather pattern means recall and latency depend on per-shard search plus a merge step.

Embedding caching avoids recomputing vectors for unchanged text (documents or repeated queries), saving model inference cost. Semantic caching returns a stored answer when a new query is sufficiently similar (by embedding distance) to a previously answered one, cutting both retrieval and generation cost/latency. The risk with semantic caching is serving a stale or subtly mismatched answer, so similarity thresholds and TTLs must be tuned.

A post-retrieval filter discards chunks that are not actually relevant or entailing for the query, reducing noise fed to the generator and curbing hallucination from distracting context. It also enables fallbacks like 'no answer found' or triggering web search when retrieved context is insufficient. This is a core component of corrective/self-reflective RAG (e.g., CRAG, Self-RAG) patterns.

A bad final answer can stem from either failed retrieval (the right context was never fetched) or failed generation (good context but the model misused it); a single end-to-end score cannot localize the fault. Measuring retrieval (context recall/precision) and generation (faithfulness, answer relevance) separately lets you target the actual bottleneck. This component-wise diagnosis is critical for efficient iteration and debugging.