Design: LLM Application

Ingest and chunk documents (300-800 tokens, semantic or recursive splitting), embed them, and store in a vector DB. At query time embed the question, retrieve top-k with hybrid (dense+BM25) search, rerank, then prompt the LLM with the retrieved context and a grounding instruction. Add citations, an 'I don't know' fallback when retrieval score is low, caching, and offline eval on a QA set. Log queries + retrieved chunks for monitoring and iteration.

Prefer semantic/recursive chunking that respects structure (headings, paragraphs, code blocks) over fixed-size splits, with 10-20% overlap to preserve context across boundaries. Chunk size trades recall vs precision: small chunks retrieve precisely but lose context; large chunks add noise and cost. Attach metadata (source, section, date) for filtering and citations. Test a few sizes against a labeled eval set rather than guessing.

Ground responses in retrieved context and instruct the model to answer only from it; return 'not found' when context is insufficient. Use lower temperature, add citations so answers are verifiable, and post-check claims against sources. Constrain outputs with schemas/function calling, add a verification pass or self-consistency for high-stakes answers, and evaluate faithfulness (does the answer follow from context) continuously.

RAG retrieves relevant private context at inference time without retraining, keeps data fresh, and lets you cite sources — usually the first choice before fine-tuning.

Separate retrieval and generation metrics. For retrieval: recall@k, MRR, and context precision (are retrieved chunks relevant). For generation: faithfulness (grounded in context), answer relevance, and correctness vs reference. Build a labeled QA set from real queries, use an LLM-as-judge with rubrics plus human spot-checks, and track metrics per change to catch regressions before shipping.

Fine-tune to change style, format, or teach a narrow skill/behavior the base model lacks, or to reduce prompt length and latency for a fixed task. RAG is better for injecting changing factual knowledge and citing sources. Often combine both: RAG for facts, light fine-tuning (or LoRA) for tone and task adherence. Fine-tuning needs quality labeled data and an eval harness to avoid regressions.

Stream tokens so users see output immediately; cache embeddings, retrievals, and full responses for repeated queries (semantic cache). Use a smaller/distilled model or route easy queries to it; shorten prompts and context; enable prompt caching for shared prefixes. Parallelize retrieval and pre-compute where possible; use speculative decoding and batching at the serving layer. Set token limits and early-stop.

Route by difficulty: cheap/small model for simple queries, large model only when needed. Cache responses and reuse prompt prefixes (prompt caching). Trim context to the minimum relevant tokens, cap max output tokens, and batch requests. Track cost per request and per feature, set budgets/quotas, and consider self-hosting open models at scale where volume justifies the ops cost.

Input filters (PII, prompt-injection, jailbreak detection), output filters (toxicity, policy, PII leakage), grounding checks, and a schema validator for structured outputs. Add rate limits, an allowlist of tools/actions, human-in-the-loop for high-risk actions, and refusal behavior for out-of-scope requests. Log everything for audit and red-team regularly.

Treat retrieved/user content as untrusted data, never as instructions: keep system instructions separate, and don't let tool outputs directly command the model. Sanitize and delimit external content, constrain tools with least privilege and confirmation for destructive actions, and add an output policy check. Detect injection patterns, and never expose secrets in prompts.

Define a tool registry with typed schemas; the model plans and emits structured tool calls (function calling), you execute them and feed results back in a loop until done. Add step limits and timeouts, validate arguments, apply least-privilege permissions, and require confirmation for irreversible actions. Log the trace for debugging, and evaluate task success end-to-end, not just single calls.

A semantic cache stores past query embeddings and responses; a new query that is embedding-similar (above a threshold) returns the cached answer, skipping the LLM call. It cuts latency and cost for repetitive or paraphrased queries (FAQs, support). Risks: stale answers and false-positive matches, so tune the similarity threshold, scope by user/context, and set TTLs.

Use RAG to retrieve only relevant chunks rather than stuffing everything. For summarization, use map-reduce or refine (summarize chunks, then combine). Consider hierarchical retrieval (summaries then details), long-context models when justified, and sliding windows with overlap. Always prioritize the most relevant/recent content and stay within token and cost budgets.

Hybrid search (dense + keyword/BM25) plus a cross-encoder reranker on the top candidates; query rewriting/expansion and HyDE for vague questions; metadata filtering (date, source, permissions); and better chunking. Add multi-query retrieval and deduplication. Measure each change against a retrieval eval set (recall@k, context precision) rather than assuming.

Attach owner/ACL metadata to every chunk and filter the vector search by the requesting user's permissions before retrieval — never rely on the LLM to withhold. Isolate tenants (namespaces/partitions), enforce row-level security in the store, and re-check access on the retrieved set. Log access for audit and test that users can't retrieve others' data.

Track latency (p50/p95), token usage/cost, error and timeout rates, cache hit rate, and retrieval scores. Capture user feedback (thumbs, edits), sampled traces (prompt, context, output), and quality metrics via periodic LLM-as-judge or human review. Alert on cost spikes, quality drops, and safety-filter triggers; log inputs/outputs for debugging and continuous eval.

The generation step: the prompt may under-instruct grounding, context may be truncated or poorly ordered (lost-in-the-middle), the model may over-rely on parametric knowledge, or chunks lack the needed detail. Fixes: stronger grounding instructions, rerank/curate context, place key info at start/end, lower temperature, and add a faithfulness check.

Diagnose whether the failure is retrieval or generation by inspecting retrieved chunks for sample queries — fix the dominant bottleneck rather than guessing.

Use the provider's structured-output/function-calling mode or constrained decoding with a JSON schema; validate the output and retry with the error message on failure. Give a clear schema and one example in the prompt, keep temperature low, and avoid free-text mixing. For critical fields, add server-side validation and defaults.

Define a primary metric (task success, deflection, satisfaction) and guardrail metrics (cost, latency, safety). Roll out to a small traffic slice, run offline eval first, then online with randomized assignment. Watch for regressions, use canary + shadow traffic for risky changes, and require both quality and cost/latency to pass before full rollout.

Using a strong LLM to score outputs against a rubric for scalable eval. Pitfalls: position/verbosity/self-preference bias, inconsistency, and being gamed. Mitigate with clear rubrics, pairwise comparison, randomized order, few-shot anchors, temperature 0, and human calibration on a sample. Never use it as the sole gate for high-stakes decisions.

Store prompts as versioned, reviewable artifacts (in git or a prompt registry) with metadata and eval scores; never hardcode-and-forget. Tie each version to eval results, support rollback, and A/B new versions. Separate templates from data, track which version served each request for reproducibility, and gate changes behind the eval harness.

Stream natural-language tokens for responsiveness, but buffer/parse structured segments (tool calls, JSON, citations) so partial structures aren't shown malformed. Emit citations after the grounded span is complete, and pause streaming while executing a tool then resume. Handle mid-stream errors gracefully and support cancellation.

Self-host when volume makes per-token API cost prohibitive, when data residency/privacy forbids sending data out, or when you need custom fine-tuning/latency control — at the cost of GPU ops, scaling, and reliability work. Use hosted APIs for speed to market, frontier quality, and low/variable volume. Many teams start on API and migrate hot paths to self-host.

Redact/mask PII on input, filter outputs for secrets and PII, and never place credentials in prompts. Scope retrieval by permissions, disable training on your data (or self-host), log access, and add DLP checks. For high-risk domains, add human review and strict tool permissions. Encrypt data at rest and in transit.

Keep a rolling window of recent turns plus a running summary of older context (summary buffer) to stay within tokens. Store durable facts/preferences in a separate memory store retrieved as needed. Include only relevant history, and for RAG re-retrieve per turn using the resolved query (coref-resolved). Cap memory size and expire stale entries.

A representative, labeled set of real user questions with gold answers and the source passages, covering easy/hard, in-scope/out-of-scope, and adversarial cases. Use it to measure retrieval recall and generation faithfulness/correctness, set release thresholds, and catch regressions on every change. Grow it from production failures.

Classify query difficulty/intent (heuristics or a small classifier), send simple/high-confidence queries to the cheap model and hard ones to the large model; optionally escalate on low confidence or failed validation. Cache aggressively. Monitor quality per route so cost savings don't degrade experience.

A small draft model proposes several tokens that the large model verifies in one pass, accepting the longest correct prefix — cutting latency with identical output distribution. It helps when a good draft model exists and decoding is the bottleneck, at the cost of extra memory and complexity.

Batch requests, parallelize with rate-limit-aware concurrency and retries/backoff, checkpoint progress for resumability, and dedupe identical inputs. Use the cheapest model that meets quality, cache, and validate outputs with schema + sampling. Track cost and throughput, and run a small quality audit before the full run.

Retrieve passages, instruct the model to cite the specific source for each claim, and return spans/links to the passages. Verify that cited text actually supports the claim (faithfulness check), and suppress uncited claims for high-stakes use. Show sources in the UI so users can verify.

Detect low retrieval relevance or intent mismatch and respond with a graceful fallback ('I don't have that information') rather than guessing. Offer to hand off to a human or a search link, and log these to expand coverage. Calibrate the threshold to trade coverage vs wrong answers.

Use a sandbox/staging with mock tools and seeded state; assert on the full trajectory (correct tools, arguments, order) and final state, not just text. Add adversarial and failure-injection cases, require confirmations for destructive steps, and gate production behind success-rate thresholds. Replay recorded traces for regression testing.

An approximate index like HNSW or IVF-PQ trades a little recall for large speed/memory gains at that scale; exact/flat search is too slow. Tune ef/nprobe for the recall-latency point, add quantization to fit memory, shard/replicate for throughput, and support metadata filtering.

Run incremental ingestion: detect changed/new/deleted docs (webhooks, change feeds, or hashes), re-chunk and re-embed only those, upsert by stable IDs, and remove deletes. Version content, backfill on schema changes, and monitor staleness. Avoid full rebuilds except for embedding-model upgrades.

Old and new embeddings are incompatible (different vector space), so you must re-embed the entire corpus and rebuild the index; queries must use the same model as the stored vectors. Plan a migration with dual-write/backfill and a cutover to avoid mixed spaces that silently degrade retrieval.

Route high-risk or low-confidence outputs to a review queue before they reach the user or take effect; capture reviewer decisions as labels to improve the system. Use confidence/guardrail signals to decide what needs review, keep latency acceptable with async or sampled review, and track reviewer agreement.

Trim boilerplate and redundant instructions, retrieve fewer but better chunks (rerank), compress context (summaries or extractive selection), reuse cached prefixes, and move stable instructions into a fine-tune. Measure quality on an eval set as you cut, since over-trimming context hurts grounding.

Business: deflection/containment rate, resolution rate, CSAT, and escalation rate. Quality: faithfulness/correctness and helpfulness. Ops: latency, cost per conversation, and safety-filter rate. Track these together — high deflection with low CSAT means it's dodging, not helping.

Add timeouts, retries with backoff, and a fallback model/provider (multi-provider routing). Serve cached answers when possible, degrade to a simpler flow or a 'try later' message, and use circuit breakers to shed load. Monitor provider health and keep the app functional without the LLM where feasible.

Detect query language, retrieve with multilingual embeddings (or translate query to the corpus language), and answer in the user's language with a grounding instruction. Keep source citations, evaluate per language (quality varies), and handle scripts/tokenization differences. Consider per-language indexes if quality demands.

Cost and latency rise with tokens, and 'lost-in-the-middle' means the model attends less to middle context, hurting quality — more context can lower accuracy. Retrieve and rerank to include only the most relevant passages, and order key info at the start/end.

Collect explicit (ratings, edits) and implicit (follow-ups, abandonment) feedback, mine failures into new eval cases and retrieval fixes, and periodically refresh prompts/fine-tunes. Close the loop: label hard cases, expand coverage, and re-evaluate. Treat production logs as your best data source.

Keep secrets server-side in a secrets manager, never in client apps or prompts; proxy all model calls through your backend. Rotate keys, scope them least-privilege, rate-limit and authenticate users, and monitor for abuse. Never log secrets or full prompts containing them.

When the relevant information is bounded and fits the window (a single contract, a codebase module, a meeting transcript) and you need holistic reasoning over all of it, long context avoids retrieval errors. RAG wins for large/changing corpora, citations, and cost. They combine: retrieve to fill a long window.

Cap steps/iterations and total tokens/time, detect repeated states/actions and break, require progress toward the goal, and add budgets with hard stops. Validate each tool result, and fall back to asking the user or returning best-effort when limits hit. Log traces to tune limits.

Jailbreak and prompt-injection attempts, data-exfiltration via tools/retrieval, PII/toxicity/policy violations, hallucination on adversarial questions, and over-broad tool permissions. Test unhappy paths and multilingual attacks. Fix with guardrails, least-privilege tools, and refusal behavior; re-test regularly.

Tune on an eval set: too small misses relevant context (low recall), too large adds noise, cost, and lost-in-the-middle. Common start is k=3-8 with a reranker over a larger candidate pool. Adjust by chunk size and answer type, and measure recall@k vs answer quality.

Client → API/backend (auth, rate-limit) → orchestration (query rewrite, retrieval, rerank, prompt build) → vector DB + metadata store for retrieval → LLM (with cache, guardrails, structured output) → response with citations. Side systems: ingestion pipeline, eval harness, logging/monitoring, and a feedback store. Keep secrets and permissions server-side.