Cost Optimization

The three main cost drivers are: (1) Input tokens—all tokens sent in the prompt including system prompt, conversation history, retrieved context, and tool definitions, billed per million tokens (e.g., $3/M for Claude 3.5 Sonnet); (2) Output tokens—generated tokens, typically 3–5× more expensive per token than input because they require sequential decoding; (3) Request overhead—some providers charge per-request fees or have minimum billing increments. For API optimization, output tokens often dominate costs for long-form generation tasks, while input tokens dominate for high-throughput RAG pipelines with large retrieved contexts.

Prompt caching (Anthropic, OpenAI) stores the KV-cache state of a long prompt prefix on the server and reuses it for subsequent requests with the same prefix. Cached input tokens cost 90% less (Anthropic) or similar discounts. Requirements: (1) the cached prefix must be identical byte-for-byte; (2) the prefix must exceed a minimum length (1024 tokens for Anthropic, 128 tokens for OpenAI); (3) the cache must be declared explicitly with cache_control markers; (4) cache TTL is typically 5 minutes (Anthropic) or session-scoped. Best for: long static system prompts, repeated RAG document sets, multi-turn conversations, and tool definition schemas.

# Anthropic cache_control example
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": long_system_prompt,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_message}]
)

Model cascading routes queries to increasingly capable (and expensive) models based on difficulty. A small, cheap model handles simple queries first; if confidence is low or the task is flagged as complex, the query escalates to a larger model. Example: GPT-4o mini handles 80% of customer support queries; complex technical questions escalate to GPT-4o (5–10× more expensive). Routing decisions use: perplexity of the small model's response, task complexity classifiers, keyword-based routing, or small model self-assessed confidence. FrugalGPT (Chen et al., 2023) demonstrated 50–90% cost reduction with equal or better quality using learned routing policies.

Quantization reduces model weight precision: FP32→FP16 (2× memory reduction), FP16→INT8 (2×), INT8→INT4 (2×). This reduces VRAM requirements, allows larger batch sizes, and speeds up matrix multiplications. GPTQ (post-training quantization for transformers) and AWQ (activation-aware weight quantization) are standard for LLMs. INT4 quantization of a 70B model reduces VRAM from ~140GB (BF16) to ~35GB, enabling single A100 deployment. Quality tradeoffs: INT8 loses <1% accuracy on most tasks; INT4 loses 1–3% on reasoning benchmarks; QA benchmarks are more sensitive than factual recall. GGUF format enables CPU inference with mixed quantization.

Speculative decoding uses a small draft model to generate N tokens speculatively, then a large target model verifies all N tokens in parallel (one forward pass instead of N sequential passes). If the target model agrees with the draft's tokens (measured via rejection sampling), they are accepted; mismatched tokens are corrected. Because the target model's forward pass cost is nearly constant regardless of tokens verified, this yields 2–4× throughput improvement with mathematically identical output distribution (the acceptance algorithm ensures equivalence). Best speedup occurs when draft model acceptance rate is high—i.e., for simpler, predictable text. Implemented in vLLM, TensorRT-LLM, and HuggingFace.

Batching processes multiple requests simultaneously through the model's GPU kernels, amortizing fixed overhead costs. Naïve static batching groups requests of the same length, causing padding waste and idle compute when requests finish at different times. Continuous batching (Orca, implemented in vLLM) dynamically inserts new requests into running batches as slots free up after sequence completion. This eliminates wait time from static batching and achieves near-maximum GPU utilization. Continuous batching enables 10–23× higher throughput than static batching at comparable latency, critical for cost-per-request in high-traffic deployments.

System prompt optimization reduces the token count of system prompts while preserving instruction quality. Techniques: (1) Prompt compression—use LLMLingua or manual editing to remove redundant words, replacing verbose explanations with concise instructions; (2) Structured prompts—use JSON or markdown structure over verbose prose; (3) Remove examples when model is capable without them; (4) Move rarely-needed instructions to user turns only when relevant; (5) Use abbreviations or codenames that get expanded by a local preprocessing step; (6) Serve instructions from a RAG tool rather than always in context. A well-optimized system prompt can reduce from 2000 to 400 tokens—saving $0.005 per request at $3/M, or $50/day at 10K requests/day.

Output tokens are 3–5× more expensive per token than input on most providers and dominate costs for generation-heavy tasks. Control strategies: (1) Explicit length instructions—'Answer in exactly 3 bullet points', 'Respond in under 150 words'; (2) max_tokens parameter—hard cap on generation length; (3) Stop sequences—define tokens/phrases that terminate generation early when task completion is detected; (4) Format constraints—structured output (JSON) produces shorter, denser responses than prose; (5) Task decomposition—split 'write a 2000-word essay' into multiple shorter steps to enable early stopping if only part is needed; (6) Confidence-based truncation—stop when model reaches a conclusion marker. Each 1000 token reduction at $15/M output saves $0.015 per call.

Multi-turn chat sends the full conversation history with each API call. A 20-turn conversation with 200 tokens/turn generates 4000 tokens of history—each turn pays for all preceding turns as input. Strategies to manage cost: (1) Sliding window—include only the last N turns (loses older context); (2) Summary compression—periodically summarize older turns into a shorter summary using a cheap model; (3) Selective history—include only turns relevant to the current query using semantic similarity; (4) Prompt caching—cache stable early conversation turns to pay discounted rates; (5) Entity extraction—replace verbose context with structured entity state. Without management, costs grow quadratically with conversation length.

Knowledge distillation trains a small student model to mimic a large teacher model's outputs. For LLMs: (1) token-level distillation—minimize KL divergence between student and teacher probability distributions over the vocabulary; (2) feature-level distillation—align intermediate layer representations; (3) sequence-level distillation—fine-tune student on teacher-generated synthetic data (GPT-4 generated answers used to train smaller models like Alpaca, Vicuna). Student models (7B–13B) can reach 80–90% of teacher (GPT-4 70B+) performance on specific domains at 10–50× lower inference cost. Task-specific distillation on narrow domains achieves better cost-quality ratios than general-purpose student models.

FlashAttention (Dao et al., 2022) reimplements attention to be IO-aware: rather than materializing the full N×N attention matrix in GPU HBM (high-bandwidth memory), it computes attention in tiles that fit in SRAM (shared memory). This reduces HBM reads/writes from O(N²) to O(N), dramatically reducing memory bandwidth requirements. FlashAttention is 2–4× faster than naive attention for long sequences and enables 4–8× longer context within the same VRAM budget. FlashAttention-2 and -3 further optimize for multi-GPU parallelism and new hardware (H100 tensor cores). It is now standard in virtually all transformer training and inference frameworks.

MoE models replace dense FFN layers with N expert FFNs, routing each token to the top-k experts (typically k=2) via a learned router. Total parameters may be 8×–16× the dense baseline, but each token activates only k/N parameters. Mistral 8×7B (46.7B total params) activates ~12.9B per token—comparable to a 13B dense model in FLOPs but with 46.7B knowledge capacity. Cost implications: training is expensive (all expert gradients needed); inference costs are similar to a dense model of the 'activated' size. MoE models achieve better quality-per-FLOPs than dense models. Challenge: all expert weights must fit in VRAM, requiring large GPU clusters for big MoE models.

Semantic caching stores previous LLM responses indexed by query embeddings. For new queries, if cosine similarity to any cached query exceeds a threshold (e.g., 0.92), the cached response is returned without an API call. GPTCache and Redis with vector similarity support this pattern. Cost savings depend on query repetition rate: customer support systems with high query repetition may cache 30–60% of requests, reducing API costs proportionally. Implementation requires: (1) embedding model for query vectorization; (2) vector store for similarity search; (3) threshold tuning (too low → wrong answers returned; too high → low cache hit rate); (4) TTL management for time-sensitive information.

Throughput optimization maximizes tokens/second processed across all requests—critical for batch processing, content generation pipelines, and cost-per-token minimization. Techniques: large batch sizes, continuous batching, model parallelism, high-end GPUs (H100), tensor parallelism across multiple GPUs. Latency optimization minimizes time-to-first-token and total response time for individual requests—critical for interactive applications. Techniques: small batch sizes (or batch size 1), speculative decoding, INT4 quantization (faster decode), fewer sampling steps, server colocation with users (edge deployment). These goals conflict: large batches improve throughput but increase queueing latency. Serving infrastructure must tune for the application's latency-throughput tradeoff.

LoRA (Low-Rank Adaptation) freezes all pretrained weights and trains only small rank-decomposition matrices A (d×r) and B (r×k) per target layer, where r << min(d,k). For a 7B model with full fine-tuning: all ~7B parameters require gradients and optimizer states (Adam: 3× model size = ~84GB for fp16+adam). LoRA at rank 16 on attention layers: ~50M trainable parameters, ~600MB for weights + optimizer states. Training VRAM drops from 80+ GB to 16–24GB for 7B models, enabling single A100/A10G fine-tuning. Training time drops proportionally. QLoRA additionally quantizes the frozen model to 4-bit, fitting 7B fine-tuning in 12GB VRAM.

A typical RAG pipeline token budget: system prompt (200–1000 tokens), conversation history (500–2000 tokens), retrieved context (500–3000 tokens per chunk × 3–5 chunks = 1500–15000 tokens), user query (10–200 tokens), generated output (100–500 tokens). Total: 2000–17000+ tokens per call. Optimization: (1) cap retrieved context to 3–5 chunks maximum; (2) compress system prompt; (3) use prompt caching for stable system prompt + documents; (4) chunk more precisely so less irrelevant text is retrieved; (5) use LLMLingua to compress retrieved chunks 2–4×; (6) manage conversation history with rolling summaries; (7) use a cheaper model for RAG and escalate to expensive model only for complex synthesis.

Tensor parallelism splits individual layer computations across multiple GPUs—each GPU holds a shard of each weight matrix and computes a portion of each matmul, with results aggregated via all-reduce. For a 70B model requiring 4× A100 (80GB) with tensor parallel degree 4: latency per token is similar to a 1-GPU system (parallelism hides computation time) but hardware cost is 4× per request. Pipeline parallelism (different layers on different GPUs) has higher throughput but adds inter-GPU pipeline bubbles. The cost-optimal deployment depends on batch size: high-throughput batch processing prefers fewer, larger GPUs per model; low-latency requires parallelism to fit the model at acceptable latency.

Long-context models (1M token Gemini 1.5 Pro) can ingest entire document corpora at once, eliminating RAG infrastructure complexity. Cost comparison: Gemini 1.5 Pro at $7/M input tokens for a 100K-token document corpus = $0.70 per query. A RAG pipeline retrieves ~2000 tokens of relevant context: at $3/M input = $0.006 per query—117× cheaper per call. For infrequent, broad synthesis queries over large corpora, long context may be cost-justified. For high-frequency specific queries, RAG dramatically outperforms on cost. Hybrid approaches use RAG for most queries and long-context only for complex synthesis tasks where the full corpus is needed.

Spot (AWS) or preemptible (GCP) GPU instances run at 60–90% discount vs. on-demand pricing but can be reclaimed by the provider with minimal notice. For LLM training: use checkpointing every 5–15 minutes so a preemption loses at most a few minutes of progress; restart training from checkpoint on a new spot instance. For batch inference: tasks are inherently retryable if the result is not needed in real-time. Using spot H100s ($4–6/hr vs. $12–15/hr on-demand) reduces training costs 60–80%. Not suitable for low-latency serving (preemptions cause availability drops). Reserved instances (1–3 year commitments) save 30–40% for steady-state inference serving.

Chinchilla (Hoffmann et al., 2022) found that previous LLMs were undertrained relative to their size. The compute-optimal ratio is approximately 20 tokens of training data per model parameter. For a 7B model: optimal = 140B tokens; for 70B: 1.4T tokens. Training larger models on fewer tokens (GPT-3 approach) or smaller models on more tokens yields suboptimal cost-efficiency. Practically: a 7B model trained on 1T+ tokens (LLaMA 2) outperforms a 30B model trained on 300B tokens at the same compute budget. For cost-efficient deployment: a well-trained small model is cheaper to serve at equivalent quality to a larger undertrained model.

For deterministic or nearly-deterministic queries (temperature=0, fixed prompt), the output is always the same. Exact output caching stores query hash → response in a key-value store (Redis, Memcached) and returns cached responses without API calls. Use cases: document summarization (same document → same summary), code explanation, product description generation for a fixed catalog. TTL should be set based on how frequently the underlying information changes. Unlike semantic caching (approximate match), exact caching requires perfect prompt identity. Cost impact: 100% API call elimination for cache-hit queries. Combined with semantic caching, a well-designed cache can handle 40–70% of requests without API calls.

PagedAttention (vLLM) manages KV-cache memory using virtual memory paging inspired by OS memory management. Instead of pre-allocating a contiguous block for each sequence's full maximum length, it allocates memory in fixed-size pages (blocks) that can be non-contiguous and shared across sequences. Benefits: (1) eliminates internal fragmentation from pre-allocation; (2) enables copy-on-write sharing of KV-cache for parallel sampling and beam search; (3) achieves 2–4× better GPU memory utilization vs. contiguous allocation; (4) allows batch sizes 4–8× larger. Larger batch sizes directly reduce cost per token by amortizing fixed GPU overhead across more requests.

Cost monitoring architecture: (1) token tracking—log input/output token counts per request with model/endpoint metadata; (2) cost calculation—multiply tokens by current model pricing, tracking per feature/user/team; (3) aggregation—store in time-series database (InfluxDB, CloudWatch Metrics) with dimensions for model, user_id, feature_name; (4) alerting—set budget thresholds with alerts when daily/monthly spend exceeds targets; (5) anomaly detection—alert on sudden cost spikes (e.g., runaway prompt loops); (6) cost attribution—attribute costs to product features for ROI analysis. Tools: LangSmith, Helicone, LiteLLM's cost tracking, custom middleware logging. Monthly reporting enables identifying high-cost features for optimization.

API provider costs (2025): GPT-4o ~$5/M input, $15/M output; Claude 3.5 Sonnet ~$3/M input, $15/M output; Gemini 1.5 Pro ~$7/M input, $21/M output. Self-hosted Llama 3.1 70B on an A100 80GB GPU: ~$3/hr for cloud instance handling ~150K tokens/hr = $0.02/K tokens = $20/M tokens (all-inclusive, but you're paying for capacity whether used or not). Self-hosting becomes cost-competitive at high utilization (>60%+ GPU occupancy) and for models competitive with GPT-4 class. Smaller models (8B–13B) on A10G ($1.5/hr, ~500K tokens/hr) cost ~$3/M tokens—competitive with API pricing with full data privacy.

Early exit methods add classifiers at intermediate transformer layers to predict when sufficient confidence is reached to output an answer without processing remaining layers. If the intermediate classifier's confidence exceeds a threshold (e.g., 0.95), computation stops early, saving the FLOPs of remaining layers. DeeBERT and PABEE apply this to BERT for classification; LLM Surgeons and SkipDecode extend to generative models. Speedup varies by task difficulty: simple queries may exit at layer 6 of 32 (80% computation saved); hard queries exit at the final layer. Average speedup: 1.5–3× for mixed workloads with 95–99% accuracy retention relative to full model.

Tool definitions (JSON Schema) in function calling consume input tokens: a single tool schema is typically 50–200 tokens; 10 tools = 500–2000 tokens prepended to every request. At $3/M input, 1000 tokens of tool definitions for 100K daily requests costs $30/day. Minimization strategies: (1) use minimal tool schemas—concise descriptions, remove unnecessary parameter descriptions; (2) only include tools relevant to the current task (dynamic tool selection); (3) use tool routing—pre-classify the query and include only the subset of tools applicable; (4) batch tool calls where possible to reduce round trips; (5) cache tool definition tokens with prompt caching. Streamlining 10 tools from 1500 to 500 tokens saves ~$20/day at 100K requests.

In transformer inference, the prefill phase processes the full prompt in parallel (high GPU utilization), while decode phase generates tokens one at a time (low GPU utilization—memory-bandwidth bound). Split prefill-decode architectures assign prefill to one GPU cluster and decode to another, optimizing each independently. Disaggregated serving (DistServe, Splitwise) separates these phases: prefill pods process prompts at full throughput; decode pods maintain KV-cache and generate tokens. This prevents decode's low-utilization tokens from blocking prefill batches, improving overall system throughput 2–3× for typical chat workloads at the cost of network transfer of KV-cache between pods.

Model A/B testing routes a percentage of production traffic to a new (cheaper) model while keeping the existing model as control. Implementation: (1) shadow mode—send requests to both models, compare outputs offline (no user impact); (2) canary deployment—route 5–10% of traffic to new model, monitor quality metrics and error rates; (3) user cohort assignment—consistently route specific users to the test model for longitudinal quality assessment; (4) automated evaluation—use LLM-as-judge to compare responses between models at scale. Metrics to monitor: task completion rate, user satisfaction signals, error rates, token costs. A 2-week A/B test on 10% traffic provides statistical significance for most quality metrics before full migration.

Temperature, top-p, and top-k parameters affect the number of output tokens generated (via stopping behavior) but not the per-token compute cost. Higher temperature produces more diverse outputs that may require more tokens to complete thoughts; temperature 0 (greedy) produces the most predictable, often shorter outputs. Sampling with top-p or top-k adds marginal overhead for vocabulary filtering but is negligible in practice. The main indirect cost impact: temperature 0 enables exact output caching (deterministic responses); higher temperatures prevent caching. For high-volume applications, using temperature 0 where output variety isn't required enables output caching and 100% cost elimination for repeated queries.

GPU inference is 10–100× faster than CPU for large models due to high memory bandwidth and parallel compute. However, CPUs are cheaper per hour and require no specialized hardware allocation. CPU inference is viable for: (1) small models (<7B parameters) with acceptable latency (1–5 tokens/sec); (2) batch processing with no latency requirement; (3) edge deployment where GPU is unavailable; (4) development and testing. llama.cpp enables efficient quantized CPU inference (INT4): a 7B Q4 model generates ~10 tokens/sec on modern CPUs, ~80 tokens/sec on A10G GPU. Cost comparison: a powerful CPU server ($0.50/hr, 10 tokens/sec) vs. A10G ($1.50/hr, 80 tokens/sec)—GPU is 3× cost but 8× faster, making GPU cheaper per token for high-throughput use.

LiteLLM provides a unified OpenAI-compatible interface to 100+ LLM providers (Anthropic, Google, Azure, AWS Bedrock, Together, Groq, etc.). Cost optimization features: (1) lowest-cost routing—automatically route to the cheapest provider meeting latency SLA; (2) fallback routing—if primary provider fails or rate-limits, fall back to secondary; (3) load balancing across API keys and providers; (4) cost tracking with per-provider breakdown; (5) budget limits per user/team with automatic blocking at threshold. Centralized proxy deployment enables token cost logging, caching, and routing decisions without changing application code. Particularly valuable for multi-cloud AI deployments.

A GPT-4 class model's general capability is only fully needed for general tasks. For a specific task (sentiment classification, entity extraction, SQL generation, email writing), fine-tuning a 7B model on ~1000–10000 task-specific examples (generated with GPT-4 or human-labeled) often matches GPT-4 performance on that task at 50–200× lower inference cost. Process: (1) collect task data (human or synthetic); (2) fine-tune Llama/Mistral/Phi with LoRA or full fine-tuning; (3) A/B test against GPT-4 on task-specific metrics; (4) if quality matches, migrate production traffic. GPT-4 is used only for data generation and initial benchmarking, not for serving.

Standard multi-head attention has separate K and V projections for each of H heads. KV-cache stores K and V for all H heads: O(H × seq_len × d_head) memory. GQA (Ainslie et al., 2023) groups multiple query heads to share the same K and V head. With G groups (G < H), KV-cache reduces to O(G × seq_len × d_head). GQA with G=8 reduces KV-cache by H/G× (e.g., 32 query heads / 8 groups = 4× KV reduction). Smaller KV-cache means: larger batch sizes, less memory bandwidth during decode, lower VRAM requirements. Llama 2 70B and Mistral use GQA; it is now standard in modern LLM architectures.

TCO components: (1) Inference compute—GPU hours × utilization × cost/hr (on-demand vs. reserved vs. spot); (2) API costs—if using external providers, tokens × rate; (3) Storage—model weights (70B = ~140GB), vector database (dimension × doc count × 4 bytes), logs; (4) Networking—inter-GPU bandwidth for tensor parallelism, data transfer costs; (5) Engineering—setup, monitoring, maintenance FTE cost; (6) Evaluation—LLM judge API calls for quality monitoring; (7) Redundancy—N+1 GPU allocation for HA. For 10M tokens/day at GPT-4o pricing ($5/M input, $15/M output, 50/50 mix): ~$100K/month. Equivalent self-hosted 70B on H100s at 70% utilization: ~$30K/month GPU cost + $50K/yr engineering overhead.

Model merging combines multiple fine-tuned LoRA adapters or full models into one model that retains capabilities from all merged models. TIES-Merging, DARE-TIES, and Model Soup average weights with conflict resolution. Example: merge a code-tuned model, a math-tuned model, and an instruction-following model into one model serving all three tasks. Benefits: single model deployment (one GPU, one serving instance) instead of three—reducing infrastructure cost 3×. Tasks requiring specialized capabilities can be served without routing to separate model instances. Merging quality varies: models fine-tuned from the same base merge better than from different bases.

Streaming sends tokens to the client as they are generated rather than waiting for the complete response. From a compute cost perspective, streaming is identical—the same tokens are generated. However, streaming dramatically improves user experience: time-to-first-token (TTFT) determines when the user first sees output (target <1s), while streaming delivers subsequent tokens progressively. Users can start reading and even interrupt generation if the response is already sufficient, potentially reducing effective output token count. For cost, early user interruption of a 500-token response when they're satisfied at 200 tokens saves 60% of output cost. Most API providers support streaming via SSE.

Structured pruning removes entire model components (attention heads, FFN neurons, layers) rather than individual weights, enabling actual inference speedup (unstructured/magnitude pruning of individual weights doesn't help without specialized sparse hardware). LLM pruning approaches: (1) head pruning—remove less important attention heads based on gradient sensitivity; (2) layer pruning—remove entire transformer blocks; (3) width pruning—reduce FFN hidden dimensions. ShortGPT (2024) found that many transformer layers are nearly redundant and can be dropped with minimal quality loss. LLM-Pruner applies structured pruning to compress LLaMA 20–50% with 1–5% quality loss. Post-pruning fine-tuning recovers most quality. Pruned models need no specialized hardware—standard matrix multiply benefits from reduced dimensions.

Greedy decoding (argmax at each step) costs 1 forward pass per token. Sampling (temperature/top-p/top-k) also costs 1 forward pass per token—identical to greedy. Beam search with beam width B costs B forward passes per step: B=5 is 5× more expensive than greedy. For chat and generation applications, beam search is rarely worth the cost—sampling with top-p often produces higher quality and more natural text than beam search. Beam search is justified for: translation, structured output where the globally optimal sequence matters. Best-of-N sampling (generate N independent sequences, return the best) costs N× but is embarrassingly parallel. For most modern LLM deployments, greedy or sampling decoding is the cost-optimal choice.

Amdahl's Law states that the speedup of a system is limited by the fraction that cannot be parallelized. For LLM systems: if token generation (memory-bandwidth bound, not parallelizable across tokens) constitutes 80% of wall time, improving prefill (20%) by 10× only improves total latency by ~18%. Optimization effort should focus on the dominant bottleneck. For latency-sensitive chat: decode throughput and KV-cache management (PagedAttention, GQA) dominate. For batch processing: prefill throughput and batching efficiency dominate. For cost: the most expensive operation (output tokens for API, GPU utilization for self-hosted) should be targeted first. Measure before optimizing—profiling reveals true bottlenecks.

Context window costs are linear: doubling input tokens doubles input cost. For API providers with long-context pricing tiers (Gemini charges more per token above 128K), costs scale super-linearly. For RAG: the optimal context is enough to answer the question accurately—empirically 1000–4000 tokens of retrieved context. Studies (Anthropic, 2023) show diminishing returns beyond 5 retrieved chunks for most QA tasks. The optimal chunk count is 3–5 high-relevance chunks; adding more increases cost without proportional quality gain. For tasks requiring synthesis of many documents, map-reduce patterns process each document in smaller context windows then aggregate, rather than loading all documents into one long context.

Embedding models charge per token (e.g., text-embedding-3-small: $0.02/M tokens) or CPU compute for self-hosted models. For a document corpus of 10M tokens, initial embedding costs $0.20 (API) or equivalent compute. The savings come from not re-embedding documents that haven't changed. An incremental indexing pipeline only embeds new or modified documents. Query embedding: each user query (~20 tokens) costs $0.0000004—negligible. Document embedding caching: store hash(document_text) → embedding vector; on re-index check hash first. For static corpora (legal databases, product catalogs), this reduces embedding costs to near-zero after initial indexing. Only document updates trigger re-embedding.

Multi-Query Attention (Shazeer, 2019) is the extreme case of GQA with G=1: all query heads share a single K and V head. This reduces KV-cache by H× (up to 32× for 32-head models) and minimizes memory bandwidth during decode—the primary bottleneck. MQA trains faster and infers faster at the cost of up to 5% quality degradation on some benchmarks. GQA with G=8 (used in Llama 2 70B) achieves a better tradeoff: 4–8× KV reduction with <1% quality loss. The original PaLM model used MQA. For production cost optimization, GQA is generally preferred over MQA as it provides most of the memory savings with better quality retention.

Asynchronous processing decouples user request submission from response delivery, enabling: (1) optimal batching—collect requests and process in cost-efficient batches rather than immediately at low utilization; (2) off-peak processing—run expensive batch jobs during cheaper off-peak hours on spot instances; (3) user experience—return a job ID immediately and deliver results when ready (acceptable for non-latency-sensitive tasks like document summarization, report generation); (4) queue-based load leveling—smooth traffic spikes that would require expensive burst capacity. Tools: Celery + Redis for Python, SQS + Lambda for AWS, Cloud Tasks for GCP. Async processing can reduce costs 30–50% for workloads tolerant of 5–60 second delays.

LLM FinOps practices: (1) Cost attribution—tag every API request with cost center, team, feature, user tier for chargeback; (2) Budget guardrails—per-team monthly budgets with automated alerts at 80% and hard stops at 100%; (3) Cost-per-unit tracking—cost per document summarized, per chat session, per API call by feature; (4) Model optimization working group—periodic review of highest-cost features for model downgrade or prompt optimization opportunities; (5) Reserved capacity planning for predictable workloads vs. on-demand for variable; (6) ROI measurement—compare LLM cost to value generated (support deflection rate, time saved). Enterprises with $100K+/month LLM spend typically form dedicated FinOps teams managing model tiers and usage policies.

Weight quantization (GPTQ, AWQ) quantizes the model's weight matrices to int4/int8, reducing model size and speeding up matrix multiplications. KV-cache quantization quantizes the attention keys and values stored in the KV-cache between decoding steps. Since KV-cache can consume 30–60% of inference VRAM for long sequences, int8 KV-cache quantization halves this memory with minimal quality loss. vLLM supports FP8 KV-cache; TensorRT-LLM supports INT8 KV-cache. Combined weight + KV quantization can reduce VRAM 3–4× compared to full FP16 inference, enabling larger batch sizes and higher throughput at fixed hardware cost.