Cost optimization for ML in production

Scale-to-zero deallocates all replicas when traffic is idle so you pay nothing during silence, but every cold start reloads the model and re-warms caches, adding latency and compute. It wins for spiky or low-QPS endpoints (dev, internal tools, long-tail models), but for steady, latency-sensitive traffic the repeated cold starts and the headroom needed to absorb them can cost more than a small always-on fleet. The decision hinges on request inter-arrival time versus cold-start duration.

Spot instances offer 60-90% discounts over on-demand because the cloud can reclaim them with short notice. They are safe for training when the job checkpoints frequently to durable storage and can resume from the last checkpoint, ideally with a fallback to on-demand for the final critical steps. Distributed training frameworks should handle node loss gracefully (elastic training) so a single preemption does not kill the whole run.

Quantization stores weights and/or activations in lower precision, shrinking memory footprint and enabling smaller/cheaper GPUs (or higher batch throughput per GPU), and often speeds up matmuls on hardware with low-precision tensor cores. The main risk is accuracy degradation, especially from activation outliers, so you must validate on held-out and production-representative data and consider per-channel scaling or quantization-aware training. Quantization can also be a no-op for cost if the kernel isn't actually using the low-precision path.

Prompt caching stores the model's internal state for a repeated prefix (e.g., a long system prompt or shared context) so subsequent requests reusing that prefix read cached tokens at a steep discount and run faster. The key constraint is that the cached portion must be an exact, stable prefix and stays valid only for a limited TTL, so you should structure prompts with static content first and dynamic content last. It is most effective for high-reuse prefixes like RAG instructions, tool schemas, or few-shot examples.

A cascade routes each request first to a small/cheap model and escalates to a larger expensive model only when the cheap one is uncertain or fails a quality check. Because most production traffic is easy, this serves the bulk of requests cheaply while preserving quality on hard cases, dramatically lowering blended cost per request. The trade-offs are added routing latency/complexity and the need for a reliable confidence or verification signal to decide escalation.

Larger batches amortize fixed per-call overhead and keep GPU compute units saturated, raising throughput (requests/sec per dollar) until you hit a hardware bottleneck. The limit is set by memory (KV cache / activations) and by your latency SLO: bigger batches add queueing delay, so beyond a point you trade unacceptable p99 latency for marginal cost gains. Continuous/in-flight batching (e.g., vLLM) decouples this by adding requests to a running batch dynamically.

Unit economics ties infrastructure spend to a business-meaningful unit, e.g., cost per inference, per active user, per resolved ticket, or per 1k tokens. Tracking it (rather than just the total cloud bill) reveals whether scaling traffic improves or erodes margins and lets you set budgets and alerts that map to value delivered. It is the foundation for ROI decisions like model swaps, caching investments, or pricing.

They give roughly 30-70% discounts in exchange for committing to a baseline of usage (instance family or spend) over 1-3 years. The central risk is over-commitment: if your workload shrinks, migrates instance types, or you adopt cheaper hardware, you keep paying for unused commitment (stranded capacity). The standard pattern is to cover the steady baseline with commitments and absorb spikes with on-demand/spot.

Distillation trains a smaller 'student' model to mimic a larger 'teacher' (matching its soft logits/behavior), retaining much of the teacher's quality at a fraction of the parameters. In production this replaces an expensive model with a cheaper student for the bulk of traffic, reducing GPU size, latency, and cost per request. It works best when you have abundant unlabeled inputs the teacher can label, and you validate that quality loss is acceptable on your task.

You pay for allocated GPU time regardless of whether the SMs are doing useful work, so low utilization means you are buying idle silicon. Common causes are small batch sizes, memory-bound kernels, CPU-side preprocessing or data-loading stalls, network/IO waits, and over-provisioned replicas. Raising effective utilization via batching, right-sizing, MIG/fractional GPUs, or co-location directly lowers cost per inference.

MIG partitions a single physical GPU (e.g., an A100/H100) into multiple isolated instances with dedicated memory and compute slices, so several small models or low-QPS endpoints can share one card with hardware isolation. This avoids dedicating a whole expensive GPU to a model that needs only a fraction of it, raising effective utilization and cutting cost. The trade-off is fixed partition sizes and reduced peak performance per slice.

First, retrieve fewer, higher-quality chunks (better embeddings, reranking, dedup) so you stuff less context into the prompt, since input tokens dominate cost. Second, cache: cache the stable system/instruction prefix (prompt caching) and cache full answers for repeated queries (semantic/exact response cache). Together these shrink per-request tokens and avoid redundant calls entirely.

Precomputing features in batch and serving from a low-latency store is cheap per read and predictable, but wastes compute/storage on features never requested and can be stale. On-demand (online) computation is always fresh and avoids storing unused features, but adds per-request compute and latency that scales with traffic. The cost-optimal design precomputes high-reuse, expensive-to-derive features and computes cheap, low-reuse or freshness-critical features on demand.

Output tokens are typically billed at a higher rate than input tokens and are generated sequentially, so unbounded generation is both expensive and slow. Capping max_tokens and adding stop sequences prevents runaway or verbose completions, directly bounding the most expensive part of the call. Combined with instructions for concise output, this can cut output token spend substantially with no model change.

resp = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=256,           # cap expensive output
    stop_sequences=["\n\nEND"],
    messages=[{"role": "user", "content": prompt}],
)

Autoscaling decisions should track the resource that is actually the bottleneck; GPU inference is usually GPU-bound while CPU stays low, so a CPU-utilization HPA never scales up under real load (causing latency breaches) or scales on idle CPU noise (wasting money). The fix is to scale on GPU utilization, request concurrency, queue depth, or a batch-latency SLO metric. Choosing the right signal prevents both over-provisioning and SLO violations.

Speculative decoding uses a small, fast draft model to propose several tokens which the large target model then verifies in a single parallel forward pass, accepting the longest correct prefix. Because verification is cheaper than generating each token autoregressively, it increases tokens-per-second and throughput per GPU while preserving the target model's output distribution. Higher throughput on the same hardware lowers cost per token, with gains depending on the draft-model acceptance rate.

GPU/accelerator pricing varies significantly across regions and across hardware (e.g., newer GPUs or alternative accelerators like Inferentia/TPUs can offer better price-performance for the right workload). Moving training or batch inference to cheaper regions/hardware can cut spend materially. The catches are data egress/transfer costs, data-residency/compliance constraints, latency to users, and accelerator-specific software/portability work to actually realize the price-performance.

Batch inference processes many inputs together on a schedule, letting you maximize GPU utilization, use spot instances, and avoid keeping endpoints warm, so cost per prediction is far lower. Real-time inference must hold capacity ready for unpredictable, latency-bound traffic, incurring idle headroom and cold-start mitigation costs. If a use case tolerates latency (e.g., nightly scoring, enrichment), moving it to batch is one of the largest cost wins available.

An exact-match cache returns a stored answer only when the request string matches exactly, while a semantic cache embeds the query and returns a cached answer when a prior query is similar above a threshold, catching paraphrases for far higher hit rates. The added risk is false hits: a semantically 'close' query may have a materially different correct answer, so you need a well-tuned similarity threshold, scope/tenant isolation, and freshness controls. The cost benefit is avoiding LLM calls entirely on cache hits.

Right-sizing matches instance type, replica count, and GPU/CPU/memory allocation to the model's actual resource profile and the traffic's load shape, eliminating over-provisioned headroom. To do it well you need load-test throughput/latency curves per instance type, real traffic distributions (peak vs p50, diurnal patterns), and the SLO. Without this you either over-pay for idle capacity or breach latency under spikes.

ML pipelines accumulate large volumes of training data, features, embeddings, logs, and model artifacts; keeping everything in hot storage is expensive and usually unnecessary. Lifecycle policies that move infrequently accessed data to cheaper warm/cold/archive tiers cut storage spend, while retrieval cost and latency for archived data is acceptable for rarely re-trained datasets. The trade-off is higher access cost/latency on retrieval, so set tiers by access frequency.

Retraining on a fixed schedule regardless of need burns compute (often the largest training line item) and may even degrade a perfectly good model. Control it with trigger-based retraining: retrain only when monitoring detects data/concept drift, performance decay below SLO, or sufficient new labeled data, rather than blindly on a cron. Pair this with cheaper incremental/partial retraining where possible and track retraining ROI (accuracy gain per dollar).

PEFT methods like LoRA freeze the base model and train only small low-rank adapter matrices, slashing trainable parameters, optimizer-state memory, and thus GPU size and training time. At serving time, many lightweight adapters can share one base model in memory (multi-LoRA serving), avoiding a separate full model per task/tenant. The result is dramatically lower training and serving cost for customization, with usually minor quality trade-offs versus full fine-tuning.

Cost-aware selection picks the model that minimizes total cost to meet a quality bar, not just the lowest list price per token. A cheaper model may need longer prompts, more retries, more tokens to reach the same quality, or more human correction downstream, raising true cost per successful task above a pricier but more capable model. You must evaluate blended cost per resolved task (including retries, tokens, and failure cost), not headline token price.

Experimentation leaves behind running notebook VMs, unattached GPUs, idle endpoints, leftover dev clusters, and unused volumes/snapshots that bill continuously with no traffic. They are caught with tagging/labeling conventions, automated idle-detection (low utilization over time), TTL/auto-shutdown policies on dev resources, and regular cost-anomaly reviews. Enforcing ownership tags makes orphaned spend attributable and reclaimable.

During generation, each request holds a KV cache proportional to sequence length, and it is this memory (not raw FLOPs) that often caps how many concurrent requests fit on a GPU, which in turn caps throughput per dollar. Techniques include PagedAttention (vLLM) to eliminate fragmentation, KV-cache quantization, grouped-query/multi-query attention to shrink cache size, prefix sharing for common prompts, and bounding context length. Each lets more concurrent requests share a GPU, lowering cost per request.

An LLM gateway centralizes all model calls, giving one place to enforce caching, rate limits and per-team budgets, route to the cheapest adequate model/provider, log token usage for chargeback, and add retries/fallbacks. This visibility and control prevents uncontrolled spend, enables A/B cost comparisons, and lets you swap providers without touching app code. Without it, cost governance is scattered and hard to attribute.

For small models, low/spiky QPS, or latency-tolerant workloads, GPUs sit idle and their hourly premium isn't recovered, so CPUs (especially with optimized runtimes like ONNX Runtime/OpenVINO and quantization) can deliver lower cost per request. GPUs win when the model is large or traffic is high enough to keep them saturated via batching. The decision is a break-even between the GPU's higher throughput and its higher hourly cost relative to your actual load.

Aggressive retries on slow/failing model calls multiply token and compute spend, and a retry storm during an outage can amplify both cost and the incident. Tune with bounded retry counts, exponential backoff with jitter, idempotency, circuit breakers, and timeouts set to the SLO so you fail fast instead of paying for doomed long calls. The goal is resilience without paying repeatedly for work that won't succeed.

Early stopping halts training once a validation metric stops improving for a set patience, avoiding wasted epochs (and their GPU cost) and overfitting. Guards include a sufficient patience window, monitoring a smoothed/representative validation metric, and restoring the best checkpoint rather than the last. This converts a fixed-epoch budget into a need-based one, directly cutting compute spend on plateaued runs.

Grid search exhaustively evaluates combinations and scales exponentially, wasting compute on unpromising regions. Random search covers the space far more cheaply for the same budget, and Bayesian optimization plus early-termination schedulers like Hyperband/ASHA concentrate compute on promising configs and kill bad trials early. Choosing the latter strategies reaches comparable quality at a fraction of the GPU-hours.

Without explicit cost budgets and alerts, spend silently grows with traffic, prompt bloat, or a model swap, and is only noticed on the monthly bill, too late to react. Treating cost as an SLO means setting per-endpoint/per-tenant budgets, real-time cost-per-request dashboards, anomaly alerts, and automated guardrails (rate caps, model downgrade on budget breach). This makes cost regressions as visible and actionable as latency or error-rate regressions.

Chinchilla scaling laws show that for a fixed compute budget, models are often undertrained on data; balancing parameter count and training tokens yields better quality per FLOP than just making the model bigger. For practitioners this means a smaller model trained on more data can beat a larger under-trained one at lower training and, crucially, lower inference cost. Since inference cost recurs forever, favoring a smaller compute-optimal model often minimizes lifetime total cost.

When many concurrent requests ask for the same computation (e.g., the same popular query or embedding), coalescing detects in-flight duplicates and lets them share a single model call rather than each triggering its own. This collapses redundant compute during bursts and cache-miss stampedes, lowering load and cost while improving latency. It complements caching by handling the window before a result is cached (the 'thundering herd').