ML system design

Scoring millions of items per request with a heavy ranking model is computationally infeasible at low latency. Candidate generation uses cheap retrieval (e.g., ANN over embeddings) to reduce millions to hundreds, and a richer ranking model then scores only that small set. This decouples recall (cheap, high-coverage) from precision (expensive, high-fidelity), letting each stage be optimized independently. Many production stacks add a third lightweight re-ranking/policy layer for diversity and business rules.

Training-serving skew occurs when feature values or transformations differ between training and inference, causing degraded online performance despite good offline metrics. The most robust remedy is to share a single feature-transformation pipeline/code path (or a feature store) between training and serving so the exact same logic computes features in both environments. Logging served features for use as training data ('log-and-train') further guarantees parity.

The offline store holds large historical feature data (often in a columnar warehouse/data lake) optimized for high-throughput batch reads to build training sets with point-in-time correctness. The online store holds the latest feature values in a low-latency key-value store (e.g., Redis/DynamoDB) for millisecond reads at serving time. Both are populated from the same feature definitions to avoid training-serving skew.

Point-in-time correctness ensures each training label is joined only with feature values that were actually available at or before the event timestamp, preventing future information from leaking into features. Without it, the model trains on data it won't have at inference, inflating offline metrics and failing in production. This typically requires as-of (temporal) joins keyed on event time.

CTR is a proxy that can be gamed by clickbait, so you should define a North Star/guardrail metric tied to long-term value (retention, session depth, revenue) and run the test long enough to observe it. Use guardrail metrics that block launch if degraded, and consider long-term holdback groups and counterfactual logging to measure delayed effects. The lesson is to align the optimization objective and evaluation metric with business value, not just immediate engagement.

Shadow deployment sends live production traffic to the new model in parallel with the current model, but its predictions are not served to users-only logged. This validates latency, throughput, infrastructure stability, and prediction distribution under real load without any user-facing risk. It is ideal for catching serving bugs and training-serving skew before a canary or A/B test exposes real users.

Data (covariate) drift is a change in the input feature distribution P(X), while concept drift is a change in the relationship P(Y|X) between inputs and the target. Data drift can sometimes be tolerated if the decision boundary still holds, but concept drift directly degrades accuracy and usually requires retraining or relabeling. Monitoring must track both input distributions and, where labels are available, prediction quality over time.

Delayed labels mean you cannot immediately measure true accuracy, so you rely on proxy/early signals and distribution monitoring in the short term while reserving labeled evaluation for when ground truth matures. Training pipelines must account for an 'embargo'/maturation window so recent unlabeled data isn't naively treated as negatives, and you should report metrics with appropriate maturation lag. Naive online learning is risky here because feedback is slow and noisy.

Exact KNN requires comparing a query against every vector (O(N) per query), which is too slow for millions/billions of vectors at low latency. ANN indexes like HNSW trade a small amount of recall for sub-linear search by navigating a graph or partitioned space, achieving millisecond retrieval. The recall/latency tradeoff is tunable via parameters like ef_search (HNSW) or nprobe (IVF).

Hallucination despite good retrieval often stems from the model not being constrained to the context, so add explicit grounding instructions, require citations/quote spans, and use a verification step (e.g., an LLM-judge or NLI/entailment check) that the answer is entailed by retrieved passages. Reducing context noise via reranking and better chunking helps, and abstaining ('I don't know') when retrieval confidence is low prevents fabrication. Lowering temperature also reduces ungrounded generation.

A model registry is a central system that versions trained models along with their metadata, lineage, metrics, and stage (e.g., staging/production/archived). It enables reproducible deployment, auditability, rollback to known-good versions, and governance/approval workflows. It decouples model training from deployment by giving serving systems a stable reference to promoted versions.

Choose batch when predictions can be precomputed and don't depend on fresh request-time inputs-e.g., daily product recommendations, churn scores, or email targeting-where latency is non-critical and inputs change slowly. Batch is simpler, cheaper, and lets you reuse warehouse compute. Online serving is required when predictions depend on real-time context (search query, current cart) or must reflect very fresh data.

A/B tests fix traffic allocation and decide a winner after the test, accepting regret from sending traffic to inferior variants throughout. Bandits adaptively shift traffic toward better-performing arms during the experiment, minimizing regret, which is valuable for short-lived content (news, promotions) or many variants. The tradeoff is that adaptive allocation complicates clean statistical inference and measurement of delayed/long-term metrics, and assumes reward feedback is reasonably fast.

Cold start occurs when a new user or item has no interaction history, so collaborative-filtering signals are unavailable. For new items, use content-based features (text, image, metadata embeddings) to place them in the same space as existing items, and apply exploration (e.g., bandit boosting) to gather initial feedback. Hybrid models that fuse content and collaborative signals smoothly transition as data accumulates.

Some features are expensive to compute (aggregations over large windows, graph features) and cannot be calculated within the request latency budget. Precomputing and materializing them to the online store asynchronously lets serving do a fast key lookup instead of on-the-fly computation. The tradeoff is feature freshness: materialized features lag real-time, so freshness SLAs must match the use case.

A two-tower model learns separate encoders for the query/user and the item, producing embeddings in a shared space such that relevant pairs have high dot-product (or cosine) similarity. Item embeddings are precomputed and indexed for ANN retrieval, while the user tower runs at request time, enabling fast large-scale retrieval. Training commonly uses in-batch negatives with a contrastive/sampled-softmax loss.

# Two-tower scoring
import torch
user_emb = user_tower(user_features)      # [B, d]
item_emb = item_tower(item_features)      # [N, d] precomputed/indexed
scores = user_emb @ item_emb.T            # [B, N] dot-product similarity

Prefill processes the entire prompt in parallel and is compute-bound, while decode generates one token at a time autoregressively and is memory-bandwidth-bound (dominated by KV-cache and weight reads). Because requests have different prompt and generation lengths, naive static batching wastes GPU; continuous (in-flight) batching admits and evicts sequences per step to keep the GPU saturated. This phase asymmetry also drives techniques like chunked prefill and prefill/decode-disaggregated serving.

The KV cache stores the key and value tensors of all previously processed tokens so each new decode step avoids recomputing attention over the whole sequence. Its size grows linearly with sequence length and batch size, consuming large amounts of GPU memory and often becoming the binding constraint on concurrency. Techniques like PagedAttention (vLLM), quantized KV cache, and grouped-query attention (GQA) reduce this footprint.

LLM judges exhibit position bias (favoring the first/last answer), verbosity bias (favoring longer responses), and self-preference (favoring outputs from the same model family). Mitigations include randomizing/swapping answer order and averaging, using rubric-anchored prompts with explicit criteria, length-controlling comparisons, calibrating against human-labeled samples, and using a different/stronger judge model. Pairwise comparison with order swapping is generally more reliable than absolute single-answer scoring.

Lambda architecture maintains two separate code paths-a batch layer for accurate historical computation and a speed layer for low-latency approximate results-then merges them, which duplicates logic and maintenance. Kappa architecture uses a single stream-processing path for both real-time and reprocessing (by replaying the log), reducing duplicate logic. Kappa is favored when stream tech can also handle reprocessing; lambda persists when batch and stream semantics genuinely differ.

Offline metrics are computed on logged data collected under the old model's policy, so they suffer from distribution shift, feedback loops, and position/selection bias not captured offline. Online A/B testing measures actual user behavior under the new policy, and counterfactual/off-policy evaluation (e.g., inverse propensity scoring) can partially bridge the gap before launch. The principle: offline metrics gate, online experiments decide.

A feedback loop arises when a model's predictions influence future training data-e.g., a recommender only shows items it scores highly, so it only collects feedback on those, reinforcing its own biases. This causes degeneracy (popularity rich-get-richer), reduced exploration, and metrics that look good but reflect the model's own selection. Mitigations include exploration, logging propensities for debiasing, and counterfactual evaluation.

Caching frequent or identical requests (e.g., popular query embeddings or scores) cuts latency and compute dramatically. The risk is staleness: cached predictions can become outdated when the model is updated or inputs change, so you need cache invalidation tied to model version and appropriate TTLs. Cache keys must include the model version (and relevant feature inputs) to avoid serving predictions from a retired model.

Canary deployment routes a small percentage of traffic to the new model version, monitors metrics, and gradually increases if healthy, limiting blast radius. Blue-green keeps two full environments and switches all traffic at once (with instant rollback), but doesn't progressively de-risk. Canary is preferred for ML when you want to observe real prediction quality on a slice before full exposure.

This is a sliding/tumbling window aggregation over an event stream, best computed with a stream processor (Flink, Spark Structured Streaming, Kafka Streams) that maintains windowed state and writes the rolling count to the online store. Computing it on-demand at request time would be too slow over raw events. Watermarks handle late-arriving events, and the aggregation must be keyed by user.

Quantization reduces weights/activations to lower precision, shrinking memory footprint and increasing throughput/lowering latency and cost, which is critical for large models. The main tradeoff is potential accuracy degradation, mitigated by techniques like quantization-aware training (QAT), per-channel/group scaling, and keeping sensitive layers in higher precision. Post-training quantization is fast but may lose more accuracy than QAT.

Vertical scaling adds more resources to a single node (bigger GPU/more memory), while horizontal scaling adds more nodes/replicas behind a load balancer. GPU inference typically scales horizontally with autoscaling replicas because a single model fits one or few GPUs and traffic is parallelizable, while vertical scaling (or multi-GPU sharding) helps when a single model is too large for current hardware. Horizontal scaling also improves availability.

Without labels, monitor unsupervised proxies: input feature distributions (PSI/KL divergence vs training), prediction score distribution shifts, model confidence/entropy, and the rate of edge cases or fallbacks. Track operational metrics (latency, error rate) and downstream business proxies, and alert on statistically significant deviations. Periodic delayed-label backfills then confirm true quality.

PSI quantifies how much a variable's distribution has shifted between a reference (training) period and a current period by binning values and summing (curr%-ref%)*ln(curr%/ref%). It is widely used to detect feature or score drift, with rules of thumb like <0.1 stable, 0.1-0.25 moderate shift, >0.25 significant shift. It is label-free, making it useful for early drift alerts.

import numpy as np

def psi(ref, cur, bins=10):
    edges = np.percentile(ref, np.linspace(0, 100, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf  # capture out-of-range values
    r, _ = np.histogram(ref, edges)
    c, _ = np.histogram(cur, edges)
    r = r / r.sum() + 1e-6
    c = c / c.sum() + 1e-6
    return float(((c - r) * np.log(c / r)).sum())

Logged interaction data is biased because users mostly click highly-ranked items regardless of relevance (position bias). Recording the propensity (e.g., examination probability per position) lets you apply inverse-propensity weighting to debias the training labels and to perform unbiased off-policy evaluation of new rankers. Without propensities you cannot counterfactually estimate how a different ranking would have performed.

A durable log/queue decouples producers from consumers, buffering bursts so downstream feature processors aren't overwhelmed, and providing replayability for reprocessing/backfills. It enables multiple independent consumers (training logger, online materializer, monitoring) to read the same stream, and gives per-partition ordering and at-least-once delivery guarantees. This decoupling improves resilience and scalability.

Pipelines fail and get retried, so non-idempotent steps can double-count events, duplicate features, or corrupt aggregates, leading to data quality bugs that silently degrade models. Designing steps so reprocessing the same input yields the same output (via deduplication keys, upserts, deterministic partitions) makes pipelines safely retryable and backfillable. This is foundational for reliable, reproducible ML data.

Tail latency is reduced by request hedging/timeouts with backups, limiting per-request work (capping candidate counts, feature lookups), warming caches and model/JIT warmup, batching with strict deadlines, and avoiding garbage-collection/cold-start stalls via pre-allocation and replica pre-warming. Isolating slow dependencies and using circuit breakers prevents one slow component from dominating p99. Profiling the critical path is essential since the mean hides tail behavior.

Multi-task learning shares representations across objectives, improving data efficiency (sparse conversions benefit from abundant click signal) and reducing serving cost via one model. The key risk is negative transfer and gradient conflict, where optimizing one task harms another, plus the need to combine task outputs into a single ranking score with tunable weights. Architectures like MMoE/PLE or gradient-balancing methods mitigate task interference.

Item embeddings are expensive to compute but change slowly, so generating them in batch and storing them in an ANN index lets the online path do a cheap vector lookup. Only the query/user embedding needs real-time computation, keeping serving fast. The tradeoff is that item embeddings can become stale until the next batch refresh, requiring a refresh cadence matched to item churn.

Options include learned embedding tables (mapping IDs to dense vectors), the hashing trick to bound dimensionality with collision tradeoffs, frequency-based truncation (group rare values into 'other'), and target/count encoding computed with point-in-time correctness to avoid leakage. Embeddings are standard in deep recommenders, while hashing controls memory at the cost of collisions. The choice balances memory, leakage risk, and the value of rare categories.

Speculative decoding uses a small, fast draft model to propose several tokens ahead, which the large target model then verifies in a single parallel forward pass, accepting the longest correct prefix. Because verification is parallel and cheaper than generating each token sequentially, it speeds up decoding while preserving the target model's output distribution (lossless). The speedup depends on the draft model's acceptance rate.

A distilled model retains much of the larger model's quality while drastically reducing latency, memory, and serving cost, which matters for high-QPS, latency-sensitive features. At scale, cost and p99 latency often outweigh marginal accuracy, and a smaller model enables higher batch throughput and cheaper horizontal scaling. The decision is driven by accuracy-per-dollar and the latency budget, not raw benchmark scores.

If the feature store is slow or unavailable, blindly waiting can cascade failures and blow the latency budget; a circuit breaker detects failures and short-circuits to a fallback (default feature values, a cached/last-known value, or a simpler model). This trades some accuracy for availability and bounded latency, keeping the overall service responsive during dependency outages. Fallback behavior should be explicitly designed and monitored, not accidental.

Automatically deploying every retrained model risks promoting one degraded by data quality issues, drift artifacts, or training bugs. A champion-challenger workflow evaluates the new (challenger) model against the current (champion) on held-out and/or live shadow traffic, promoting only if it meets metric and guardrail thresholds. This adds a quality gate and enables automatic rollback to the champion if the challenger regresses.