MLOps and deployment

A model registry is a centralized store that versions trained models along with their metadata, lineage, and stage transitions (e.g., Staging, Production, Archived). It enables governance, reproducibility, and controlled promotion/rollback of models across environments, decoupling training from deployment.

A canary deployment routes a small percentage of live traffic to the new model version while the majority continues hitting the current version. It validates the new model's behavior and performance on real traffic with limited blast radius before a full rollout.

In shadow deployment, the new model receives a copy of live production traffic but its predictions are not returned to users—only logged for comparison. Unlike canary, it carries zero user-facing risk because the candidate model never serves real responses.

Data drift is a change in the input feature distribution P(X), while concept drift is a change in the relationship between inputs and target P(Y|X). Concept drift degrades model accuracy even when inputs look statistically similar, and typically requires retraining with fresh labels rather than just monitoring inputs.

The Kolmogorov-Smirnov (KS) test compares the empirical cumulative distribution functions of the reference and current data to detect distribution shift in continuous features. The Population Stability Index (PSI) and Wasserstein distance are also common alternatives.

A feature store provides a centralized, versioned repository of features with consistent definitions shared between training and serving, eliminating train/serve skew. It typically has an offline store (for batch training) and a low-latency online store (for real-time inference), backed by the same transformation logic.

Train/serve skew is a discrepancy between how features are computed during training versus serving, leading to degraded online performance despite good offline metrics. A common cause is duplicated, divergent transformation code paths (e.g., pandas in training, hand-written Java/SQL at serving) or different data freshness.

Batch inference is preferred when predictions are not needed instantly, inputs are available in bulk, and you can amortize compute (e.g., nightly churn scoring, recommendation precomputation). It is cheaper and simpler to scale, whereas online inference is required when latency is user-facing and inputs arrive on demand.

A serving framework wraps trained models behind a standardized inference API and handles concerns like batching, GPU scheduling, multi-model hosting, versioning, and autoscaling. It lets teams deploy models as scalable network services without rebuilding serving infrastructure per model.

Dynamic batching collects multiple incoming inference requests within a short time window and runs them as a single batched forward pass. It dramatically improves GPU throughput and utilization at the cost of slight added latency, which is configurable via a max batch size and timeout.

Quantization reduces weights/activations from FP32/FP16 to lower-precision integers, shrinking model size and memory bandwidth while speeding up inference on supported hardware. It trades a small, usually acceptable accuracy drop for major latency and cost reductions, and is widely used for edge and LLM serving.

CPU-based autoscaling reacts to CPU utilization, which is a poor proxy for GPU-bound or latency-sensitive inference workloads. Custom/external metrics (e.g., requests-per-second, GPU utilization, queue depth, p99 latency) scale on signals that actually reflect inference load, giving more responsive and accurate scaling.

DVC versions large datasets and model artifacts by storing lightweight metafiles in Git while pushing the actual large files to remote object storage (S3, GCS). This gives reproducible data/model lineage tied to code commits without bloating the Git repository with binaries.

MLflow Tracking logs experiments—parameters, metrics, and artifacts for individual runs. The Model Registry is a separate component that catalogs registered model versions, manages stage transitions, and enables governance/approval workflows on top of logged models.

The champion is the current production model serving traffic, while one or more challengers are evaluated against it (via shadow or A/B traffic) on live data. If a challenger outperforms the champion on agreed metrics, it is promoted, enabling continuous, evidence-based model improvement.

A liveness probe restarts a container that has become unresponsive, while a readiness probe controls whether a pod receives traffic—critical because a model may need time to load weights into memory/GPU before it can serve. Without a proper readiness probe, traffic is routed to pods that aren't yet ready, causing errors.

Containerization packages the model with its exact runtime, libraries, and system dependencies into a portable, reproducible image. This eliminates 'works on my machine' issues and ensures consistent behavior across dev, staging, and production environments.

Using `latest` makes deployments non-deterministic and irreproducible—different nodes or future pulls may fetch different image contents, breaking rollbacks and auditing. Production should pin immutable, versioned tags (or digests) so the exact artifact is always known.

Continuous training can be triggered by schedules, the arrival of sufficient new labeled data, or monitoring signals such as detected drift or a performance drop below an SLA threshold. The retrained model then passes automated validation gates before promotion.

A/B testing splits users into groups, each seeing a single model's output, and compares aggregate metrics. Interleaving mixes results from both models within a single user's response and attributes engagement per item, yielding higher statistical sensitivity with less traffic—popular in ranking/search.

Model lineage is the end-to-end record linking a deployed model to the code, data, features, hyperparameters, and environment used to produce it. It is essential for reproducibility, auditing, debugging regressions, and meeting regulatory requirements (e.g., explaining a model's provenance).

A cold start is the latency incurred when a new function instance must initialize—loading the runtime and the (often large) model into memory—before serving the first request. It makes serverless poorly suited to large models or strict tail-latency SLAs unless mitigated with provisioned concurrency or smaller models.

Mean latency hides the tail of slow requests, while p99 (the 99th percentile) captures the worst-case experience that real users hit during spikes, GC pauses, or batching delays. Optimizing tail latency directly protects user experience and downstream timeout budgets.

A rollback restores a previously known-good model when a new one causes errors or metric regressions. Blue-green deployment keeps the old (blue) and new (green) versions running simultaneously and switches traffic via the router; if green fails, traffic instantly reverts to blue with near-zero downtime.

Prompt templates materially change LLM behavior, so they must be versioned, tested, and tied to evaluation results just like model weights. Prompt versioning tracks the externalized instruction/template (and its variables) independently from the underlying model version, since either can change and affect outputs.

The KV cache stores the key and value tensors of previously generated tokens so the model avoids recomputing attention over the full sequence at each decoding step. It is central to autoregressive throughput but consumes large GPU memory that grows with batch size and context length, making its management (e.g., PagedAttention) a key serving optimization.

Continuous batching adds and removes sequences from the running batch at the token level as requests finish or arrive, rather than waiting for the whole batch to complete. This keeps the GPU saturated despite highly variable generation lengths, greatly improving throughput versus static batching.

The vector index becomes stale as the underlying knowledge base changes—new documents aren't indexed, deleted ones remain, or embeddings were generated by an older embedding model. Operationally this requires an incremental indexing/re-embedding pipeline and monitoring of retrieval relevance.

HNSW gives sub-linear approximate nearest-neighbor search with very low latency at scale, but uses more memory and returns approximate results tunable via parameters (ef, M). A flat index is exact but scans all vectors, making it accurate yet too slow/expensive for large corpora.

Schema validation rejects malformed or out-of-range requests before they reach the model, preventing crashes, silent garbage predictions, and security issues from untrusted input. It also enforces a stable contract between clients and the service, surfacing breaking changes early.

from pydantic import BaseModel, Field

class PredictRequest(BaseModel):
    age: int = Field(ge=0, le=120)
    income: float = Field(ge=0)
    features: list[float] = Field(min_length=10, max_length=10)

Ground-truth metrics (e.g., accuracy, AUC) require labels that often arrive with delay or not at all, so they can't be computed immediately. Proxy/operational metrics—prediction distribution, drift, confidence, business KPIs—are available in real time and act as early warning signals until labels materialize.

Logging the exact inputs, model version, and predictions (and later joining delayed ground-truth labels) creates the dataset needed to measure live performance, detect drift, debug failures, and assemble training data for retraining. Without this closed loop, you cannot reliably evaluate or improve the deployed model.

Low GPU utilization under load usually signals a bottleneck elsewhere—CPU-bound pre/post-processing, small un-batched requests, data loading/network I/O, or Python GIL contention—so the GPU sits idle waiting for work. Fixing it involves batching, offloading preprocessing, or increasing concurrency to keep the GPU fed.

Model decay (staleness) is the gradual decline in a deployed model's performance over time as the world drifts away from the training distribution. The standard mitigation is a monitored retraining strategy—scheduled or trigger-based continuous training on fresh data with automated validation before redeployment.

Pickle is insecure (it can execute arbitrary code on load) and brittle across library/Python versions, so a model pickled in one environment may fail to load in another. Safer, more portable formats like ONNX, SavedModel, TorchScript, or safetensors are preferred for cross-environment deployment.

ONNX is a framework-agnostic intermediate representation that lets a model trained in PyTorch/TensorFlow run on optimized runtimes (ONNX Runtime, TensorRT) across diverse hardware. This decouples training framework from serving stack and often unlocks graph optimizations and faster inference.

A quality gate automatically checks a new model against thresholds—minimum metric performance, no regression versus the current production model, fairness/slice checks, and inference latency—before allowing promotion. It prevents shipping models that pass training but would degrade production behavior.

Multi-model serving packs multiple models onto shared compute, improving hardware utilization and cost-efficiency for many low-traffic models that would otherwise each waste a dedicated GPU. Frameworks load/unload models on demand and route requests, at the cost of more complex memory management and noisy-neighbor risk.

A model card is standardized documentation describing a model's intended use, training data, performance across relevant subgroups, limitations, and ethical considerations. It supports transparency, governance, and informed decisions by downstream users and auditors about whether/how to use the model.

An idempotent pipeline produces the same end state when a step is retried, which is essential because distributed jobs frequently fail and re-run. Without idempotency, retries can duplicate data, create conflicting model versions, or double-promote artifacts, causing inconsistent and irreproducible deployments.