CI/CD for machine learning

Traditional CI/CD versions and tests only code, whereas ML CI/CD must additionally version and validate data and models, which are first-class artifacts. This introduces Continuous Training (CT): pipelines can be automatically retriggered to retrain and redeploy models when new data or drift appears, not just when code changes. Therefore ML pipelines test data schemas, model quality, and skew in addition to unit/integration tests.

MLOps level 0 is a fully manual, script-driven process where data scientists hand off a trained model to engineers for deployment, with infrequent releases and no CI/CD or automated retraining. There is a disconnect between ML and operations, and monitoring is typically absent. Levels 1 and 2 progressively add pipeline automation (CT) and full CI/CD automation.

Training-serving skew is a discrepancy between how features are computed during training versus at serving time, leading to degraded online performance despite good offline metrics. Common causes include different code paths for batch vs. online feature computation and data that changes between training and inference. CI/CD pipelines mitigate it by sharing feature transformation logic (e.g., a feature store), adding skew-detection tests, and validating serving outputs against training distributions.

Because model quality is bounded by data quality, pipelines validate incoming data before training to fail fast on bad inputs. Typical checks include schema conformance (types, required columns), value ranges and allowed categories, missing-value rates, and distribution drift relative to a reference. Tools like TensorFlow Data Validation, Great Expectations, or Pandera codify these checks so violations block the pipeline.

A model registry is a centralized, versioned store of trained models with metadata, lineage, metrics, and stage labels (e.g., Staging, Production, Archived). In CI/CD it acts as the handoff point: a CT pipeline registers a candidate model, automated gates promote or reject it, and the deployment job pulls the approved version. Examples include MLflow Model Registry, SageMaker Model Registry, and Vertex AI Model Registry.

from mlflow import MlflowClient
c = MlflowClient()
# Newer MLflow favors alias-based promotion over stages:
c.set_registered_model_alias(name="churn", alias="production", version=7)

CT triggers are the events that automatically kick off retraining pipelines without manual intervention. Three common ones are: scheduled/cron triggers (e.g., nightly), data-driven triggers (arrival of a new batch or threshold of new labels), and performance/drift triggers fired by monitoring when metrics degrade or distributions shift. A robust system combines these with manual on-demand triggers.

Seeds control pseudo-random operations but not nondeterminism from GPU kernels (e.g., atomic ops), library/framework versions, hardware differences, or data ordering. Full reproducibility additionally requires versioning data and code (e.g., Git + DVC), pinning dependency versions and container images, enabling deterministic algorithm flags, and recording the full environment and config. Even then, bitwise reproducibility on GPUs may require disabling certain optimizations.

import torch
torch.use_deterministic_algorithms(True)
torch.manual_seed(42)
# Also set CUBLAS_WORKSPACE_CONFIG=:4096:8 for deterministic cuBLAS

A model validation gate decides whether a candidate is good enough to deploy, going beyond raw test metrics. It typically requires the candidate to beat the current production model (or a baseline) on a holdout, checks performance on critical data slices for fairness/regressions, and verifies operational properties like latency and size. Only if all gates pass is the model registered for promotion; otherwise the pipeline fails and blocks release.

Git is optimized for text diffs, not large binaries; committing them bloats history, slows clones, and hits size limits. The standard alternative is to store data/models in object storage (S3, GCS) and track lightweight pointers with tools like DVC or Git LFS, so Git versions metadata while artifacts live remotely. The pipeline then pulls the exact versioned artifact by hash during CI runs.

dvc add data/train.parquet
git add data/train.parquet.dvc .gitignore
dvc push  # to S3/GCS remote

In shadow deployment, the new model receives a copy of live production traffic and produces predictions that are logged but not served to users; the existing model still serves responses. The benefit is validating real-world performance, latency, and stability with zero user risk. The limitation is that you cannot measure the new model's effect on user behavior/business KPIs since its outputs never influence users.

Blue-green keeps two full environments (old=blue, new=green) and switches all traffic at once after validating green, enabling instant rollback by flipping back. Canary gradually shifts a small percentage of traffic to the new model, monitors metrics, and increases the share only if healthy, limiting blast radius. Canary gives finer risk control and real metric-based gating; blue-green gives simpler, faster cutover but a larger blast radius at switch time.

Containers package the model, code, runtime, system libraries, and dependencies into an immutable image, ensuring the exact same environment runs in CI, staging, and production. This eliminates 'works on my machine' drift caused by differing OS/library versions, especially important for CUDA/driver-sensitive ML workloads. Images are versioned/tagged (ideally by digest) and stored in a registry so deployments are byte-for-byte reproducible.

Code tests verify deterministic logic (data transforms, feature functions, pipeline plumbing) and run fast on every commit; model tests verify learned behavior, which is stochastic and slower. Example code test: assert a normalization function maps a known input to an expected output. Example model behavioral test: assert invariance (a sentiment model gives the same label when a person's name changes) or directional expectations on perturbed inputs.

CACE stands for 'Changing Anything Changes Everything', from Google's 'Hidden Technical Debt in Machine Learning Systems' paper (Sculley et al., 2015): because models entangle all input signals, altering one feature, hyperparameter, or data source can shift the entire model's behavior. This means you cannot test changes in isolation as with modular code, so CI/CD must rely on holistic end-to-end evaluation, regression testing on slices, and careful monitoring of every release. It motivates strong validation gates rather than assuming local changes have local effects.

A feature store provides a single, versioned definition of features served consistently to both offline training (point-in-time correct historical values) and online inference (low-latency lookups). By reusing the same transformation logic and stored values, it eliminates duplicate feature code that causes training-serving skew. It also enables feature reuse, governance, and point-in-time joins that prevent label leakage in training pipelines.

Covariate/data drift is a change in the input distribution P(X) while the relationship stays the same; concept drift is a change in P(Y|X), i.e., the underlying input-output relationship itself shifts. Both can degrade live performance, but concept drift typically requires retraining with fresh labels, whereas data drift may sometimes be handled by re-weighting or expanded data. Monitoring distinguishes them so CT triggers retraining appropriately rather than firing on harmless input shifts.

It brings model evaluation results directly into the code review process so reviewers can see the quantitative impact of a change (metric deltas, confusion matrices, drift plots) before merging. This turns model quality into a reviewable, auditable part of the PR rather than a separate manual step. Tools like CML or GitHub Actions generate a report from the CI run and post it back to the PR via the API.

cml comment create report.md  # posts metrics & plots to the PR

Hardcoded secrets leak into version control, logs, and built images, creating a serious security exposure that is hard to rotate. The correct approach is to inject them at runtime from a secrets manager (Vault, AWS/GCP Secret Manager) or CI masked variables/OIDC, scoped with least privilege. OIDC-based short-lived credentials are preferred over long-lived static keys to limit blast radius.

Parameterization externalizes environment- and run-specific values (data paths, hyperparameters, compute size, target environment) as inputs rather than hardcoding them in pipeline code. This lets the same pipeline definition run unchanged across dev/staging/prod, supporting reproducible experiments and environment promotion. In MLOps level 2, pipelines themselves become artifacts that are built, tested, and deployed with parameters supplied per environment.

Data leakage occurs when information unavailable at prediction time (e.g., future values or target-derived features) leaks into training, producing over-optimistic offline metrics that collapse in production. A CI pipeline can automatically check it by enforcing point-in-time correct joins, asserting no overlap between train/test split keys (e.g., user IDs), and flagging features whose correlation with the target is suspiciously near-perfect. Time-based splits and split-integrity assertions are common automated guards.

assert set(train_ids).isdisjoint(set(test_ids)), 'ID overlap = leakage'

Unit tests verify components in isolation, but ML pipelines fail at integration boundaries: schema mismatches between stages, serialization/format incompatibilities, and the model artifact not loading in the serving environment. An integration test runs the full pipeline on a small representative sample end-to-end, confirming components interoperate and the deployed model can actually score requests. This catches the 'each part works but the whole breaks' failures that are common in stitched-together ML systems.

Model-as-artifact ships only the serialized model (and metadata) to be loaded by a pre-existing serving runtime, so the CI/CD release is a registry promotion plus a config pointer update. Model-as-service packages the model together with its serving code into a deployable unit (container/endpoint) that is built, tested, and rolled out as a whole. The artifact approach decouples model updates from infra; the service approach gives tighter control but rebuilds/redeploys infrastructure on each model change.

Use a tiered strategy: run fast unit/code tests and a 'smoke' train on a small data sample for every commit, and reserve full-scale training and exhaustive evaluation for merges to main or scheduled runs. Cache dependencies and intermediate datasets, use pipeline step caching/memoization (e.g., DVC, Kubeflow), and run heavy GPU jobs only when relevant code paths change. This balances rapid feedback with thorough validation without paying full training cost on every push.

Model lineage is the complete, traceable record linking a deployed model to the exact code commit, dataset version, hyperparameters, environment, and training run that produced it. In regulated domains (finance, healthcare) it enables reproducing any past decision, answering 'why did the model predict this', and demonstrating compliance during audits. CI/CD systems capture lineage automatically via experiment tracking and registry metadata so every promoted model is fully attributable.

LLM outputs are open-ended and non-deterministic, so a single accuracy number cannot capture quality, safety, or regressions. CI/CD instead uses evaluation suites combining reference-based metrics, LLM-as-judge scoring on curated prompt sets, rubric/criteria checks, hallucination/groundedness and safety tests, plus regression comparison against the current production version. Pinning model/prompt versions and running these eval suites as gates makes GenAI releases testable and reproducible.

A rollback strategy lets you quickly revert to a previously known-good model when a new release degrades metrics or causes incidents. ML rollbacks are harder because the 'good' state depends on a specific model+data+feature-pipeline combination, and the new model may have already altered downstream data (feedback loops) or feature definitions. Mitigations include keeping prior model versions hot in the registry, versioning feature transforms alongside models, and automated metric-based auto-rollback on canaries.

Each step's output is keyed by a hash of its inputs and code, so if nothing changed, the cached result is reused instead of re-executing the step. This dramatically speeds up reruns of long pipelines (e.g., skipping expensive data prep when only the training code changed) and saves compute cost. It also enforces deterministic dependency tracking, since a changed input invalidates that step and all downstream steps.

A data contract is an agreed, versioned specification of a dataset's schema, semantics, quality SLAs, and ownership between data producers and ML consumers. It prevents upstream changes (renamed columns, changed units, new nulls) from silently breaking models downstream. They are enforced in CI/CD by automated schema/quality validation at ingestion that fails the pipeline or blocks publishing when the contract is violated, with alerts to the producing team.

Monitoring closes the feedback loop: production metrics (performance, drift, latency, data quality) are the very signals that trigger continuous training and gate the next release. Without monitoring there is no automated way to detect degradation, so retraining becomes blind scheduling or manual guesswork. Tying monitoring to CT/CD makes the system self-correcting, automatically retraining and redeploying when observed quality crosses thresholds.

In many systems true labels arrive late or never (e.g., loan default after months), so you cannot compute live accuracy in real time to gate or trigger retraining. This is addressed with proxy signals: monitor input/prediction drift and prediction-distribution shifts as early warnings, sample-and-label a subset for delayed evaluation, and use business KPIs as indirect quality measures. Retraining triggers then combine drift-based proxies with periodic batch evaluation once labels mature.

CI for the pipeline builds, unit/integration-tests, and packages the pipeline components and their container images whenever pipeline source code changes. CD of the pipeline then deploys that tested pipeline definition to the target environment, where it can run continuous training. In other words, the deployment artifact is the training pipeline itself, not just a single model, enabling rapid, reliable iteration on the ML system's logic.

Mutable tags such as 'latest' can be overwritten, so the artifact a deployment resolves to may silently change between runs, breaking reproducibility and complicating rollback and audits. Pinning by content digest/hash guarantees the exact same bytes are deployed every time and that lineage records point to an unambiguous artifact. This prevents 'it worked yesterday' incidents caused by a moving tag.

In champion-challenger, the existing champion model and one or more challengers serve disjoint, statistically meaningful slices of live traffic so you can compare their effect on business KPIs with an A/B test before promoting a winner. A canary, by contrast, is primarily a safe rollout mechanism: a small traffic share to detect failures/regressions before full rollout, not a controlled experiment for measuring causal business lift. A/B answers 'which model is better for the business'; canary answers 'is the new model safe to release'.

Use a lockfile that pins exact, hash-verified versions of all transitive dependencies (e.g., pip-tools/uv lock, Poetry, or conda-lock) rather than loose version ranges, and build inside pinned base container images. CI installs strictly from the lockfile so every run resolves identical packages, and dependencies are updated deliberately via a regenerate-and-test step. This prevents silent breakage from upstream releases of frameworks like NumPy, PyTorch, or scikit-learn.

pip-compile --generate-hashes requirements.in
pip install --require-hashes -r requirements.txt