Model registry and versioning

Experiment tracking logs individual training runs—hyperparameters, metrics, and artifacts—in a transient, exploratory context. A model registry stores curated, production-candidate artifacts with lifecycle stages, versioning, immutability guarantees, and governance controls. The registry is downstream of tracking: you promote the best run's artifact into it after evaluation.

The three stages are Staging (integration and QA testing), Production (live serving), and Archived (deprecated, no longer active). Each model version can independently occupy a stage, enabling a sequential promotion pipeline from training output to live service. Note: MLflow 2.x+ deprecated these fixed stages in favor of named aliases, which offer greater flexibility for multi-version deployment patterns—professionals should prefer aliases for new work.

Pass `registered_model_name` directly to the logging call (e.g., `mlflow.sklearn.log_model(..., registered_model_name='MyModel')`), or call `mlflow.register_model(model_uri, name)` after the run. Both create a new version under the named registered model; the URI uses the `runs:/<run_id>/model` scheme.

mlflow.register_model(
    model_uri=f"runs:/{run_id}/model",
    name="FraudClassifier"
)

SemVer communicates API compatibility, not performance regressions or data drift. ML versions must also record the training dataset URI and content hash, evaluation metrics snapshot, feature schema, framework and library versions, and the originating experiment run ID so that behavior can be reproduced and audited independently of the version number.

DVC stores large artifacts in a remote storage backend (S3, GCS, Azure Blob, etc.) and commits lightweight `.dvc` pointer files to Git. A `git checkout` of a specific tag combined with `dvc pull` reproducibly retrieves the exact model artifact for that commit, coupling code and model versions atomically without bloating the repository.

dvc add models/classifier.pkl
git add models/classifier.pkl.dvc .gitignore
git commit -m "chore: add model v1"
dvc push

Model lineage is the traceable chain from raw data through preprocessing, training run, registered model version, and deployment. Registries enable it by linking each version to its source run, which logs dataset paths and hashes, code commit SHAs, and hyperparameters—making any production artifact fully reproducible and auditable without manual documentation.

The champion is the current Production version; one or more challengers are newer versions deployed alongside it receiving a small traffic slice. Both versions coexist in the registry simultaneously (via aliases like 'champion' and 'challenger'). When challenger metrics surpass the champion's over a sufficient sample, it is promoted to champion and the former champion is archived—all without retraining.

A model card is a structured document describing intended use, performance metrics across demographic subgroups, known limitations, training data provenance, and ethical considerations. Storing it in the registry ensures every version has associated documentation visible to consumers and auditors, enabling informed deployment decisions and satisfying regulatory requirements such as the EU AI Act.

Stages were global, fixed to three names, and prevented multiple concurrent production variants (e.g., two regional models both in 'Production'). Aliases are arbitrary named pointers—'champion', 'shadow', 'blue', 'green', 'canary'—that can be assigned without constraint to any version, enabling flexible multi-version deployment patterns and custom promotion vocabularies.

client.set_registered_model_alias(
    name="RevenueModel",
    alias="champion",
    version="14"
)

After retraining, register the model as a new version and run an evaluation job that compares held-out metrics against the current production baseline. Use the registry client API to set the 'production' alias only if accuracy exceeds the baseline by a threshold and fairness constraints pass; otherwise the pipeline fails and the version remains unaliased. This can be encoded in GitHub Actions, Vertex AI Pipelines, or SageMaker Pipelines.

client = MlflowClient()
if new_auc > prod_auc + 0.005 and max_bias < 0.05:
    client.set_registered_model_alias(
        name="FraudModel", alias="production", version=str(new_ver)
    )
else:
    raise ValueError("Promotion gate failed: metrics below threshold")

Vertex AI integrates model deployment with Vertex Endpoints natively, enforces IAM-based access control via Google Cloud's resource hierarchy, and handles multi-region artifact replication in a single API call. It emits audit logs automatically through Cloud Audit Logs. MLflow's governance relies on the operator to configure RBAC, artifact storage backend replication, and audit logging separately.

A model package group is a container for all versioned model packages (versions) of a related model. Each package has an approval status: PendingManualApproval, Approved, or Rejected. EventBridge rules or SageMaker Pipelines listen for status changes and automatically trigger endpoint updates or block deployment, enabling human-in-the-loop governance before any version reaches production.

Immutability means once a model version's artifact is written to the registry it cannot be overwritten—only new versions may be created. This guarantees that any historical prediction can be reproduced exactly by retrieving the same version, satisfying requirements from regulations like GDPR's right to explanation, the Federal Reserve's SR 11-7 model risk management guidance, and the EU AI Act's auditability mandates.

Rollback re-aliases the previous known-good version to the 'champion' (or 'production') alias and updates the serving infrastructure to resolve that alias. No retraining is needed because the artifact is immutably stored. The serving layer pulls the updated alias pointer and loads the older artifact, completing the rollback in seconds.

client.set_registered_model_alias(
    name="FraudDetector",
    alias="champion",
    version="7"  # previous stable version
)

Tag the new version with a 'shadow' alias while the current version holds 'champion'. Route 100% of live traffic to the champion for actual responses, but asynchronously mirror each request to the shadow model and log its predictions without surfacing them to users. Compare shadow metrics against champion over time; if shadow consistently wins, swap aliases. The registry stores both versions immutably throughout the evaluation period.

Essential metadata includes: source run ID and experiment ID, Git commit SHA of training code, training and validation dataset URIs with content hashes, Python and framework library versions, hyperparameters, evaluation metrics, input/output feature schema (column names and types), and the identity of the pipeline or user that registered the version. Together these form a complete audit trail.

When registering a model, the registry records the exact feature view name and version used during training. At serving time, the inference pipeline retrieves features using the same feature view version, guaranteeing identical transformation logic between training and inference. Without this coupling, independently evolving transformation code in the serving pipeline can introduce silent skew that degrades model accuracy.

LLMs introduce multi-gigabyte artifacts that stress storage and replication. Prompt templates change model behavior without touching weights, requiring first-class prompt versioning. A single checkpoint may have multiple quantization variants (FP16, INT8, GGUF) serving different hardware. LoRA adapters are fine-tuning deltas separate from base weights. Registries must support hierarchical versioning (base model → fine-tune → quantization) and store prompt templates as versioned artifacts.

The pyfunc flavor wraps any model—sklearn, XGBoost, PyTorch, or custom—behind a uniform `predict(data)` interface. When loaded from the registry via `mlflow.pyfunc.load_model(uri)`, the caller uses one consistent API regardless of framework. The logged `MLmodel` manifest specifies which loader (flavor) handles deserialization, making downstream serving infrastructure fully framework-agnostic.

model = mlflow.pyfunc.load_model("models:/FraudClassifier/champion")
predictions = model.predict(inference_df)

Register the new version with a 'canary' alias. Configure the serving layer (Kubernetes Ingress weight, Istio VirtualService, or a feature flag) to route 5-10% of traffic to the canary endpoint while 90-95% goes to the 'champion' version. Monitor canary metrics (latency, accuracy, business KPIs); if healthy, incrementally shift traffic and finally swap aliases. If degraded, instantly revert traffic to champion by updating the alias.

Webhooks fire HTTP POST callbacks to configured endpoints when registry events occur—version created, stage transitioned, alias changed, or version deleted. A concrete use case: when a model version is promoted to Production, the webhook triggers a Slack notification to the platform team and simultaneously invokes a Lambda function that updates the SageMaker endpoint, eliminating polling and enabling event-driven deployment automation.

After training in PyTorch or TensorFlow, export to ONNX and log it as a secondary artifact alongside the native format in the same registry version. The native checkpoint is retained for fine-tuning and retraining; the ONNX artifact is served by ONNX Runtime across CPUs, GPUs, and edge devices without any framework installation. This dual-artifact pattern decouples training framework from serving runtime.

torch.onnx.export(model, dummy_input, "model.onnx", opset_version=21)
mlflow.log_artifact("model.onnx", artifact_path="onnx")

Azure ML allows registering an `Environment` object—specifying a Docker base image, conda YAML, or pip requirements—alongside the model in the registry entry. When the model is deployed to an endpoint, Azure ML uses the pinned environment to build the inference container, guaranteeing the same Python version, library versions, and OS dependencies that were used during evaluation, preventing environment drift between validation and production.

Tags are key-value metadata labels attached to a model or version for filtering and governance (e.g., `{'team': 'fraud', 'dataset': 'v3', 'compliant': 'true'}`); they do not affect request routing. Aliases are named pointers that serving infrastructure resolves to a specific version at query time (e.g., 'champion' → version 12). Use tags for discovery and policy enforcement; use aliases as dynamic deployment targets that update without redeploying serving infrastructure.

RBAC assigns roles such as data-scientist (can register versions and transition to Staging only), ML-engineer (can transition to Production), and admin (full lifecycle control including deletion). Registry APIs enforce role checks on every state transition so that neither automated pipelines nor individuals can bypass approval gates. Cloud registries integrate with IAM (AWS IAM, GCP IAM, Azure RBAC) for centralized identity management.

A model package bundles the serialized artifact, inference code (preprocessing and postprocessing), environment specification (Docker image or conda YAML), input/output schema, and metadata into a single deployable unit. This shift-left packaging at registration time means any environment—staging, edge, cloud—can deploy the package without reconstructing the serving pipeline, reducing integration errors between data science and engineering teams.

Store the base model once as a dedicated parent registry entry. Register each LoRA adapter as a separate lightweight version, recording the parent base model name and version in the adapter's metadata tags. At inference, load the base checkpoint and merge the adapter delta at runtime. This avoids duplicating the multi-GB base weights for every fine-tune while preserving full versioning and rollback capability for adapters independently.

mlflow.log_artifacts("lora_adapter/")
mlflow.set_tag("base_model_name", "llama-3.1-8b")
mlflow.set_tag("base_model_version", "1")
mlflow.set_tag("adapter_type", "lora_r16")

Co-registration links a specific dataset version—URI plus content hash or a DVC tag—to the model version at registration time. If model performance degrades in production, engineers can trace back to the exact data the model was trained on and determine whether the root cause is data quality, distribution shift, or code change. Without co-registration, reproducing the training environment requires manual archaeology.

When a version is promoted to Production, the registry emits a webhook or EventBridge event containing the model name, version, endpoint, and feature schema. A listener service consumes this event and automatically provisions monitoring jobs in tools like Evidently AI, WhyLabs, or SageMaker Model Monitor, scoped to the new version's schema and baseline statistics. This closes the observability loop without manual post-deployment setup.

'Evaluate before register' runs full offline evaluation in the training pipeline before creating a registry entry; only models passing all gates appear in the registry, keeping it clean and preventing clutter. 'Register then evaluate' creates a Staging version first, then runs evaluation and promotes or rejects it—preferred when evaluation requires registry-integrated tooling such as comparison dashboards, human review interfaces, or multi-version A/B evaluation pipelines.

Each Hub repository is a Git repository; large weight files are stored in Git LFS while model cards, config.json, and tokenizer files are regular Git objects tracked in history. Model versions are Git commits or tags; the `revision` parameter in `from_pretrained` pins loading to a specific commit SHA or tag, enabling reproducible retrieval of any historical checkpoint without maintaining a separate versioning layer.

model = AutoModel.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    revision="v1.0.0"  # tag or commit SHA
)

Key challenges are artifact replication latency (a promoted model may not be available in all regions simultaneously), consistency (an alias swap must appear atomically before traffic is rerouted), and data residency compliance (EU-trained models may be prohibited from replicating outside the EU). Solutions include using globally replicated cloud registries (Vertex AI, SageMaker multi-region), enforcing immutable content-addressed artifact IDs with region-local caches, and tagging versions with allowed-region metadata for data residency enforcement.

After automated evaluation promotes a model to Staging, the pipeline suspends and dispatches a review notification (Slack, email, Jira ticket) with a deep link to the registry version's metrics dashboard and model card. A designated approver inspects subgroup performance, bias reports, and business metrics, then approves or rejects via the registry UI or API. The pipeline resumes and transitions to Production only upon approval; a rejection triggers an alert and archives the version.

A namespace hierarchy such as `{team}/{project}/{model_name}` organizes the registry like a filesystem. It prevents name collisions across teams, enables team-scoped RBAC so team A cannot modify team B's models, and allows policy enforcement at the namespace level—for example, all entries under `/risk/` require dual approval and undergo quarterly reviews. It also makes discovery via search or CLI intuitive as model count grows.

Each candidate model version is registered in the registry and assigned a distinct alias (e.g., 'variant-a', 'variant-b'). The A/B testing framework routes user cohorts to specific aliases using consistent hashing on user ID, ensuring reproducibility of assignments. Prediction logs are tagged with the model version alias; an analysis job reads these logs to compute conversion rates, revenue lift, or metric deltas per version with statistical significance tests, and the winner's alias is swapped to 'champion'.

In a RAG system the registry must version not only the LLM (or fine-tuned adapter) but also the embedding model (since embedding space changes break index compatibility), the vector index itself (a snapshot or version pointer to the FAISS/Pinecone index), the retrieval pipeline configuration (chunk size, top-k, reranker model), and the system prompt template. Changing any component without versioning the others breaks reproducibility and makes debugging regressions extremely difficult.