Shadow and canary deployment

Shadow deployment (also called traffic mirroring or dark launch) routes a copy of live production traffic to a new model version simultaneously with the production model, but only the production model's responses are returned to users. The shadow model processes requests in parallel without affecting user experience, allowing teams to compare outputs, latency, and resource usage before promoting the new model. It is ideal for validating ML models on real-world data without any user-facing risk.

Canary deployment gradually shifts a small percentage of live traffic to a new model or service version, monitoring key metrics before expanding the rollout. The name derives from the 'canary in a coal mine' analogy — miners used canaries as early warning systems for toxic gases, just as the canary group detects problems before they affect all users. Unlike shadow deployment, real users in the canary group see new responses, making it a limited-blast-radius production test.

In shadow deployment, no real user sees the new model's output — responses from the candidate model are discarded or logged for offline analysis only, giving zero user-facing risk. In canary deployment, a subset of real users receives the new model's actual responses, meaning errors or regressions directly impact those users. Shadow deployment is preferred for pre-release validation when correctness is uncertain; canary is preferred when you have sufficient confidence and need real user feedback on live response quality.

Teams should define primary success metrics (e.g., CTR, conversion rate) and pre-specify alpha (typically 0.05) and power (≥0.80) to compute the minimum detectable effect and required sample size before starting the experiment. The canary must remain active until that sample size is reached to avoid peeking bias and false positives. Tools like Bayesian sequential testing or CUPED variance reduction can shorten experiment duration while maintaining statistical validity.

Native Kubernetes achieves approximate traffic splits by scaling replica counts proportionally: running 1 canary replica alongside 9 stable replicas under the same Service selector gives roughly 10% canary traffic. For example, labeling 9 pods 'version: stable' and 1 pod 'version: canary', with the Service selecting both via a shared 'app: mymodel' label. This is coarse-grained; a service mesh (Istio, Linkerd) or ingress controller (NGINX, Traefik) should be used for exact percentage-based splitting.

# Stable deployment: 9 replicas
apiVersion: apps/v1
kind: Deployment
metadata: {name: model-stable}
spec:
  replicas: 9
  selector: {matchLabels: {app: model, version: stable}}
---
# Canary deployment: 1 replica
apiVersion: apps/v1
kind: Deployment
metadata: {name: model-canary}
spec:
  replicas: 1
  selector: {matchLabels: {app: model, version: canary}}
---
# Service selects both via shared 'app: model' label
apiVersion: v1
kind: Service
spec:
  selector: {app: model}  # matches stable + canary pods

Shadow deployment logs a candidate model's predictions on live traffic alongside eventual ground truth labels (e.g., user clicks, purchase events, fraud outcomes) to build a dataset that reflects the real production distribution. This avoids the train-serve skew problem common with offline datasets. The collected shadow predictions and ground truth enable offline evaluation against the same distribution before any rollout, and feed back into the training pipeline for next-generation models.

Feature flags decouple model deployment (artifact release) from traffic activation, enabling instant rollback without redeployment — simply toggle the flag to redirect all traffic back to the stable model. They allow targeting specific user cohorts (beta users, regions, device types) for the canary, which is more flexible than percentage-based routing alone. Systems like LaunchDarkly, Unleash, or custom flag services integrate with model servers so canary exposure is controlled at runtime without infrastructure changes.

Istio uses a VirtualService with weighted routing rules referencing two DestinationRule subsets — 'stable' and 'canary' — to route exact percentages of HTTP/gRPC traffic at Layer 7. For instance, a VirtualService assigns 95 weight to the stable subset and 5 weight to the canary subset. This supports header-based routing for deterministic user pinning and integrates with Prometheus for metric scraping, enabling automated canary analysis via Flagger or Argo Rollouts.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: model-vsvc
spec:
  hosts: ["model-svc"]
  http:
  - route:
    - destination:
        host: model-svc
        subset: stable
      weight: 95
    - destination:
        host: model-svc
        subset: canary
      weight: 5
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: model-dr
spec:
  host: model-svc
  subsets:
  - name: stable
    labels: {version: stable}
  - name: canary
    labels: {version: canary}

The fastest rollback is to set the canary traffic weight to 0 (in Istio, Argo Rollouts, or a feature flag) without deleting the canary deployment, preserving the artifact for post-mortem debugging. Automated canary analysis tools (Flagger, Argo Rollouts) can trigger this rollback automatically when a metric such as P99 latency exceeds a threshold for a configurable duration. After rollback, the team should profile the model for computational bottlenecks (unoptimized ops, missing TorchScript compilation, memory layout issues) before re-attempting the canary.

Key metrics include: (1) model-level — prediction latency (P50/P95/P99), throughput, prediction score distribution, and feature drift (PSI/KL divergence); (2) business-level — fraud recall, false positive rate, and dollar value of approved fraudulent transactions; (3) infrastructure-level — CPU/memory/GPU utilization and error rate (HTTP 5xx, timeouts). Comparing these metrics between canary and stable populations with statistical tests determines promotion readiness. Alerting on anomalous score distributions can catch silent model failures that do not surface as HTTP errors.

Shadow scoring runs a new model in parallel with the production model on live traffic and compares predictions across demographic subgroups (age, gender, race proxies) before deployment. Because users are never exposed to shadow outputs, fairness testing is conducted on the actual production distribution — not a curated evaluation set that may underrepresent edge groups — catching disparate impact issues pre-launch. This pattern is especially valuable in regulated domains (credit, hiring, healthcare) where fairness is a compliance requirement.

Personalization models require consistent experience — a user should not toggle between stable and canary responses per request, as this could generate conflicting recommendation state. Service meshes solve this using consistent hashing on a user identifier (cookie or user-id header) as the routing key, ensuring all requests from a given user always route to the same model version. In Istio, this is achieved with a consistentHash load balancing policy in the DestinationRule or header-based match in the VirtualService.

Argo Rollouts is a Kubernetes controller that extends the native Deployment resource with advanced delivery strategies including canary and blue-green. It integrates with metric providers (Prometheus, Datadog, New Relic) through an AnalysisTemplate CRD that defines success and failure criteria; the controller automatically pauses, promotes, or aborts the rollout based on metric evaluation. For ML deployments, AnalysisTemplates can query custom metrics like prediction drift, model accuracy on shadow labels, or business KPIs.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 10m}
      - analysis:
          templates:
          - templateName: p99-latency-check
      - setWeight: 50
      - pause: {duration: 30m}
      - setWeight: 100

Train-serve skew occurs when the features seen at training differ from those at serving time (encoding differences, missing value handling, timestamp leakage). In shadow deployments, both the production and candidate models must consume features from the same serving pipeline or feature store (Feast, Tecton, Vertex Feature Store) so any prediction difference reflects only the model. Additionally, shadow logs should capture raw features alongside predictions so offline debugging can distinguish skew from genuine model behavioral differences.

Classical ML outputs (classification scores, regression values) are easily compared programmatically using KL divergence, PSI, or RMSE. LLM outputs are open-ended text, making automated equivalence testing fundamentally harder — comparison requires LLM-as-judge scoring, semantic similarity metrics (BERTScore, ROUGE), or human preference evaluation, all of which are slower, costlier, and noisier than scalar metric comparisons. Shadow deployment for LLMs must therefore pair traffic mirroring with a robust offline evaluation harness including both automated and human-in-the-loop review.

A pause step halts the canary rollout at a given traffic percentage until a human explicitly approves continuation (via GitOps PR, Argo Rollouts approval, or PagerDuty integration), giving engineers time to review dashboards and business metrics requiring human judgment. An automatic promotion step uses pre-defined metric thresholds to advance traffic without human intervention, suitable when metrics are well-understood and reliable. Best practice is to combine both: automate metric-gated intermediate promotions (5% → 20% → 50%) but require human approval before the final 100% step.

An SRE error budget is the allowed fraction of failure (1 - SLO) over a rolling window; a canary deployment gate can be configured to abort if the canary's error rate would consume a defined fraction of the remaining error budget. This ties deployment decisions to reliability commitments rather than arbitrary thresholds — a nearly exhausted error budget justifies aborting even a small regression, while a full budget may tolerate a slightly higher canary error rate. Tools like Nobl9 and Sloth expose SLO burn rate metrics that Flagger or Argo Rollouts can query as AnalysisTemplate metrics.

A MAB dynamically reallocates traffic toward the better-performing variant in real-time, maximizing cumulative reward (e.g., conversions) during the exploration period — unlike canary, which uses a fixed split and purely measures without optimizing. The trade-off is that MAB produces statistically noisier comparisons because it simultaneously optimizes and explores, making it harder to attribute performance differences to the model versus traffic changes. Canary with fixed splits provides cleaner causal inference; MAB is preferred when minimizing regret during the test period outweighs the need for clean statistical isolation.

Progressive delivery is a superset of continuous delivery that releases features to increasingly larger audiences using risk-controlled mechanisms: feature flags, canary deployments, A/B tests, and ring-based rollouts. Shadow deployment acts as a pre-progressive-delivery gate — validating the candidate model on real traffic before any user exposure. Canary deployment is the first progressive delivery step, limiting blast radius, followed by successive ring expansions through full promotion. Platforms like Flagger, Argo Rollouts, and LaunchDarkly orchestrate this entire pipeline.

A/B testing intentionally exposes two user groups to different experiences to measure causal effect on business outcomes — real users in group B experience a different treatment. Shadow deployment never changes user experience, so it cannot measure causal business impact (e.g., user engagement with a new recommendation) but incurs zero user risk. Additionally, A/B testing may require user consent under GDPR/CCPA in personalization contexts, while shadow deployment does not expose users to different treatment and typically falls outside the same consent scope.

Dark launch (sometimes used interchangeably with shadow deployment) exercises a new code path or model using live traffic before it goes live, with all outputs suppressed from users. In ML serving, a dark launch may load the candidate model in the serving process and run inference on every request, returning only the production model's output — validating correctness, memory footprint, and latency before any routing change. Some teams distinguish dark launch (invoked within the same process) from shadow deployment (separate service receiving mirrored traffic), but the user-impact principle is identical.

Shadow evaluation routes user prompts to both the current production LLM and a candidate (new model or updated system prompt) in parallel, logs both responses, then asynchronously scores them using an LLM judge (e.g., GPT-4 or Claude as evaluator) on dimensions like helpfulness, accuracy, and safety. Because LLM-as-judge evaluation is asynchronous, it does not add latency to users. Aggregate win-rate statistics across thousands of shadow pairs determine whether the candidate is ready for canary rollout, forming a data-driven promotion gate.

PSI measures distribution shift between a reference distribution (stable model predictions) and the current distribution (canary model predictions). PSI < 0.1 indicates no significant shift; 0.1–0.2 warrants investigation; >0.2 signals a major distribution change. During canary deployment, if the canary model's prediction score distribution shows PSI > 0.2 relative to the stable model, it suggests the model is behaving differently on production traffic — either due to a bug, data drift, or intentional architectural change — requiring scrutiny before full rollout.

import numpy as np

def compute_psi(expected: np.ndarray, actual: np.ndarray, buckets: int = 10) -> float:
    """PSI > 0.2 triggers canary investigation."""
    breakpoints = np.linspace(0, 1, buckets + 1)
    exp_pct = np.histogram(expected, bins=breakpoints)[0] / len(expected)
    act_pct = np.histogram(actual, bins=breakpoints)[0] / len(actual)
    # Avoid log(0)
    exp_pct = np.clip(exp_pct, 1e-4, None)
    act_pct = np.clip(act_pct, 1e-4, None)
    return float(np.sum((act_pct - exp_pct) * np.log(act_pct / exp_pct)))

# Usage during canary monitoring
psi_score = compute_psi(stable_scores, canary_scores)
if psi_score > 0.2:
    alert("Major prediction distribution shift in canary — investigate before promoting")

Shadow deployment doubles per-request GPU compute: every production inference request triggers an identical shadow inference. For GPU-intensive models, this requires either dedicated GPU capacity for the shadow service or fractional GPU allocation via NVIDIA MIG on A100/H100 GPUs. Shadow requests must be dispatched asynchronously (fire-and-forget) with their own queue, ensuring production P99 latency is unaffected even if the shadow service is saturated. Cost governance is critical — GPU shadow deployment can double inference costs for the entire validation period.

Seldon Core provides a native shadow field in its SeldonDeployment CRD that mirrors a configurable percentage of traffic to a shadow predictor alongside the production predictor. The routing is handled by the Seldon Engine sidecar — it sends the request to both, returns only the production response to the caller, and logs the shadow response for comparison. Combined with Seldon's Alibi Detect integration, shadow predictions can be automatically scored for data drift, outlier detection, and concept drift before promotion.

Canary analysis is the continuous evaluation of a canary deployment against defined success criteria (latency, error rate, custom model metrics) throughout the progressive traffic shift. Flagger, a Kubernetes operator, automates this by scraping Prometheus metrics at configurable intervals, comparing canary versus baseline values, incrementing traffic weight on success, or rolling back on failure. It also supports webhook integrations for load testing (triggering k6 or Vegeta during the canary) and manual approval gates, making promote-or-rollback decisions programmable and auditable.

The safest approach is the expand-contract pattern: first deploy the migration that adds the new column with a nullable default without changing application code, letting all replicas (stable and canary) continue working. The canary model then reads the new column during its test period. Only after full canary promotion is the contract phase executed — removing the old column and making the new one non-nullable. This ensures backward compatibility between stable and canary versions that must coexist in the database simultaneously.

-- Step 1 (expand): Add nullable column, deploy canary alongside stable
ALTER TABLE predictions ADD COLUMN new_feature FLOAT;

-- Step 2: Canary period — both stable and canary read their respective columns
-- Stable reads old_feature; canary reads new_feature

-- Step 3 (contract): After full promotion
ALTER TABLE predictions ALTER COLUMN new_feature SET NOT NULL;
ALTER TABLE predictions DROP COLUMN old_feature;

Load balancer mirroring (Nginx mirror directive, Envoy request_mirror_policy, AWS ALB mirroring) duplicates incoming HTTP requests at the network layer without modifying application code, enabling shadow deployment of any service transparently. Application-level shadow deployment requires the serving code to explicitly fan out requests to both models, giving finer control (conditional mirroring, sampling rate, payload transformation) but coupling shadow logic to the application. Load balancer mirroring is simpler to deploy and vendor-neutral; application-level is more flexible for complex ML workflows with pre/post-processing pipelines.

# Envoy request_mirror_policy example
route:
  cluster: model-stable
  request_mirror_policies:
  - cluster: model-shadow
    runtime_fraction:
      default_value:
        numerator: 10      # Mirror 10% of traffic
        denominator: HUNDRED
      runtime_key: routing.request_mirror_fraction

When shadow and production models use different feature versions (different time windows, aggregation methods), the shadow service must call the feature store with the candidate model's own feature view references or point-in-time parameters, not share the production model's cached feature vector. Feature stores like Feast support versioned feature views; the shadow service specifies its own feature reference set. Logging both the production and shadow feature vectors alongside predictions is essential for debugging whether discrepancies are intentional (feature difference) or due to a lookup bug.

In Argo Rollouts, the baseline is an optional replica of the stable deployment serving the same traffic percentage as the canary, enabling apples-to-apples metric comparison at equal traffic levels. Without a baseline, canary metrics are compared against the full stable deployment processing a different request mix, introducing confounding. For ML deployments, baseline comparison is critical because model performance often varies with traffic distribution — comparing a 5% canary cohort against a 95% stable cohort may falsely penalize or excuse the canary model.

Sampling-based models produce non-deterministic outputs, making direct output comparison between shadow and production unreliable. To ensure reproducible shadow evaluation, fix the random seed per request (passing the same seed to both models) or use deterministic decoding (temperature=0, greedy) specifically for shadow comparison runs while preserving stochastic behavior in production. Alternatively, evaluate aggregate statistical properties (mean token length, refusal rate, topic distribution) rather than individual outputs, using sufficiently large shadow sample sizes to detect systematic differences.

TensorFlow Serving's model server can load multiple named model versions simultaneously using a ModelServerConfig or version policy. By loading both 'stable' (version N) and 'canary' (version N+1), the serving binary routes requests to the appropriate version based on the request's model_spec.version field. For shadow deployment, the gateway layer duplicates each incoming request — sending one to stable for the user response and one to canary as a fire-and-forget call — sharing the TF runtime to save memory compared to two separate serving processes.

# ModelServerConfig loading two versions simultaneously
model_config_list {
  config {
    name: "fraud_model"
    base_path: "/models/fraud_model"
    model_version_policy {
      specific { versions: 42   # stable
                 versions: 43 } # canary
    }
  }
}

# Gateway fire-and-forget shadow call (Python)
import asyncio

async def serve(request):
    stable_task = asyncio.create_task(call_tf_serving(request, version=42))
    asyncio.create_task(call_tf_serving(request, version=43))  # shadow, fire-and-forget
    return await stable_task

Long-delay conversion metrics are incompatible with short canary windows. The solution is to use short-lag proxy metrics highly correlated with long-term conversion (e.g., day-7 retention, feature adoption, email engagement) as the canary gate, calibrated against historical proxy-to-revenue relationships. Additionally, maintain a held-out cohort from the canary exposure tracked to 30-day conversion as a lagging post-deployment validation. Surrogate modeling of long-term outcomes from short-window signals is an active research area at companies like Netflix and Spotify.

Blue-green deployment maintains two identical environments with an instantaneous DNS or load-balancer cutover from old to new — the entire user base switches atomically, enabling instant rollback via DNS flip. Canary deployment progressively shifts traffic percentage-by-percentage with metric-gated promotion, limiting blast radius. Blue-green is preferred for well-tested urgent releases (hotfixes) where instant switchover matters; canary is preferred for ML models with uncertain real-world performance. Blue-green requires double infrastructure during transition; canary requires a service mesh or smart load balancer for traffic splitting.