Shadow and canary deployment quiz

Question 1

What is shadow deployment in the context of ML model serving?

Accepted Answer

Shadow deployment mirrors live traffic to a new model without affecting user-facing responses — only the production model's output is returned

Answer

Shadow deployment returns responses from both old and new models to users simultaneously

Answer

Shadow deployment is another name for blue-green deployment where traffic switches atomically

Answer

Shadow deployment only works with batch inference pipelines running overnight

Question 2

What is canary deployment and why is it named 'canary'?

Accepted Answer

Canary deployment gradually shifts a small percentage of live traffic to a new version, named after the coal-mine safety practice of using canaries as early warning signals

Answer

Canary deployment is named after the Canary Islands where Netflix engineers first developed the pattern

Answer

Canary deployment sends all traffic to a new version and rolls back automatically if errors exceed a threshold

Answer

Canary deployment is identical to A/B testing with an equal 50/50 traffic split

Question 3

What is the key operational difference between shadow deployment and canary deployment?

Accepted Answer

In shadow deployment, no real user sees the candidate's output; in canary, a subset of real users receive and experience the new model's actual responses

Answer

Shadow deployment uses 50% traffic while canary uses only 5% by convention

Answer

Shadow deployment runs entirely offline; canary deployment runs online

Answer

Canary deployment logs responses without serving them to users, just like shadow deployment

Question 4

How should teams determine statistical significance when evaluating a canary deployment for an ML recommendation model?

Accepted Answer

Pre-specify alpha, power, and minimum detectable effect to compute required sample size before the canary, and avoid early stopping to prevent peeking bias

Answer

Always use p < 0.01 with a fixed 24-hour window regardless of traffic volume

Answer

Statistical significance is only needed for shadow deployments, not canary deployments

Answer

Use a chi-square test across all metrics simultaneously without multiple testing correction

Question 5

In Kubernetes, which approach correctly approximates a 10% canary traffic split using native primitives without a service mesh?

Accepted Answer

Run 1 canary replica alongside 9 stable replicas sharing the same Service selector label, achieving an approximate 10% split

Answer

Use a PodDisruptionBudget to limit stable replicas to 90% availability while canary takes the rest

Answer

Deploy canary pods to a separate Namespace and configure NetworkPolicy for the split percentage

Answer

Use a Kubernetes TrafficSplit CRD (SMI) to set weight 10 for canary and weight 90 for stable

Question 6

How can shadow deployment be used to collect labeled training data for continuous learning pipelines?

Accepted Answer

Shadow deployment logs candidate model predictions on live traffic alongside eventual ground truth labels, enabling offline evaluation and retraining that reflects the true production distribution

Answer

Shadow deployment cannot collect labels because users never interact with shadow outputs and no ground truth is generated

Answer

Shadow deployment trains the model online using live request-response pairs as they arrive

Answer

Shadow deployment is only useful for latency benchmarking and is not suited for data collection

Question 7

What is the role of feature flags in a canary deployment strategy for ML models?

Accepted Answer

Feature flags decouple model deployment from traffic activation, enabling instant rollback by flag toggle without redeployment, and allow targeting specific user cohorts for the canary

Answer

Feature flags require redeployment to update canary traffic percentages

Answer

Feature flags are incompatible with ML model serving because models are stateless artifacts

Answer

Feature flags should only be used in blue-green deployments, not progressive canary rollouts

Question 8

How does Istio implement precise canary traffic splitting for ML model inference services?

Accepted Answer

Istio uses a VirtualService with weighted route rules referencing DestinationRule subsets, assigning exact percentage weights to stable and canary subsets at Layer 7

Answer

Istio uses PodAntiAffinity rules in Deployments to distribute traffic across stable and canary replicas

Answer

Istio requires configuring separate Namespaces for stable and canary services with NetworkPolicy

Answer

Istio delegates traffic splitting to the Kubernetes Scheduler using resource requests and limits

Question 9

What is the recommended rollback strategy when a canary deployment for a deep learning model shows a P99 latency regression?

Accepted Answer

Set the canary traffic weight to 0 without deleting the deployment, preserving the artifact for post-mortem debugging, ideally triggered automatically by an AnalysisTemplate threshold

Answer

Delete the canary Deployment immediately to stop all traffic and recover production performance

Answer

Gradually reduce canary traffic over 24 hours to avoid sudden load shifts on the stable service

Answer

Redeploy the stable model version as a new Deployment to override the canary endpoint

Question 10

What monitoring metrics should be tracked during a canary deployment for a fraud detection ML model?

Accepted Answer

Canary monitoring for fraud models requires prediction latency, throughput, score distribution (PSI), feature drift, and business KPIs (recall, false positive rate) compared between canary and stable populations

Answer

Only HTTP error rate and P99 latency need to be monitored during a fraud detection canary

Answer

Only infrastructure metrics (CPU, memory, GPU) are relevant; business metrics are too volatile

Answer

Business metrics are too slow to compute during a canary window and should only be evaluated post-deployment

Question 11

What is 'shadow scoring' and how does it address model fairness concerns in regulated domains?

Accepted Answer

Shadow scoring runs a candidate model on live traffic and compares predictions across demographic subgroups without exposing users to different outputs, enabling fairness testing on the actual production distribution

Answer

Shadow scoring measures model latency across geographic regions to detect performance fairness

Answer

Shadow scoring is only applicable to NLP classification models, not tabular models used in credit scoring

Answer

Shadow scoring logs model confidence scores to detect when the model is uncertain about predictions

Question 12

How does a service mesh handle sticky sessions in canary deployments for ML personalization models?

Accepted Answer

Service meshes implement consistent hashing on a user identifier (cookie or user-id header) in the DestinationRule, ensuring all requests from a given user always route to the same model version

Answer

Sticky sessions are achieved by assigning dedicated IP address ranges to each model version pod

Answer

Canary deployments for personalization should avoid sticky sessions to ensure uniform random sampling

Answer

Sticky sessions in ML canary deployments require a StatefulSet instead of a standard Deployment

Question 13

What is Argo Rollouts and how does it automate canary analysis for ML model deployments?

Accepted Answer

Argo Rollouts is a Kubernetes controller that defines canary steps in a Rollout resource and evaluates Prometheus/Datadog metrics via AnalysisTemplates, automatically promoting or aborting based on configurable success criteria

Answer

Argo Rollouts is a feature of ArgoCD that handles only GitOps manifest synchronization

Answer

Argo Rollouts replaces Helm for templating and only supports blue-green deployments

Answer

Argo Rollouts requires Istio service mesh to implement any form of traffic splitting

Question 14

How do you prevent train-serve skew from contaminating shadow deployment comparisons?

Accepted Answer

Both models must consume features from the same serving pipeline or feature store so prediction differences reflect the model, not upstream data inconsistencies; log raw features alongside predictions for debugging

Answer

Use a separate feature pipeline for the shadow model to isolate it from any production dependencies

Answer

Retrain the shadow model exclusively on production logs to automatically eliminate skew

Answer

Train-serve skew is not a concern in shadow deployment because the shadow model never affects users

Question 15

What is the primary limitation of shadow deployment for validating generative AI (LLM) outputs compared to classical ML?

Accepted Answer

LLM outputs are open-ended text that cannot be compared with simple scalar metrics; evaluation requires LLM-as-judge, semantic similarity (BERTScore), or human review — making shadow comparison costlier and noisier than for classical ML

Answer

Shadow deployment is not possible for LLMs because transformer inference is stateful across tokens

Answer

The primary limitation is pure cost — LLMs always double cloud spend during any shadow evaluation period

Answer

Shadow deployment for LLMs must use exact string matching, which is too strict for meaningful comparison

Question 16

What is the difference between a 'pause step' and an 'automatic promotion step' in a canary rollout, and when should each be used?

Accepted Answer

Pause steps halt the rollout awaiting human approval; automatic steps advance based on pre-defined metric thresholds; combining both — automated intermediate steps plus human final approval — is best practice for ML deployments

Answer

Pause steps advance traffic automatically if no metric anomalies are detected; automatic steps require human sign-off

Answer

Pause steps are only used in blue-green deployments; automatic steps are exclusive to canary rollouts

Answer

Automatic steps immediately roll back the deployment if any single metric exceeds its threshold

Question 17

How does the SRE error budget concept integrate with canary deployment gates?

Accepted Answer

Canary gates can be configured to abort the rollout if the canary's error rate would consume a significant fraction of the remaining SLO error budget, linking deployment risk to reliability commitments rather than arbitrary fixed thresholds

Answer

Error budgets are only applicable to infrastructure services and cannot gate ML model canary deployments

Answer

Error budget consumption can only be calculated after 100% of traffic is shifted to the canary

Answer

Error budgets require a post-incident review to be filed before a canary can advance to the next step

Question 18

Why is a multi-armed bandit (MAB) considered an alternative to canary deployment for model selection, and what is its key trade-off?

Accepted Answer

MAB dynamically reallocates traffic to the better-performing model to minimize regret during the test, but produces statistically noisier comparisons than a fixed-split canary because it optimizes while exploring

Answer

MAB is identical to canary deployment but uses Bayesian inference rather than frequentist hypothesis testing

Answer

MAB requires a fixed 50/50 traffic split to correctly estimate reward distributions for each arm

Answer

MAB always outperforms canary deployment because it continuously adapts to observed performance signals

Question 19

What is 'progressive delivery' and how do shadow and canary deployments fit within it?

Accepted Answer

Progressive delivery is a superset of continuous delivery encompassing risk-controlled release mechanisms; shadow deployment validates the candidate before any user exposure, and canary progressively exposes it to real users in increasing percentages

Answer

Progressive delivery is a synonym for continuous integration with automated testing pipelines

Answer

Shadow deployment is the final step in progressive delivery, executed after full canary promotion

Answer

Progressive delivery only applies to web frontend deployments and is incompatible with ML model serving

Question 20

How does shadow deployment differ from A/B testing in terms of causality and regulatory considerations?

Accepted Answer

A/B testing measures causal business impact by intentionally exposing user groups to different experiences; shadow deployment cannot establish causal business impact but avoids user exposure and typically sidesteps the same consent requirements

Answer

Shadow deployment and A/B testing are equivalent in business impact measurement; only infrastructure implementation differs

Answer

A/B testing does not require user consent under GDPR because users are not aware of which variant they see

Answer

Shadow deployment requires more users than A/B testing to reach statistical significance for equivalent metrics

Question 21

What is the 'dark launch' pattern and how does it relate to shadow deployment in ML serving pipelines?

Accepted Answer

Dark launch invokes a candidate model on live traffic — either within the same serving process or via mirrored copy — suppressing its output from users; functionally equivalent to shadow deployment, though some teams distinguish in-process (dark launch) from separate-service (shadow) implementations

Answer

Dark launch always requires a completely separate model server from the production model server

Answer

Dark launch is used exclusively for A/B testing new UI components in frontend applications

Answer

Dark launch requires explicit user consent since user requests are processed by the new model

Question 22

In LLM deployment, what is 'shadow evaluation' and how does it leverage LLM-as-judge?

Accepted Answer

Shadow evaluation logs parallel LLM responses and asynchronously scores them with an LLM judge on helpfulness, accuracy, and safety, aggregating win-rate statistics to gate canary promotion decisions

Answer

Shadow evaluation for LLMs requires synchronous human raters for every request, making it impractical at production scale

Answer

Shadow evaluation uses BLEU and ROUGE scores as the sole comparison metric between production and candidate LLMs

Answer

LLM-as-judge evaluation must block the production response to ensure scoring completes before the next request

Question 23

How can Population Stability Index (PSI) function as a canary gate metric for ML model monitoring?

Accepted Answer

PSI compares prediction distributions between stable and canary models; PSI < 0.1 is stable, 0.1–0.2 warrants investigation, and PSI > 0.2 signals a major shift requiring canary investigation or abort

Answer

PSI > 0.1 always means the canary must be immediately aborted with no further analysis

Answer

PSI can only measure input feature drift and cannot be applied to model output prediction distributions

Answer

PSI is computed exclusively on numerical features and is inapplicable to classification probability outputs

Question 24

What infrastructure considerations are required to run shadow deployments for GPU-based deep learning inference?

Accepted Answer

Shadow deployment doubles per-request GPU compute, requiring dedicated or MIG-partitioned GPU capacity for the shadow service, asynchronous fire-and-forget dispatch to avoid production P99 impact, and explicit cost governance since inference costs can double during the validation period

Answer

Shadow deployment for GPU models requires no additional hardware if the candidate model is quantized to INT8

Answer

GPU shadow deployments can reuse the production GPU by batching shadow and production requests in a single kernel call

Answer

GPU shadow deployments must be synchronous to ensure bitwise consistency between production and shadow outputs

Question 25

How does Seldon Core's native shadow deployment type work for ML model validation?

Accepted Answer

Seldon Core provides a native shadow field in its SeldonDeployment CRD; the Seldon Engine sidecar mirrors traffic to a shadow predictor, returns only the production response, and logs shadow outputs for drift analysis via Alibi Detect

Answer

Seldon Core requires an Istio VirtualService to implement shadow routing between predictor versions

Answer

Seldon Core's shadow deployment blocks the production response until the shadow model finishes inference

Answer

Seldon Core shadow mode only works with TensorFlow Serving endpoints, not custom Python predictors

Question 26

What is canary analysis in Flagger and what does it automate for ML model rollouts?

Accepted Answer

Flagger continuously scrapes Prometheus metrics, automatically promotes traffic on success or triggers rollback on failure, with optional webhook-triggered load tests and manual approval gates for fully automated progressive delivery

Answer

Flagger requires manual metric evaluation for each canary step before advancing traffic

Answer

Flagger is only compatible with Istio service mesh and cannot use Linkerd or NGINX

Answer

Flagger performs canary analysis only after 100% traffic is shifted to the new version

Question 27

How should database schema migrations be handled in a canary deployment where the new ML model version requires an additional feature column?

Accepted Answer

Use the expand-contract pattern: add the new nullable column first in a separate migration, run the canary against both columns, then remove the old column only after full promotion to ensure stable-canary backward compatibility

Answer

Run the full schema migration atomically with the canary deployment to minimize the mixed-state window

Answer

Use a completely separate database for the canary model to avoid schema compatibility concerns during rollout

Answer

Deploy the canary first and run the schema migration after the canary stabilizes at 100% traffic

Question 28

How does request mirroring at the load balancer level differ from application-level shadow deployment for ML inference?

Accepted Answer

Load balancer mirroring (Nginx mirror, Envoy request_mirror_policy) duplicates requests at the network layer without application code changes; application-level shadow deployment provides finer control over sampling rates and payload transformation but couples shadow logic to the application

Answer

Load balancer mirroring modifies the HTTP response body before delivery to users, unlike application-level mirroring which suppresses shadow outputs

Answer

Application-level shadow deployment is always preferred because it has significantly lower network overhead than load balancer mirroring

Answer

Load balancer mirroring is proprietary to AWS ALB and not available in open-source proxies like Nginx or Envoy

Question 29

How should feature store lookups be handled in shadow deployment when the shadow model uses a different feature version than the production model?

Accepted Answer

The shadow service should call the feature store using the candidate model's own feature view references, logging both the production and shadow feature vectors alongside predictions to distinguish intentional differences from lookup bugs

Answer

The shadow model must always use identical features to the production model to ensure a valid head-to-head comparison

Answer

Feature store lookups should be shared between shadow and production services to reduce latency and cost

Answer

Different feature versions cannot be served simultaneously from a single feature store instance

Question 30

What is a 'baseline' in Argo Rollouts canary analysis and why is it critical for ML deployments?

Accepted Answer

A baseline is an optional replica of the stable model serving the same traffic percentage as the canary, enabling apples-to-apples metric comparison that eliminates traffic mix confounding — critical for ML models whose performance varies with request distribution

Answer

A baseline in Argo Rollouts is the minimum acceptable metric threshold value required for canary promotion

Answer

The baseline is the previous aborted canary version, retained for historical metric comparison in the dashboard

Answer

Argo Rollouts automatically creates a baseline by splitting existing stable Deployment pods into two equal groups

Test yourself