ML Observability

ML observability extends traditional software monitoring (uptime, latency, error rates) to cover the unique failure modes of ML systems. Traditional monitoring detects infrastructure failures; ML observability detects model degradation — when the model continues to respond (no errors) but produces increasingly wrong or biased predictions. Key dimensions: data quality monitoring (schema changes, null rates, distribution shifts in input features), model performance monitoring (prediction distribution drift, accuracy on labeled subsets), business metric monitoring (downstream KPIs that reflect model impact), and operational metrics (inference latency, throughput, cost per prediction). ML systems fail silently — a drifted model serves confident but wrong predictions without triggering any traditional alerts. Comprehensive ML observability requires purpose-built tools beyond standard Datadog/Prometheus stacks.

Data drift (also called covariate shift) is a change in the statistical distribution of input features over time compared to the training distribution. Detection methods: Kolmogorov-Smirnov (KS) test compares empirical CDFs of a feature's training vs. production distributions — suitable for continuous features. Population Stability Index (PSI) measures distribution change across buckets: PSI < 0.1 (no drift), 0.1-0.2 (minor drift), >0.2 (significant drift). Jensen-Shannon (JS) divergence is a symmetric, bounded version of KL divergence. Wasserstein distance (Earth Mover's Distance) measures the minimum cost to transform one distribution into another. Chi-squared test for categorical features. Run these tests on rolling windows of production data against a fixed training reference set and alert when thresholds are crossed.

from scipy.stats import ks_2samp
stat, p_value = ks_2samp(train_feature, prod_feature)
print(f'KS stat: {stat:.3f}, p-value: {p_value:.4f}')
# Alert if p_value < 0.05 (distributions differ significantly)

PSI quantifies how much a feature's distribution has shifted between a reference period (training data) and a comparison period (production data). Compute by binning the reference distribution into B buckets (typically 10-20 equal-frequency or equal-width bins), measuring the proportion of production data falling in each bucket, and computing PSI = sum over bins of (prod% - ref%) * ln(prod%/ref%). Interpretation thresholds: PSI < 0.10 — negligible drift, no action needed; 0.10-0.20 — minor drift, investigate; PSI > 0.20 — significant drift, model likely degraded, retrain or investigate data pipeline. PSI was originally developed for credit scoring and is widely used in financial ML. Compute PSI for all input features and flag any feature exceeding the threshold.

def compute_psi(expected, actual, bins=10):
    breakpoints = np.percentile(expected, range(0, 100+bins, bins))
    expected_perc = np.histogram(expected, breakpoints)[0] / len(expected) + 1e-8
    actual_perc = np.histogram(actual, breakpoints)[0] / len(actual) + 1e-8
    return ((actual_perc - expected_perc) * np.log(actual_perc / expected_perc)).sum()

Data drift (covariate shift) is a change in the distribution of input features P(X). Concept drift is a change in the relationship between inputs and the target P(Y|X) — the underlying pattern the model learned becomes invalid even if input distributions remain stable. Example: a credit risk model trained before an economic shock may have learned that employment status strongly predicts default; after the shock, that relationship changes as employed people also default. Concept drift is harder to detect because it requires ground truth labels, which often arrive with a delay. Detection approaches: monitor prediction distribution shift (proxy for concept drift if labels are unavailable); monitor performance metrics on labeled subsets; use drift detectors (ADWIN, DDM, EDDM) that track statistical properties of the error stream over time.

Evidently AI is an open-source Python library for evaluating, testing, and monitoring ML models. Core functionality: pre-built reports for data drift (feature-level drift with multiple statistical tests), model performance (classification/regression metrics vs. reference period), data quality (null rates, value distributions, duplicates), and prediction drift. Use evidently in batch monitoring pipelines by computing reports on rolling production windows against a training reference dataset. Reports output as interactive HTML dashboards or JSON for integration with BI tools. Evidently Tests framework provides programmatic test suites with pass/fail assertions for CI/CD data validation. In 2024, Evidently added LLM evaluation capabilities (text descriptors, toxicity, sentiment, semantic similarity). The open-source library is the foundation; Evidently Cloud adds a managed monitoring platform.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, ClassificationPreset
report = Report(metrics=[DataDriftPreset(), ClassificationPreset()])
report.run(reference_data=train_df, current_data=prod_df)
report.save_html('drift_report.html')

WhyLogs is a lightweight, open-source data logging library that generates statistical summaries (called profiles) of datasets using a constant-memory approximate algorithm rather than storing raw data. Profiles capture: feature-level statistics (min, max, mean, median, histograms), categorical frequency counts, missing value rates, and type information. Profiles are mergeable — distributed systems can compute partial profiles and merge them for fleet-wide statistics without data centralization. WhyLogs integrates into Spark, Kafka Streams, Python batch jobs, and Pandas pipelines with minimal overhead. The WhyLabs platform provides profile storage, drift visualization, and alerting. Key advantage over sampling-based approaches: profiles represent the full data population without privacy risks from raw data storage. Merge profiles across time windows for trend analysis.

import whylogs as why
results = why.log(df)  # constant memory profile
profile = results.profile()
profile.view().to_pandas()  # summary statistics

SHAP computes the contribution of each feature to a specific prediction by treating prediction as a game where features are players and prediction value is the game payout. The Shapley value for feature i is the average marginal contribution of feature i across all possible subsets of features. SHAP values satisfy three desirable axioms: local accuracy (SHAP values sum to the difference between the prediction and the expected model output), missingness (absent features have zero SHAP value), and consistency (if a feature contributes more in model A than B, its SHAP value is higher in A). Model-specific algorithms: TreeSHAP (exact, O(TLD²) for tree ensembles), DeepSHAP (approximate for neural networks using DeepLIFT), LinearSHAP (exact for linear models), KernelSHAP (model-agnostic, expensive). Use for individual prediction explanations and global feature importance.

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

LIME (Local Interpretable Model-agnostic Explanations) explains a single prediction by fitting a simple interpretable model (linear regression) to the local neighborhood around the input point. LIME perturbs the input, gets the black-box model's predictions on perturbed samples, weights samples by proximity to the original, and fits a linear model to these weighted samples. The linear model's coefficients are the explanation. LIME is model-agnostic (any black box), computationally cheaper than KernelSHAP for high-dimensional inputs, and intuitive for non-technical stakeholders. Prefer LIME when: the model is a true black box without SHAP accelerators (e.g., a wrapped API), when text or image explanations are needed (LIME has native image/text support via superpixels and word masks), or when runtime constraints prevent KernelSHAP's O(features²) computation.

from lime.lime_tabular import LimeTabularExplainer
explainer = LimeTabularExplainer(X_train, feature_names=features, mode='classification')
exp = explainer.explain_instance(X_test[0], model.predict_proba, num_features=10)
exp.show_in_notebook()

The delayed labels problem occurs when ground truth labels for model predictions take days, weeks, or months to arrive (e.g., loan default outcomes arrive 90 days later; fraud labels after manual review; email open rates after a delay). Without labels, you cannot compute accuracy metrics in real time. Strategies: (1) proxy metrics — monitor business outcomes correlated with model quality (click rate, purchase rate); (2) prediction distribution monitoring — detect shifts in the model's output distribution as an early warning signal; (3) partial label evaluation — use available labels immediately (subset that arrive quickly) for leading indicators; (4) log everything with timestamps and join labels when they arrive for retrospective analysis; (5) simulation testing — replay historical scenarios with known outcomes; (6) surrogate metrics — compare with a simpler rule-based baseline that has faster feedback.

# Log predictions with timestamps for later joining with labels
predictions_log.append({
    'request_id': req_id,
    'prediction': model_output,
    'timestamp': datetime.utcnow().isoformat(),
    'features': features_dict
})
# Join with labels when they arrive via request_id

A production feature monitoring pipeline computes statistical summaries of serving features over rolling time windows and compares them against a training reference baseline. Architecture: (1) Feature logging — log all inference features with request IDs and timestamps to a data store (Kafka → Delta Lake, or directly to BigQuery/Snowflake); (2) Batch computation — scheduled Spark/Airflow job computes drift statistics (KS test, PSI, null rate) per feature for each time window (hourly, daily); (3) Alerting — violations above thresholds trigger PagerDuty/Slack alerts; (4) Dashboard — Grafana/Superset visualizations of drift trends over time. Compute drift against: the full training set as reference (stable baseline) and the previous week's production data (detect recent regime changes). Monitor both individual features and feature correlations (joint distribution drift).

# Feature monitoring with Evidently in Airflow
@task
def compute_drift_report():
    prod_df = load_production_features(window='24h')
    report = Report(metrics=[DataDriftPreset()])
    report.run(reference_data=train_df, current_data=prod_df)
    drift_results = report.as_dict()
    alert_if_drifted(drift_results)

Arize AI is a managed ML observability platform purpose-built for production ML monitoring. Key capabilities: real-time model performance monitoring (accuracy, precision, recall, NDCG for ranking) with ground truth label joining when they arrive; multi-dimensional slicing — break down performance by any feature combination to find subgroups where the model underperforms (fairness analysis); embedding drift detection using cosine similarity and UMAP visualizations of embedding distributions — critical for LLM and computer vision models; LLM traces and spans for RAG and agent workflows; A/B model comparison; and custom metrics via Python SDK. Arize Phoenix is the open-source sibling focused on LLM tracing (traces, spans, retrieval evaluation). Arize integrates via SDK: log predictions and actual values at inference time and when labels arrive.

import arize
client = arize.Client(space_key='xxx', api_key='yyy')
client.log(
    prediction_id=req_id,
    model_id='fraud-detector',
    model_version='v3',
    prediction_label=prediction,
    features=feature_dict
)

OpenTelemetry (OTel) is a vendor-neutral observability framework providing unified APIs for traces, metrics, and logs. For ML serving, OTel instruments inference servers to capture: distributed traces across the full inference pipeline (API gateway → preprocessing → model inference → post-processing → response), with each step as a span including feature extraction latency, model forward pass time, and post-processing duration; custom metrics (prediction confidence, class distribution, batch size); and structured logs with trace context for correlation. OTel data exports to any backend (Jaeger, Tempo, Prometheus, Datadog, Honeycomb). LLM-specific OTel semantics (OpenLLMetry, Traceloop) capture token counts, model name, prompt and completion content as span attributes. This makes ML inference pipelines fully observable alongside the rest of the application infrastructure without vendor lock-in.

from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span('model_inference') as span:
    span.set_attribute('model.version', 'v3')
    span.set_attribute('batch.size', len(batch))
    result = model.predict(batch)

Fairness metrics measure whether a model's predictions are equitable across protected groups (race, gender, age, geography). Key metrics: Demographic Parity — equal positive prediction rates across groups; Equal Opportunity — equal true positive rates (sensitivity) across groups; Equalized Odds — equal TPR and FPR across groups; Predictive Parity — equal precision across groups; Individual Fairness — similar inputs receive similar predictions. Monitor in production by logging predictions with sensitive attributes, computing fairness metrics on sliding windows, and alerting when disparities exceed thresholds. Fairness-accuracy trade-offs exist — optimize for the fairness criterion appropriate to the domain (medical: equalized odds; criminal justice: predictive parity). Tools: Fairlearn (Microsoft), AI Fairness 360 (IBM), Aequitas, Arize slice analysis.

from fairlearn.metrics import demographic_parity_difference, equalized_odds_difference
dp = demographic_parity_difference(y_true, y_pred, sensitive_features=protected_attr)
print(f'Demographic parity difference: {dp:.4f}')

Expose model serving metrics in Prometheus format using the prometheus_client Python library: register Gauges for prediction value distributions, Counters for request counts and errors, Histograms for inference latency with pre-configured buckets. ML-specific metrics to expose: inference_request_duration_seconds (histogram), prediction_value (histogram or summary for output distribution), feature_null_count (gauge per feature), model_version_info (labeled gauge). Configure Prometheus to scrape the /metrics endpoint. Build Grafana dashboards with: latency P50/P95/P99 panels, request rate, error rate, prediction distribution histograms, and data drift score time series (computed in batch and exposed as gauge values). Set Grafana alerts with Prometheus AlertManager for threshold violations (drift > 0.2, P99 > 200ms, error rate > 0.1%).

from prometheus_client import Histogram, Counter
INFERENCE_LATENCY = Histogram('inference_duration_seconds', 'Model inference time', buckets=[0.01,0.05,0.1,0.5,1.0])
PREDICTION_VALUE = Histogram('prediction_score', 'Prediction score distribution', buckets=[i/10 for i in range(11)])
with INFERENCE_LATENCY.time():
    score = model.predict(features)
PREDICTION_VALUE.observe(score)

NannyML is an open-source Python library for post-deployment model monitoring that estimates model performance without ground truth labels using Confidence-Based Performance Estimation (CBPE) and Direct Loss Estimation (DLE). CBPE estimates accuracy, F1, and AUC from prediction confidence scores by modeling the relationship between confidence and true performance on the validation set. This enables real-time performance estimation before labels arrive. NannyML also provides univariate and multivariate drift detection (using PCA-based reconstruction error for multivariate drift) and business value estimation. It is particularly valuable for models with significant label delays (weeks/months). NannyML integrates with MLflow for logging estimates and generates HTML reports similar to Evidently. Target use cases: credit scoring, insurance, fraud detection where ground truth arrives with delay.

import nannyml as nml
cbe = nml.CBPE(y_pred_proba='pred_proba', y_true='label',
               problem_type='classification', metrics=['roc_auc'])
cbe.fit(reference_df)
results = cbe.estimate(production_df)
results.plot().show()

Univariate drift detection tests each feature independently (KS test, PSI per column), potentially missing interactions where individual features remain stable but their joint distribution shifts (e.g., height and weight individually stable but their correlation changes). Multivariate drift detection captures joint distribution shifts. Methods: (1) PCA-based reconstruction error — fit PCA on training data; production data with different correlation structure has higher reconstruction error; (2) MMD (Maximum Mean Discrepancy) — kernel-based test that captures high-dimensional distributional differences; (3) Classifier-based drift test — train a binary classifier to distinguish training vs. production samples; high AUC indicates the distributions are distinguishable; (4) NannyML's multivariate drift using PCA domain classifier. Multivariate methods are computationally heavier but necessary for models where feature interactions are critical to predictions.

# Classifier-based drift detection
from sklearn.linear_model import LogisticRegression
X = np.vstack([train_features, prod_features])
y = np.array([0]*len(train_features) + [1]*len(prod_features))
auc = cross_val_score(LogisticRegression(), X, y, scoring='roc_auc').mean()
print(f'Drift AUC: {auc:.3f}')  # 0.5 = no drift, 0.9+ = significant drift

The champion-challenger pattern maintains two active model versions simultaneously in production: the champion (current best-performing model serving the majority of traffic) and the challenger (a new candidate model serving a small traffic slice for online evaluation). Both models serve real production traffic; their predictions and outcomes are logged and compared using the same business metrics (conversion rate, revenue impact, accuracy on labeled samples). When the challenger demonstrably outperforms the champion with statistical significance, it is promoted to champion. This pattern enables safe continuous improvement: new models are validated on real traffic before full deployment, avoiding lab-to-production performance gaps. Implement with traffic splitting (90% champion / 10% challenger) and automated promotion criteria.

Integrated Gradients (Sundararajan et al., 2017) is an attribution method for neural networks that computes feature importance by integrating the gradient of the model's output with respect to each input feature along a straight path from a baseline input (typically zero or noise) to the actual input. This satisfies two key axioms: sensitivity (if a feature changes the output, it has non-zero attribution) and implementation invariance (two equivalent network implementations receive identical attributions). Unlike vanilla gradients (which are local and may saturate), integrated gradients capture the full contribution across the interpolation path. For image models, each pixel receives an attribution score. For text models, each token receives a score. Compute with: ig = (input - baseline) * ∫ gradient dα (approximated with Riemann sum over α ∈ [0,1]).

from captum.attr import IntegratedGradients
ig = IntegratedGradients(model)
attributions = ig.attribute(
    input_tensor,
    baselines=torch.zeros_like(input_tensor),
    n_steps=50
)

Automated retraining pipelines trigger model updates in response to detected degradation, eliminating the need for manual monitoring and intervention. Common trigger signals: (1) PSI or KS drift score exceeding a threshold on key features; (2) model performance (accuracy, AUC, F1 on labeled production subsets) dropping below a minimum acceptable level; (3) prediction distribution shift (output PSI > threshold); (4) scheduled time-based retraining (weekly/monthly) as a baseline; (5) cumulative new labeled examples exceeding N (enough data to justify retraining). Implement in Airflow/Prefect: a monitoring task runs daily, evaluates triggers, and if any fires, kicks off the training DAG. Guard against training instability by validating the new model against held-out evaluation sets before automatic deployment. Include alerting for human review before promoting high-risk models.

@task
def evaluate_retraining_trigger():
    psi = compute_feature_psi(train_ref, production_last_7d)
    auc = compute_model_auc_on_labeled_prod()
    if psi.max() > 0.20 or auc < AUC_THRESHOLD:
        trigger_training_dag(reason='drift_detected')
        notify_team('Retraining triggered')

The Wasserstein distance (Earth Mover's Distance) measures the minimum cost (integral of absolute CDF difference for 1D distributions) to transform one probability distribution into another, where cost equals the amount of 'earth' moved times the distance moved. For ML drift detection, W1 (1st Wasserstein distance) = ∫|CDF_ref - CDF_prod|. Advantages over KL divergence: defined even when distributions have non-overlapping support (KL divergence is infinite in that case); symmetric (KL is asymmetric); bounded and interpretable in units of the feature's domain. KL divergence (∫ p log(p/q)) measures information loss and is sensitive to small probability regions. Jensen-Shannon divergence (symmetric KL) is bounded [0,1] but still undefined for non-overlapping support. Wasserstein is recommended for production monitoring as it handles zero-probability regions gracefully.

from scipy.stats import wasserstein_distance
from scipy.special import kl_div
import numpy as np
w_dist = wasserstein_distance(train_feature, prod_feature)
print(f'Wasserstein distance: {w_dist:.4f}')

LLM evaluation goes beyond classification accuracy to assess open-ended text quality. Reference-based metrics: BLEU (n-gram overlap with reference), ROUGE (recall-oriented n-gram/sequence overlap), BERTScore (contextual embedding similarity). Reference-free metrics: Perplexity (model's uncertainty over generated text — lower is better). LLM-as-judge: use a powerful LLM (GPT-4o, Claude) to score outputs on dimensions like relevance, faithfulness (for RAG), coherence, helpfulness, and safety — measured on a 1-5 Likert scale. For RAG evaluation: context precision, context recall, answer faithfulness, answer relevance (RAGAS framework). For safety: toxicity (Detoxify, Perspective API), PII detection, hallucination rate. Production LLM monitoring requires sampling predictions for human review and automated LLM-judge scoring on statistical samples.

from ragas.metrics import faithfulness, answer_relevancy
from ragas import evaluate
result = evaluate(
    dataset=rag_dataset,
    metrics=[faithfulness, answer_relevancy]
)
print(result)

A model card is a structured document that accompanies a deployed ML model, providing transparency about its purpose, performance, limitations, and appropriate use. Contents: model description (architecture, training data, intended use), performance metrics across demographic groups and evaluation conditions, limitations and failure modes, ethical considerations (known biases, fairness evaluation), training data description, environmental impact (compute cost), maintenance contacts, and versioning history. Model cards were introduced by Google (Mitchell et al., 2019) and are now required by many enterprise AI governance frameworks, the EU AI Act (Article 13 transparency requirements for high-risk AI systems), and regulated industries. Platforms: HuggingFace Hub auto-renders model cards from README.md; AI Factsheets (IBM), Model Registry in MLflow can store cards as artifacts.

Recommendation models suffer from feedback loop degradation: the model's recommendations influence user behavior, changing the data used for future training, which further biases recommendations toward popular items (popularity bias amplification) or traps users in filter bubbles (exposure bias). Monitor for: recommendation diversity decrease over time (Gini coefficient of recommended item distribution); novelty metrics dropping (fraction of recommended items the user has not seen); popularity bias increasing (fraction of recommendations that are top-10% popular items); and offline-online metric divergence (offline NDCG improving while online engagement metrics plateau). Mitigate with: exploration strategies (epsilon-greedy, Thompson sampling), counterfactual evaluation (IPS estimators), and causal recommendation frameworks. Log exposed items alongside click labels to enable counterfactual training data collection.

# Monitor recommendation diversity
from scipy.stats import entropy
def recommendation_diversity(recs):
    item_freq = Counter(item for user_recs in recs for item in user_recs)
    probs = np.array(list(item_freq.values())) / sum(item_freq.values())
    return entropy(probs)  # higher entropy = more diverse

These are statistical detectors applied to the error stream (sequence of correct/incorrect predictions) in online learning systems. ADWIN (Adaptive Windowing) maintains a sliding window of the error rate; the window shrinks when statistical change is detected (comparing recent vs. older sub-windows using Hoeffding bounds), adapting to gradual drift without a fixed window size. DDM (Drift Detection Method) monitors the error rate and its standard deviation; raises a warning when error_rate + k*std_dev > min_error + 2*min_std, and signals drift at k=3. EDDM (Early Drift Detection Method) monitors the distance between errors (time between mistakes); increasing error frequency signals drift. All three work without labels (assuming predictions are validated in near-real-time) and are implemented in scikit-multiflow and River Python libraries for production online learning systems.

from river.drift import ADWIN
detector = ADWIN(delta=0.002)
for prediction, label in zip(predictions, labels):
    error = int(prediction != label)
    detector.update(error)
    if detector.drift_detected:
        print('Drift detected!')

Embedding distribution monitoring detects when the semantic representation of production data differs from training data — critical for deep learning models where raw feature drift may not capture semantic changes. Methods: (1) Cosine similarity distribution — compute pairwise cosine similarities in a sample of production embeddings vs. training; distributional shift in similarity scores indicates drift; (2) UMAP/t-SNE visualization — reduce embeddings to 2D and visually compare training vs. production clusters; (3) Classifier-based embedding drift — train a binary classifier to distinguish training vs. production embedding vectors; AUC measures drift magnitude; (4) Statistical tests on PCA-reduced embeddings — apply KS test on principal component scores; (5) Arize's embedding drift uses Euclidean distance between centroid distributions. Log production embeddings (or a sample) alongside predictions for offline analysis.

import umap
reducer = umap.UMAP(n_components=2, random_state=42)
train_2d = reducer.fit_transform(train_embeddings)
prod_2d = reducer.transform(prod_embeddings)
# Plot to visualize distribution overlap

Data validation tests ML pipeline inputs against predefined expectations (business rules + statistical assumptions) before training or serving. Great Expectations is an open-source framework for defining, documenting, and enforcing data quality assertions as 'expectations'. Define expectations in an ExpectationSuite: column values must be non-null (expect_column_values_to_not_be_null), values within a range (expect_column_values_to_be_between), unique (expect_column_values_to_be_unique), or follow a distribution (expect_column_mean_to_be_between). Run a Checkpoint to validate a batch of data against the suite; failed expectations fail the pipeline. GX integrates with Airflow, Spark, and Pandas. Generated 'Data Docs' provide HTML reports of validation results. For serving, validate incoming feature vectors against training-derived statistics before inference to catch upstream data pipeline failures early.

import great_expectations as gx
context = gx.get_context()
validator = context.sources.pandas_default.read_dataframe(df)
validator.expect_column_values_to_not_be_null('user_id')
validator.expect_column_values_to_be_between('age', min_value=18, max_value=120)
results = validator.validate()

Structured logging records inference events as machine-parseable JSON records rather than free-form text strings, enabling querying, aggregation, and dashboarding. Log fields for ML inference: request_id (UUID for tracing), model_id, model_version, timestamp, features (sanitized — omit PII), prediction (output value), prediction_probability (confidence), inference_latency_ms, and any upstream feature retrieval times. Use Python's structlog or loguru with JSON formatter. Ship logs to a centralized store (Elasticsearch, BigQuery, ClickHouse) via Fluentd/Logstash/Vector. Enable correlation: include trace_id and span_id (from OpenTelemetry) so inference logs can be joined with distributed traces. Implement log sampling (log 1% of requests in high-volume systems) to manage storage costs while retaining statistical representativeness.

import structlog
log = structlog.get_logger()
log.info('prediction',
    request_id=req_id,
    model_version='v3',
    prediction=score,
    latency_ms=latency,
    trace_id=otel_trace_id
)

A counterfactual explanation answers the question: 'What minimal changes to the input would flip the model's prediction?' For a loan rejection, the counterfactual is: 'If your income were $5,000 higher and your credit utilization were 10% lower, you would be approved.' This provides actionable recourse — unlike SHAP which explains why a decision was made, counterfactuals explain how to change the outcome. Generate counterfactuals using: optimization-based search (DiCE — Diverse Counterfactual Explanations), nearest neighbor in training data with a flipped label, or model gradient search. For monitoring, counterfactual distances can measure model robustness — if tiny input changes flip predictions, the model is brittle. Regulatory frameworks (EU GDPR Article 22, EU AI Act) require providing meaningful explanations to individuals subjected to automated decisions.

import dice_ml
data = dice_ml.Data(dataframe=train_df, continuous_features=['income','age'], outcome_name='approved')
model_dice = dice_ml.Model(model=model, backend='sklearn')
exp = dice_ml.Dice(data, model_dice)
cf = exp.generate_counterfactuals(query_instance=rejected_applicant, total_CFs=3)

An SLO (Service Level Objective) is an internal target for a measurable aspect of service quality; an SLA (Service Level Agreement) is a contractual commitment with penalties for violation. For ML systems, define SLOs across three dimensions: (1) Operational — inference P99 latency < 200ms, availability > 99.9%, error rate < 0.1%; (2) Model quality — AUC > 0.85 on weekly labeled sample, drift score < 0.2 on all features, prediction coverage > 99% (model responds rather than falling back); (3) Business impact — conversion rate of model-served users vs. baseline, cost per prediction. Define SLOs in error budget terms: a 99.9% availability SLO allows 8.7 hours/year downtime. Monitor SLO consumption with burn rate alerts (rapid error budget consumption triggers pages). Use SLO dashboards (Grafana SLO plugin) to visualize remaining error budget.

# Prometheus: SLO compliance rate over 30 days
sum(rate(inference_request_duration_seconds_bucket{le='0.2'}[30d]))
/
sum(rate(inference_request_duration_seconds_count[30d]))

Partial Dependence Plots (PDPs) show the marginal effect of one or two features on predicted outcomes, averaging over all other features. They reveal global model behavior: monotonic relationships, thresholds, and interaction effects. Compute by varying the target feature across its range while holding all others at observed values and averaging model outputs. ICE plots (Individual Conditional Expectation) show the PDP calculation for each individual sample rather than the average, revealing heterogeneous effects hidden by averaging. Where all ICE lines are parallel, the feature has a uniform effect; crossing lines indicate interaction effects. Limitations of PDP: correlation between features can produce misleading plots (marginalizing over correlated features creates unlikely combinations). Accumulated Local Effects (ALE) plots address this limitation by using conditional distributions instead of marginal.

from sklearn.inspection import PartialDependenceDisplay
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
PartialDependenceDisplay.from_estimator(
    model, X_train, features=['age', 'income'],
    kind='both',  # 'both' shows PDP + ICE
    ax=ax
)

Data lineage tracks the origin and transformation history of data used to train and serve ML models. In ML observability, lineage enables: (1) root cause analysis — tracing a model degradation back to a specific upstream data source change or transformation bug; (2) impact analysis — identifying which models are affected when a data source changes schema or quality; (3) reproducibility — reconstructing the exact training dataset for a past model version; (4) compliance — demonstrating data provenance for regulatory audits. Implement data lineage with: OpenLineage standard (emits lineage events from Spark, Airflow, dbt), Apache Atlas for Hadoop ecosystems, Datahub or Marquez as lineage backends. ML-specific lineage links dataset versions to model versions, connecting data pipeline runs to training runs to model registry entries. Feature stores (Feast, Tecton) provide built-in feature lineage.

Prediction anomaly detection identifies individual predictions that deviate from expected behavior — a leading indicator of model or data pipeline issues before aggregate metrics degrade. Approaches: (1) Statistical control charts — monitor rolling mean and standard deviation of prediction scores; flag predictions > 3 standard deviations from the rolling mean (3-sigma rule); (2) Isolation Forest trained on normal prediction behavior; (3) Quantile-based bounds — alert when prediction falls outside the 0.1-99.9 percentile range of training predictions; (4) Business rule violations — predictions outside valid business ranges (negative price, impossible user age); (5) Confidence score anomalies — unusually high (overconfident) or low (uncertain) probabilities for models that should be calibrated. Log anomalous predictions with full feature context for investigation. High anomaly rates (>5%) warrant immediate investigation of the serving pipeline.

from sklearn.ensemble import IsolationForest
# Train on training-time prediction scores
detector = IsolationForest(contamination=0.01)
detector.fit(train_predictions.reshape(-1,1))
# Flag production anomalies
anomalies = detector.predict(prod_predictions.reshape(-1,1)) == -1

Fiddler AI is an enterprise ML monitoring and explainability platform targeting regulated industries requiring explainable AI governance. Key differentiators: (1) Natural Language Explanations — translates SHAP/LIME outputs into human-readable explanations for business stakeholders; (2) Segment-level performance analysis — drill down to any feature segment (geography, customer tier, product type) to find localized model failures; (3) Bias detection and fairness dashboards — automated computation of demographic parity, equal opportunity, and predictive parity across protected attributes; (4) What-if scenarios — interactive tool for business users to modify inputs and see how predictions change without writing code; (5) Model comparison across time (production vs. reference) and across versions; (6) Integration with MLflow, SageMaker, and major data warehouses. Fiddler targets credit, insurance, healthcare, and financial services where explainability and fairness are regulatory requirements.

A prediction store is a centralized repository that logs every model prediction alongside its features, model version, timestamp, and ground truth labels when available. Architecture: at inference time, log to a fast write store (Kafka topic or Redis); a downstream consumer (Flink or Spark Streaming) enriches with feature metadata and writes to a long-term analytical store (Delta Lake, BigQuery, Redshift). Schema: prediction_id, request_id, model_id, model_version, timestamp, feature_hash or feature_values, prediction_label, prediction_probability, latency_ms, ground_truth (nullable, populated when label arrives). Join predictions with labels via prediction_id when labels arrive (delayed join). Query the prediction store for: per-segment accuracy analysis, drift computation, model comparison, incident investigation, and generating monitoring datasets for Evidently/NannyML reports.

# Prediction logging schema (Delta Lake)
schema = StructType([
    StructField('prediction_id', StringType()),
    StructField('model_version', StringType()),
    StructField('timestamp', TimestampType()),
    StructField('features', MapType(StringType(), FloatType())),
    StructField('score', FloatType()),
    StructField('label', IntegerType(), nullable=True)
])

Permutation feature importance measures how much a model's performance degrades when a feature's values are randomly shuffled (breaking the relationship between that feature and the target). High permutation importance = the model heavily relies on that feature. Unlike SHAP which explains individual predictions (local), permutation importance is a global dataset-level metric. For monitoring, track permutation importance over time: if an important feature's importance drops, the feature may have drifted or become uninformative. If an unimportant feature gains importance, investigate for data leakage or spurious correlation in new data. Limitations: correlated features split importance between them (shuffling one may not degrade much if the correlated feature compensates). Compute with sklearn's permutation_importance(). More reliable than impurity-based importance (which favors high-cardinality features).

from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)
for i in result.importances_mean.argsort()[::-1]:
    print(f'{feature_names[i]}: {result.importances_mean[i]:.3f}')

Canary analysis automatically evaluates whether a new model version (canary) meets promotion criteria before full rollout. Implementation: (1) Define success criteria — metrics that must remain within acceptable bounds vs. baseline: P99 latency < 1.2x baseline, business conversion rate >= baseline, error rate < threshold, accuracy on labeled sample > floor; (2) Run the canary for a minimum duration and traffic volume (statistical power requirements); (3) Automated analysis compares canary vs. baseline metrics using Welch's t-test or Mann-Whitney U test for continuous metrics, proportions test for binary metrics; (4) Promotion decision: if all criteria pass, automatically promote; if any fail, automatically rollback and alert. Tools: Argo Rollouts (Kubernetes) with built-in Prometheus analysis templates; Flagger for Istio/Linkerd. Log the analysis results with reasoning for audit trails.

# Argo Rollouts analysis template
metrics:
- name: success-rate
  successCondition: result[0] >= 0.95
  provider:
    prometheus:
      query: sum(rate(http_requests_total{status!~'5..'}[5m]))/sum(rate(http_requests_total[5m]))

RAG (Retrieval-Augmented Generation) systems have two quality dimensions requiring separate evaluation. Retrieval quality: Context Precision — what fraction of retrieved chunks are relevant to the question (precision@k); Context Recall — what fraction of necessary information was retrieved (requires reference answers); Mean Reciprocal Rank (MRR) and NDCG for ranking quality. Generation quality given retrieved context: Faithfulness — does the answer make only claims supported by the retrieved context (measures hallucination; scored by LLM judge); Answer Relevance — is the answer relevant to the original question (cosine similarity of question vs. answer embeddings or LLM judge). RAGAS is the standard open-source RAG evaluation framework implementing these metrics. For production monitoring, sample request-response pairs and evaluate with automated LLM judges (GPT-4o or Claude as evaluators) on a daily basis.

from ragas import evaluate
from ragas.metrics import context_precision, faithfulness, answer_relevancy, context_recall
results = evaluate(
    dataset,  # questions, contexts, answers, ground_truth
    metrics=[context_precision, context_recall, faithfulness, answer_relevancy]
)

Bias amplification occurs when a model's deployment creates feedback loops that reinforce existing biases: a hiring model trained on historical male-dominated data recommends men, reducing female applications, creating even more male-dominated future training data. Monitor for: (1) Demographic representation in model inputs — track the distribution of protected attributes in serving requests over time (are female candidates increasingly underrepresented in the applicant pool?); (2) Outcome disparities — compute selection rates, approval rates, or other outcome metrics by demographic group on rolling windows; (3) Model feedback loop metrics — if model outputs influence future inputs (recommendation → viewing → training data), track the diversity metrics over sequential time windows; (4) Counterfactual fairness — test if identical profiles with different protected attributes receive different scores. Alert when group disparity ratios exceed configurable fairness thresholds (e.g., selection rate ratio < 0.8, the 4/5ths rule in US employment law).

# Monitor selection rate ratio (4/5ths rule)
for group in protected_groups:
    selection_rate = predictions[predictions.group==group].mean()
    print(f'{group}: {selection_rate:.3f}')
# Alert if any group < 0.8 * max(selection_rates)

Weights & Biases (W&B) began as an experiment tracking tool but has expanded into a broader ML development and observability platform. Experiment tracking: automatic logging of metrics, parameters, and artifacts. W&B Artifacts provide dataset and model versioning with lineage graphs linking training runs to datasets to model versions. W&B Tables enable interactive slice-and-dice analysis of predictions with filtering by any column — useful for per-class, per-segment accuracy analysis. W&B Sweeps manages distributed hyperparameter optimization. W&B Launch orchestrates model training jobs on cloud compute. W&B Prompts provides LLM trace logging and evaluation for prompt engineering workflows. In production, teams instrument evaluation pipelines to log to W&B for continuous model quality tracking, making it a hub for the model quality signal across training, evaluation, and production monitoring.

import wandb
with wandb.init(project='fraud-model', name='prod-eval-2025-W01'):
    wandb.log({'auc': 0.923, 'psi_max': 0.08})
    table = wandb.Table(dataframe=prediction_df)
    wandb.log({'predictions': table})

Classification model output monitoring tracks the prediction score distribution and class label distribution. Key metrics: predicted positive rate (fraction of predictions in the positive class) — sudden shifts indicate model or data issues; score distribution drift (PSI on confidence scores between production windows and training); calibration drift (whether confidence scores still correspond to true probabilities — measured by Expected Calibration Error on labeled subsets); class confidence histogram (bimodal for well-calibrated binary classifiers; flat for overconfident models). Regression model output monitoring tracks the prediction value distribution: mean, standard deviation, min/max, quantile ranges, and PSI on prediction values. Watch for range violations (predictions outside physically possible bounds), mean shift (systematic bias), and variance changes (model suddenly uncertain or overconfident). Both benefit from comparing against a reference period using statistical tests.

# Monitor prediction calibration drift
from sklearn.calibration import calibration_curve
frac_pos, mean_pred = calibration_curve(y_true, y_pred_proba, n_bins=10)
# Expected Calibration Error
ece = np.mean(np.abs(frac_pos - mean_pred))
print(f'ECE: {ece:.4f}')

An ML metadata store is a centralized repository tracking all artifacts, executions, and lineage in the ML lifecycle. It records: datasets (with schema version, location, statistics), model training executions (hyperparameters, metrics, runtime, data used), model artifacts (weights, serialized objects, model cards), evaluation results (metrics on held-out sets, fairness analysis), deployment events (which model version was deployed when, to which environment). ML Metadata (MLMD, from Google, used in Kubeflow) is the standard library with a Python API and pluggable backends (SQLite, MySQL). MLflow's tracking server serves a similar purpose. For governance: audit trails proving which training data produced each deployed model, evidence of evaluation gates before deployment, and lineage for regulatory submissions (FDA, Basel III). Link metadata store entries to orchestration tool runs (Airflow/Kubeflow run IDs) for full traceability.

from ml_metadata import metadata_store
from ml_metadata.proto import metadata_store_pb2
connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.sqlite.filename_uri = 'mlmd.db'
store = metadata_store.MetadataStore(connection_config)

Root cause analysis (RCA) for ML metric degradation follows a systematic funnel. Step 1 — Segment: break down the metric by time window (when did it start?), geographic region, user cohort, product category. The segment with the largest degradation is the entry point. Step 2 — Feature drift: compute drift for all features in the degraded segment vs. baseline. Features with highest drift scores are candidates. Step 3 — Data pipeline validation: check upstream data sources for schema changes, null rate spikes, or volume anomalies in the degraded time window (Great Expectations or custom checks). Step 4 — SHAP analysis: compute SHAP values for recent misclassified examples and compare feature importance distribution to historical correct predictions. Step 5 — Code changes: check if any model code, feature computation, or serving code was deployed near the degradation onset. Document findings and resolution in a post-mortem.

A calibrated model's confidence scores reflect true probabilities: if the model predicts 80% positive for a group of samples, approximately 80% of them should actually be positive. Poor calibration means overconfident (predicts 95% but only 75% are positive) or underconfident models. Calibration measurement: Expected Calibration Error (ECE) bins predictions by confidence decile and computes the weighted mean absolute difference between bin accuracy and bin mean confidence. Monitor calibration drift in production by computing ECE on rolling windows of labeled samples. Calibration often drifts when the model is retrained on new data but calibration was applied to an older version, or when class imbalance shifts in production. Recalibrate with Platt scaling or isotonic regression using a recent labeled hold-out set without retraining the underlying model.

from sklearn.calibration import CalibratedClassifierCV
# Recalibrate on recent labeled production data
calibrated = CalibratedClassifierCV(base_model, cv='prefit', method='isotonic')
calibrated.fit(X_recent_labeled, y_recent_labeled)