Applied ML Case Studies

Common causes: train/serve skew (different feature computation), data leakage inflating offline metrics, distribution shift between training and live data, a label that isn't available at inference, or accuracy being the wrong metric on imbalanced data. Audit the feature pipeline for parity, check for leakage, compare train vs live distributions, and switch to a metric aligned with the business (e.g., precision/recall, AUC).

Optimize for the business cost: raise the decision threshold or use precision@k so you only contact the highest-risk users; calibrate probabilities; add cost-sensitive learning. Re-evaluate features for leakage/noise, and target interventions where uplift is highest (uplift modeling) rather than raw churn score. Measure downstream retention lift, not just AUC.

Concept drift: fraudsters adapt, so patterns change; labels also arrive late (delayed feedback). Detect drift on features and performance, retrain on recent data frequently, use online/streaming updates, and monitor with delayed-label-aware metrics. Add adversarial-robust features and a feedback loop from confirmed fraud.

Use transfer learning / a pretrained model with light fine-tuning, strong regularization, and cross-validation; augment data; consider few-shot with an LLM or weak supervision to label more. Start with a simple, well-regularized baseline, prioritize label quality, and set up active learning to grow the set efficiently.

Add diversity/novelty and freshness to ranking (e.g., MMR, exploration), avoid filter bubbles, and balance relevance with serendipity. Introduce exploration (bandits) for cold or uncertain items, penalize over-exposed items, and optimize for long-term engagement/retention rather than immediate clicks.

Measure fairness (e.g., equal opportunity, demographic parity gaps) across groups, find the source (biased labels, unrepresentative data, proxy features), and mitigate: reweight/resample, remove proxies, add fairness constraints, or post-process thresholds per policy. Involve stakeholders, document tradeoffs, and monitor fairness in production continuously.

Add interpretability (SHAP/feature importance, example-based explanations), show calibrated confidence, and provide clear failure modes and guardrails. Pilot with a human-in-the-loop, present business-metric impact from an A/B test, and give users control/override. Trust comes from transparency plus demonstrated value.

Overfitting. Add regularization (L2, dropout), get more/augmented data, reduce model capacity, use early stopping, and cross-validate. Check for train/val leakage if the gap is extreme. Tune via the validation curve.

Profile the path: model inference vs feature fetch vs I/O. Fixes: batch requests, quantize/distill the model, cache features and results, autoscale, use a faster runtime (ONNX/TensorRT), and move heavy features offline. Set SLAs, load-test, and add a fast fallback for tail latency.

CTR is a proxy; optimize the true objective. Check for clickbait/short-term gaming, novelty effects, and metric tradeoffs. Look at downstream conversion/revenue and long-term retention, run the test longer, and consider a combined objective. Don't ship on a proxy that doesn't move the business metric.

It didn't learn rare seasonal/holiday effects and there was distribution shift. Add calendar/holiday and promo features, use models that capture seasonality, incorporate external signals, and hold out similar periods for validation. Blend with human overrides for known events and widen prediction intervals.

Look for features unavailable at prediction time, target-derived features, or preprocessing fit on the full data before splitting. Signs: implausibly high metrics, a single dominant feature, or performance that collapses in production. Fix by fitting all preprocessing inside CV folds and building features with a strict time cutoff.

Domain shift: lab images differ from real photos (lighting, angle, blur, backgrounds). Collect/representative real data, augment for those conditions, fine-tune on target-domain data, and add test-time robustness. Evaluate on a realistic holdout, not lab data.

Vocabulary/topic shift. Fine-tune or continue-training on the new domain's text, expand training data, and monitor drift. Use domain-adaptive pretraining if enough unlabeled text exists, and evaluate per segment to catch localized failures.

Provide per-decision attributions (SHAP), reason codes, and counterfactuals ('increase income by X to be approved'); ensure the model avoids prohibited features and proxies. Consider a simpler interpretable model or a monotonic GBM if regulation demands. Document, validate explanations for fidelity, and keep an audit trail.

Their heavy activity dominates training. Reweight/downsample power users, add per-user normalization, evaluate on segment-balanced metrics, and include cold/casual users explicitly. Regularize and monitor performance across user tiers so the model serves the whole population.

New data may contain a pipeline bug, label errors, schema/feature drift, or a corrupted source; or the split/leakage changed. Compare feature distributions old vs new, validate labels, check the training pipeline, and use a champion/challenger gate so a worse model never auto-promotes.

Monitor performance on delayed labels, and proxy signals like prediction distribution shift, feature drift (PSI/KL), and confidence changes. Trigger retraining on metric decay or scheduled cadence, and validate the candidate against the current model before promotion. Alert on data-quality breaks too.

Use metrics for imbalance (PR-AUC, recall at fixed precision), not accuracy. Try class weights, resampling (SMOTE/undersampling), and threshold tuning; anomaly-detection framing can help. Ensure the validation set reflects real prevalence, and optimize for the business cost of false negatives vs positives.

Propose a primary business-aligned metric (e.g., PR-AUC or resolution rate) but caveat that one number hides failure modes; pair it with guardrails (fairness, latency, cost) and segment breakdowns. A dashboard with a headline metric plus drift and quality-by-segment is safer than a single KPI.

Region-specific data distributions, languages, regulations, and infrastructure differ; a globally-trained single model may underperform locally. Consider per-region fine-tuning or features, evaluate per region, handle data residency, and re-validate cost/latency at scale before rollout.

That's a leakage/availability problem. Remove it or replace with a value known at prediction time (its lagged version or a proxy), and rebuild features respecting the serving-time cutoff. Simulate serving to verify parity, and add train/serve skew monitoring.

Compare before/after on an offline relevance eval and live metrics (CTR, reformulation rate, abandonment) by segment; inspect example queries. Roll back if a regression is confirmed, then diagnose (index change, feature bug, over-optimized proxy). Gate future updates behind interleaving/A-B tests.

Calibration matters when probabilities drive decisions/thresholds or expected-value math. Fix with Platt scaling or isotonic regression on a validation set, or use calibration-friendly training. Verify with a reliability diagram and Brier score.

Expect an adaptive adversary: use robust features, frequent retraining on fresh labels, ensemble signals (content + behavior + network), rate limits, and human review of borderline cases. Monitor for drift and evasion, and avoid features that are trivial to spoof. Treat it as an ongoing arms race, not a one-off model.

Slice metrics to surface the failing segment, then boost its representation (collect/upsample), add segment-specific features or a specialized model, and set segment-level acceptance criteria. Don't let aggregate metrics hide critical failures; monitor the segment explicitly.

Quantization (int8/4) for big memory/latency wins with small accuracy loss; pruning; knowledge distillation to a smaller student; and efficient architectures. Measure accuracy vs latency/size on target hardware, and combine techniques. Validate on-device, not just on server.

Offline: metrics on a holdout + key segments, fairness, and comparison to the current champion. Then shadow/canary on live traffic, check latency/cost, and run an A/B test on a business metric with guardrails. Require the challenger to beat champion on primary metric without regressing guardrails; keep rollback ready.

Use content/metadata features so new items get scored without interaction history, add exploration (bandits) to gather feedback, and blend popularity priors. Encode items via embeddings from content, and give new items a temporary boost to bootstrap data. Monitor cold-item coverage.

R^2 hides tail errors. Inspect residuals by segment/value range, weight or model high-value cases separately, use a loss aligned to business cost (e.g., asymmetric), and report error where it matters (MAE/MAPE on high-value bucket). Consider quantile regression for intervals.

Track proxy signals immediately: prediction distribution, feature drift, confidence, and volume; capture ground truth as it arrives (days/weeks later) to compute true metrics. Alert on drift/anomalies now, compute performance on a rolling basis once labels land, and sample cases for human review.

It's an MLOps/reliability gap, not a modeling one. Add data validation (schema/range checks), pipeline tests, retries/alerting, versioned data and code, and reproducible environments. Treat the pipeline as production software with SLAs; a fragile pipeline negates model quality.

Start simple (logistic/GBM) as a baseline: it's fast, interpretable, and often strong on tabular data. Add complexity only if the baseline underperforms and data volume/signal justify it, weighing latency, maintainability, and explainability. Let measured lift, not novelty, drive the choice.

Route easy queries to a smaller model, cache responses and prompt prefixes, trim context/max tokens, batch, and fine-tune a small model for the fixed task. Consider self-hosting at high volume. Track cost per request and gate features by budget.

Reproduce training, build a labeled eval set from production data, and establish current baseline metrics and failure modes. Add versioning, monitoring, and a champion-challenger process before changing anything. Document features/assumptions and check for leakage and fairness gaps.

Use a multi-objective approach: a weighted or constrained blend, or optimize one subject to guardrails on the other; tune weights via A/B tests on long-term metrics. Make tradeoffs explicit with stakeholders and monitor both objectives so one doesn't silently degrade.

Profile cost per prediction, then batch, cache, quantize/distill, right-size hardware (GPU vs CPU), and autoscale to demand. Precompute predictions for stable inputs, and drop unnecessary calls. Set cost SLOs and track cost alongside accuracy.

Recognize that serving decisions bias logged data (e.g., only shown items get clicks). Use exploration/randomization to collect unbiased data, apply inverse-propensity weighting/counterfactual eval, log what was shown, and monitor for degenerate loops. Refresh with unbiased samples.

Support deleting a user's data and its influence: remove from datasets and feature stores, retrain or use machine-unlearning where required, and track data lineage so you know what to purge. Keep audit logs of deletion, and design pipelines to re-derive models without the deleted data.

Invest in MLOps: automated pipelines, a feature store, versioned data/models, CI/CD with an eval gate, and experiment tracking. This makes changes reproducible and safe to ship frequently. Standardize eval so every iteration is measured, and enable quick rollback.

Not 0.5 by default: pick the threshold that optimizes the business objective on a validation set given the cost of false positives vs negatives (e.g., via the PR curve or expected cost). Re-tune as prevalence shifts, and calibrate probabilities first.

Add OOD/novelty detection to abstain or flag, calibrate confidence, and train with diverse data + augmentation. Use ensembles or MC-dropout for uncertainty, set a confidence threshold to defer to humans, and monitor for inputs unlike training data.

Estimate the metric it moves (e.g., conversion, cost saved), baseline performance, expected lift, and coverage, then multiply by volume and margin; subtract build + serving + maintenance cost. Validate assumptions with a quick heuristic baseline or offline analysis before committing to a full model.

Multicollinearity destabilizes linear-model coefficients (less so for trees). Drop or combine redundant features, use regularization (ridge/elastic net) or dimensionality reduction, and check VIF. For pure prediction with trees it may be fine, but it hurts interpretability.

Bound actions with rules and confidence thresholds, require human review for high-impact/low-confidence cases, add fairness and safety checks, cap the rate/scale of automated actions, and keep an audit log and easy override/rollback. Monitor outcomes and fail safe on anomalies.

Keep a true holdout untouched until the end, rotate validation sets, and prioritize the online business metric via A/B tests over offline scores. Watch for metric gaming and leakage, and reward generalization and production lift, not leaderboard position.

Self-host models on-prem/VPC, use on-device inference, and apply techniques like federated learning or differential privacy where training on distributed data. Encrypt data, restrict access, and avoid third-party APIs that egress data. Validate the privacy/utility tradeoff.

Frame it as learning: state the hypothesis, what was measured, why it didn't work (data, signal, or metric), and what you'd try next or when to stop. Show the evidence, quantify cost, and recommend a decision. Honest negative results prevent repeated mistakes.

Adversaries optimize against observable features. Use harder-to-game signals (behavioral, network, holistic), add anomaly detection and rate limits, retrain on new evasion patterns, and keep human review for edge cases. Design metrics and features assuming they'll be attacked.

Clarify the objective and a measurable success metric, identify decisions the model informs, check data availability and label feasibility, and estimate value vs cost. Propose a simple baseline/PoC with a clear eval before committing. Align on constraints (latency, privacy, fairness) up front.