Cheat sheet · System Design

ML system design cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

A structured playbook for designing production ML systems end to end: framing the problem, building data and feature pipelines, training and serving architecture, evaluation, and post-deployment monitoring. Optimized for ML system design interviews and real-world ML/LLM engineering in 2026.

1. Problem Framing & Requirements

Translate a vague business goal into a concrete ML task: classification / regression / ranking / retrieval / generation, plus the business metric it moves (revenue, retention, engagement, cost).
Separate the ML objective (the loss you optimize) from the business objective (the KPI). Use a proxy metric only if it correlates with the KPI, and validate that link with an A/B test.
Pin down scale and SLAs early: QPS, per-request latency budget (e.g. p99 < 100 ms), data volume, training cadence (batch vs streaming), and feature freshness requirements.
Decide online vs batch prediction. Online = low-latency per-request inference; batch = precomputed predictions stored for lookup (cheaper, but staler).
Clarify constraints: privacy / PII, regulatory (GDPR, EU AI Act, model explainability), on-device vs cloud, budget, and team / maintenance capacity.
State assumptions explicitly and confirm them with the interviewer before designing — scoping is graded as heavily as the architecture itself.

2. Data & Feature Engineering

Data sources: user/event logs, transactional DBs, third-party, and derived data. Define labels carefully: explicit (ratings) vs implicit (clicks); beware label delay and selection bias.
Handle class imbalance (resampling, class weights, focal loss), missing values, and leakage — never compute a feature from information unavailable at inference time.
Use a feature store (Feast, Tecton, Vertex / SageMaker Feature Store) to share features across teams and guarantee train/serve parity via point-in-time-correct joins.
Avoid training-serving skew: reuse the exact same transformation code in training and serving, and log served features for later training (log-and-replay).
Split chronologically for time-dependent data (train < val < test by time) to avoid look-ahead leakage; use stratified random splits only for static i.i.d. data.
Version data and features (DVC, lakeFS, Delta Lake / Iceberg) so any model run is reproducible from raw inputs.

3. Modeling & Training Infrastructure

Always ship a baseline first (heuristic / logistic regression / popularity) — it sets a floor and validates the pipeline before heavy modeling.
Model choice is a tradeoff of accuracy vs latency vs interpretability vs cost. Gradient-boosted trees (XGBoost / LightGBM) dominate tabular; deep nets for text/image/sequence; two-tower retrieval for large-scale recsys.
When labeled data is scarce, prefer transfer learning / fine-tuning, parameter-efficient tuning (LoRA), or RAG over a foundation model; train from scratch only with ample data and a clear need.
Scale training with data/model parallelism (FSDP, DeepSpeed ZeRO), mixed precision, and gradient accumulation; track runs with MLflow / W&B and pin seeds plus dependency versions.
Retraining strategy: scheduled (daily/weekly), drift-triggered, or continual/online learning. Decide explicitly when and how it happens automatically.
Tune hyperparameters with Bayesian search or ASHA/Hyperband, not exhaustive grid; budget compute and use early stopping.

4. Evaluation & Offline Metrics

Pick metrics that match the task and the cost of errors: PR-AUC / F1 for imbalanced classification, ROC-AUC for threshold-free ranking quality (but it can look optimistic under heavy imbalance), RMSE / MAE for regression.
Ranking / recsys: NDCG, MAP, MRR, Recall@k — plus diversity / coverage / novelty so you don't over-optimize relevance alone.
Use sliced / per-segment evaluation to catch fairness gaps and tail failures hidden by aggregate metrics.
Calibrate probabilities (Platt scaling or isotonic regression) when downstream decisions treat the score as a true probability.
For LLM/generative systems, evaluate with task-specific suites: golden-set graders, LLM-as-judge (with bias controls), faithfulness/groundedness for RAG, and human eval; pure perplexity is not enough.
Confirm offline gains online via A/B test (or interleaving for ranking); guard against novelty and network effects, and run shadow / backtest evaluation on historical data before any live exposure.

5. Serving Architecture & Deployment

Patterns: REST/gRPC microservice, batch precompute + KV store (Redis / DynamoDB) for lookup, streaming inference, or on-device/edge for privacy and latency.
Optimize inference: quantization (int8 / fp16), distillation, pruning, compiled runtimes (ONNX Runtime / TensorRT), request batching, and caching of hot predictions/embeddings.
For LLM serving specifically: continuous (in-flight) batching, paged KV-cache, prefix/prompt caching, speculative decoding, and tensor parallelism (vLLM, TGI, TensorRT-LLM) to maximize tokens/sec per GPU.
Use a multi-stage funnel for large candidate sets: cheap retrieval (ANN) -> ranking -> re-ranking / business rules, to meet latency at scale.
Safe rollouts: shadow deployment, canary, blue-green, and gradual traffic ramp, with instant rollback and a model registry that tracks lineage.
Decouple model artifacts from serving code: load models from a registry (MLflow / SageMaker / Vertex) so a deploy is a config change. Autoscale on QPS / GPU utilization and isolate heavy models so one slow model can't starve the shared pool.

6. Monitoring, Drift & Feedback Loops

Monitor three layers: system (latency, error rate, throughput), data quality (schema, nulls, ranges, freshness), and model (prediction distribution, online metrics).
Detect drift: data/covariate drift on inputs (PSI, KL divergence, KS test) and concept drift = degrading performance once labels arrive; alert and trigger retraining.
Close the loop with delayed ground-truth labels; watch for feedback loops where the model's own outputs bias future training data (e.g. recsys popularity spirals).
Log predictions plus feature snapshots per request to enable debugging, replay, and root-cause analysis.
Automate retraining behind CI/CD for models (CT = continuous training): validate, test, and gate each new model before promotion.
For LLM apps, monitor hallucination/groundedness, toxicity/safety, prompt-injection attempts, token cost, and latency as first-class signals. Track cost-per-prediction and model staleness as explicit SLOs.

Common pitfalls

Jumping straight to model choice without scoping requirements, scale, and metrics — interviewers dock heavily for this.
Data leakage: using future information or target-derived features that won't exist at inference time, producing inflated offline scores.
Training-serving skew from duplicating feature logic in two codebases instead of sharing transformations.
Optimizing the ML loss while ignoring the business KPI, or assuming offline gains automatically translate online.
Using accuracy on imbalanced data — a 99% majority class makes accuracy meaningless; use PR-AUC / F1 with per-class analysis.
Ignoring feedback loops: the deployed model changes the very data distribution it is later retrained on.
No baseline, so there is nothing to prove the complex model actually helps.
Treating monitoring and retraining as an afterthought; models silently rot from drift after launch.
Designing for peak accuracy with no regard for latency, cost, or maintainability constraints.
For LLM systems: no eval harness or guardrails, unbounded context/token cost, and no defense against prompt injection.

Interview tips

Drive the conversation: restate the problem, ask clarifying questions, and state assumptions before drawing any architecture.
Lead with metrics — name both the offline ML metric and the online business metric, and how you'd A/B test the link between them.
Always mention a simple baseline and justify any added complexity against it.
Explicitly call out batch vs online prediction and online vs offline evaluation; interviewers probe whether you know the difference.
Be ready to deep-dive any box you draw: feature store internals, ANN retrieval, A/B test design, or drift-detection math.
Discuss train/serve parity and data leakage proactively — these are senior-signal topics.
Cover the full lifecycle through monitoring, retraining, and rollback; stopping at 'train a model' signals junior level.
Talk tradeoffs out loud (accuracy vs latency vs cost vs interpretability) rather than presenting one 'right' answer.
Quantify: give concrete numbers for QPS, latency budget, data size, and retraining cadence to ground the design.
If the prompt is LLM/GenAI-flavored, cover RAG vs fine-tuning, eval harness, guardrails, and token-cost/latency tradeoffs.

Read the full lesson Test yourself — 40 question quiz