Skip to content
elevatedevco

Cheat sheet · Data & MLOps

EDA & Data Cleaning cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

A dense, practical reference for exploratory data analysis and data cleaning covering profiling, missing data, outliers, encoding, scaling, and leakage-safe pipelines, with the formulas, code patterns, and interview probes that matter in 2026.

EDA Workflow & Profiling

  • Frame first: know the target, the unit of analysis (one row = what?), and the grain before touching distributions; mismatched grain causes silent fan-out joins and duplicated rows.
  • Univariate -> bivariate -> multivariate: start with dtypes, cardinality, missingness, and distributions; then target-vs-feature relationships; then correlations/interactions.
  • Numeric: inspect mean/median/std, skew, kurtosis, min/max, and quantiles; categorical: value_counts, cardinality, rare-level mass. Always plot (hist, box, ECDF) - summary stats hide bimodality (Anscombe's quartet).
  • Quantify skew/kurtosis: skew ~ 0 is symmetric; |skew| > 1 is highly skewed and may need log/Box-Cox. Use median over mean for skewed central tendency.
  • Automate the first pass with ydata-profiling / Sweetviz / df.describe(include='all'), but manually verify - auto-profilers misread IDs, dates, and zero-inflated columns.
  • Check duplicates, constant/near-constant columns, leakage-prone fields (post-outcome timestamps), and ID columns that masquerade as features.
  • EDA is iterative and hypothesis-driven (Tukey): each plot should answer or raise a question, not just exist.

Missing Data: Mechanisms & Imputation

  • Classify the mechanism: MCAR (missingness independent of everything), MAR (depends on observed data), MNAR (depends on the unobserved value itself). Strategy and bias depend on which.
  • Listwise deletion is only unbiased under MCAR and wastes data; under MAR/MNAR it biases estimates.
  • Simple imputation (mean/median/mode) shrinks variance and distorts correlations; median is robust to skew. Always add a binary 'was_missing' indicator - missingness is often predictive.
  • Model-based imputation: KNNImputer (local similarity) or IterativeImputer/MICE (chained regression) preserve relationships better. Note sklearn's IterativeImputer does a single imputation by default; set sample_posterior=True (and run multiple times) for proper multiple imputation that captures uncertainty.
  • For time series, use forward/backward fill or interpolation - never use future values to fill the past (backfill at a row used for prediction is leakage).
  • MNAR often needs domain handling: encode 'not applicable' as its own category or model the missingness explicitly; imputation alone hides the signal.
  • Critical: fit imputers on TRAIN only, then transform validation/test - fitting on the full dataset leaks holdout statistics.

Outliers & Distribution Fixes

  • Detect with IQR rule (Q1-1.5*IQR, Q3+1.5*IQR), z-score (|z|>3) for ~normal data, or robust z via MAD for skewed/heavy-tailed data; multivariate via IsolationForest / LOF.
  • z = (x - mu)/sigma is sensitive to the very outliers it detects; the modified z-score using median and MAD (0.6745*(x-median)/MAD) is more robust.
  • Decide intent: errors -> fix/drop; legitimate extremes -> keep but make models robust. Don't blindly delete - outliers can be the signal (fraud, churn, failures).
  • Winsorize/clip to percentiles to cap influence without discarding rows; tree models are largely outlier-insensitive, linear/distance models are not.
  • Transform skew: log1p for right-skewed non-negative data, Box-Cox (strictly positive only) or Yeo-Johnson (handles zeros/negatives) to approach normality and stabilize variance.
  • Document every removal with a rule and count; ad-hoc deletion is unreproducible and a red flag in review.

Encoding Categoricals & Feature Types

  • Nominal low-cardinality -> one-hot (drop_first to avoid the dummy-variable trap / collinearity in linear models). Ordinal -> integer mapping that respects order.
  • High-cardinality (zip, user_id) -> target/mean encoding with smoothing and cross-fold fitting, hashing, or learned embeddings; one-hot explodes dimensionality.
  • Target encoding leaks the label unless done out-of-fold (K-fold or leave-one-out) and fit on train only - this is a top interview trap.
  • Frequency/count encoding is cheap and often strong for tree models; rare-category grouping ('Other' below a threshold) controls noise and cardinality.
  • Cast types correctly: parse dates to datetime and derive (dow, month, is_weekend, cyclical sin/cos); booleans to 0/1; downcast to save memory and avoid object dtype.
  • Tree-based models (LightGBM/CatBoost) handle categoricals natively - CatBoost uses ordered target statistics to avoid leakage; prefer native handling over manual encoding when available.

Scaling, Normalization & Text/Date Hygiene

  • StandardScaler (z-score) for ~Gaussian; MinMaxScaler for bounded ranges; RobustScaler (median/IQR) when outliers present. Required for distance/gradient models (KNN, SVM, k-means, NN, regularized linear); irrelevant for trees.
  • Normalization scales rows to unit norm (e.g., for cosine similarity); standardization scales columns - different operations, often confused.
  • Always fit the scaler on train and transform test; fit-on-all is leakage. Wrap in a Pipeline/ColumnTransformer so the fit is bound to CV folds.
  • Text cleaning: normalize case, strip/collapse whitespace, fix encoding (UTF-8, mojibake), unify units and category spellings ('NY'/'New York'); standardize before grouping.
  • Dates/timezones: parse to a single tz (usually UTC), watch DST and locale formats; deduplicate on a canonical key after normalization.
  • Validate ranges and constraints (age >= 0, prob in [0,1], end >= start) with schema checks (Pandera / Great Expectations) so dirty data fails loudly in pipelines.

Leakage-Safe Pipelines & Reproducibility

  • Golden rule: every fitted transform (impute, scale, encode, select) learns parameters from TRAIN only; use sklearn Pipeline + ColumnTransformer inside cross-validation so each fold is isolated.
  • Split before you clean. Computing stats, dedup, or feature selection on the full dataset before splitting leaks the test set.
  • Time-aware data needs time-based splits (TimeSeriesSplit) and as-of joins; random k-fold leaks future into past.
  • Watch target leakage: features computed using the outcome, post-event fields, or aggregates over the label. If a feature is 'too good', suspect leakage.
  • Make cleaning deterministic and versioned: set seeds, log row counts at each step, version data + code (DVC/lakeFS), and keep raw immutable - clean in a reproducible pipeline, not by hand in a notebook.
  • Track data quality over time (drift, schema, null rates) so cleaning logic doesn't silently rot in production.
Read the full lesson