Cheat sheet · Machine Learning

Feature engineering cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Feature engineering transforms raw data into informative inputs that maximize model performance, generalization, and interpretability. This cheat sheet covers numeric/categorical/temporal transforms, encoding, scaling, leakage-safe pipelines, selection, and modern (2026) practices like feature stores and pretrained embeddings.

Core Concepts & Workflow

Feature engineering = creating, transforming, and selecting variables so the signal is more learnable; on tabular data it often delivers larger gains than swapping models, where gradient-boosted trees (XGBoost/LightGBM/CatBoost) still dominate in 2026.
Garbage in, garbage out: model capacity cannot recover information destroyed by poor representation. Good features encode domain knowledge the model would otherwise have to learn from limited data.
Standard loop: EDA -> hypothesize features from domain knowledge -> implement as a reproducible transform -> validate with cross-validation -> measure importance/impact -> iterate. Keep transforms in a pipeline, never applied ad-hoc to the full dataset.
Tree-based models are invariant to monotone feature transforms and feature scale (no need to standardize or de-skew), while linear models, SVMs, KNN, k-means, PCA, and neural nets are sensitive to scale and distribution.
Deep learning (CNNs/transformers) does representation learning end-to-end, reducing manual features for text/image/audio; but tabular, time-series, and small-data problems still benefit heavily from explicit features.
Trade-off triangle: predictive lift vs. complexity/maintenance vs. interpretability. More features can raise variance, serving latency, and leakage risk.

Numeric Transformations & Scaling

Standardization (z-score) gives mean 0 / unit variance; Min-Max maps to a bounded range like [0,1]; RobustScaler (median/IQR) is best when outliers are present. Fit on TRAIN only, then transform val/test.
Skew handling: log1p for right-skewed non-negative data; Box-Cox (strictly positive) or Yeo-Johnson (handles zero/negative) power transforms push data toward normality.
Binning/discretization (equal-width, quantile, or supervised) can capture nonlinearity for linear models and dampen outliers; it discards information and adds boundary artifacts.
Interactions (x_i * x_j) and polynomial features expose nonlinear/multiplicative relationships; ratios and differences (e.g., debt/income) often encode domain logic compactly.
Outlier strategy: clip/winsorize, cap at percentiles, or use robust models; persist the exact rule so it is reproducible at inference.
Missing values: median/mean/mode imputation, model-based (KNN/iterative), or a sentinel constant PLUS a binary 'was_missing' indicator to preserve the missingness signal.

Categorical Encoding

One-hot: safe for low cardinality with linear/NN models; explodes dimensionality and is wasteful for trees. Use drop='first' to avoid the dummy-variable trap in linear models with an intercept.
Ordinal/label encoding: only when a true order exists, or for tree models, which can split on arbitrary integer codes; never feed fake order to linear models.
Target (mean) encoding: replace a category with a smoothed target statistic; powerful for high cardinality but it LEAKS unless computed out-of-fold (or on a separate split) with smoothing.
Frequency/count encoding is simple and leakage-light; the hashing trick handles very high or unseen cardinality with fixed memory, at the cost of collisions.
CatBoost uses ordered target statistics natively, sidestepping much manual encoding; LightGBM and sklearn HistGradientBoosting accept native categorical columns.
Always plan for unseen categories at inference (reserve an 'unknown' bucket) and group rare categories into 'other'.

Temporal, Text & Cyclical Features

Datetime: extract year/month/day/day-of-week/hour/is_weekend/is_holiday; encode periodic fields with sin/cos so 23:00 and 00:00 are adjacent.
Cyclical encoding: x_sin = sin(2*pi*t/T), x_cos = cos(2*pi*t/T) for hour (T=24), month (T=12), day-of-week (T=7).
Time series: lag features (y_{t-1..t-k}), rolling/expanding aggregates (mean/std/min/max), diffs, and time-since-event; use ONLY past data and time-ordered splits.
Text (classic): TF-IDF / n-grams plus cleaning (lowercasing, stemming) for fast linear baselines; add length, sentiment, and keyword flags.
Text (modern, 2026): sentence/document embeddings from transformer encoders (e.g., MiniLM, BGE, E5) as dense vectors feeding downstream tabular models or vector search.
Image/audio: pretrained-backbone embeddings as features; geospatial: distance to POIs, geohash buckets, lat/long clustering.

Feature Selection & Dimensionality Reduction

Filter methods: variance threshold, correlation pruning, mutual information, chi-square (non-negative categorical features) — fast, model-agnostic, run first.
Wrapper methods: RFE, forward/backward selection — accurate but expensive, with a real risk of overfitting the selection to the CV folds.
Embedded methods: L1/Lasso (sparsity) and tree feature_importances_; prefer permutation importance / SHAP for model-faithful, less-biased rankings.
Beware impurity-based (Gini/gain) tree importance: it inflates high-cardinality and correlated features; prefer permutation importance or SHAP on held-out data.
Dimensionality reduction: PCA (linear, scale-sensitive) for decorrelation/compression; UMAP/t-SNE for visualization only (not as reproducible model inputs); autoencoders for nonlinear compression.
Drop redundant correlated features to cut variance and improve interpretability; for trees, redundancy mainly distorts importance attribution rather than accuracy.

Pipelines, Leakage Safety & Production

Wrap ALL stateful transforms (impute, scale, encode, select) in a single Pipeline/ColumnTransformer so fitting happens only on the training fold — the #1 defense against leakage during CV.
Data leakage = using information at train time that is unavailable (or from the future) at prediction time; symptoms: too-good CV that collapses in production. Covers target leakage and train/test contamination.
Train/serve skew: identical transform code and fitted artifacts must run offline and online. Persist fitted transformers (and category maps) and version them with the model.
Feature stores (Feast, Tecton, Databricks, Vertex) centralize feature definitions, provide point-in-time-correct historical joins for training, and low-latency online serving — the 2026 standard for sharing features across teams.
Point-in-time correctness: when joining features to events, use only feature values that existed at that timestamp to avoid lookahead leakage.
Monitor feature drift and data quality in production; null spikes, distribution shift, and stale features silently degrade models.

Common pitfalls

Fitting scalers/encoders/imputers on the full dataset before splitting — leaks test statistics into training and inflates CV scores.
Target/mean encoding without out-of-fold computation or smoothing, causing target leakage and overfitting on high-cardinality columns.
Standardizing or one-hot-encoding for tree models where it adds no value (one-hot can even hurt split quality on high-cardinality features).
Trusting impurity-based feature_importances_, which are biased toward high-cardinality/continuous and correlated features; use permutation importance or SHAP.
Random K-fold on time-series data, leaking the future into the past; use time-ordered / forward-chaining splits and lag-only features.
Creating features from future or post-outcome information (e.g., fields populated after the label is known) — classic target leakage.
Train/serve skew: different transform logic offline vs. online, or forgetting to handle unseen categories and missing values at inference.
Over-engineering: hundreds of correlated/redundant features that raise variance, latency, and maintenance with little lift; ignoring the cost side.
Encoding cyclical time as raw integers, breaking the 23->0 wraparound continuity.
Using t-SNE/UMAP coordinates as model features — they are non-parametric and not reproducibly mappable to new data.

Interview tips

Define data leakage precisely, give 2-3 concrete examples (target leakage, train/test contamination, lookahead in time series), and explain how a Pipeline plus proper CV prevents it.
Explain when scaling/encoding matters by model family (linear/SVM/KNN/NN/PCA need it; trees largely don't) — interviewers probe whether you understand WHY.
Walk through high-cardinality categorical handling end-to-end: one-hot vs. target vs. frequency vs. hashing vs. CatBoost, with leakage-safe target encoding.
Discuss target encoding's leakage trap and the out-of-fold + smoothing fix; this is a frequent senior-level discriminator.
Describe time-series feature engineering: lags, rolling windows, time-ordered splits, and why point-in-time correctness matters.
Compare filter vs. wrapper vs. embedded selection and justify permutation importance / SHAP over default tree importances.
Show production awareness: feature stores, train/serve consistency, versioning fitted transformers, and drift monitoring — strong signal for ML-engineer roles in 2026.
Have a story about a feature that gave real lift (or one that leaked and was caught), showing EDA-to-validation judgment, not just API knowledge.

Read the full lesson Test yourself — 34 question quiz