Skip to content
elevatedevco

Cheat sheet · Machine Learning

Gradient Boosting (XGBoost/LightGBM) cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Cheat sheet

Gradient boosting builds an additive ensemble of shallow decision trees, each fitted to the gradient (and, for XGBoost/LightGBM, the Hessian) of a differentiable loss, giving state-of-the-art accuracy on tabular data. XGBoost, LightGBM, and CatBoost are the dominant production libraries, differing mainly in tree-growth strategy, histogram binning, sampling tricks, and native categorical handling.

Core Algorithm & Math

  • Additive model: F_M(x) = F_0 + sum_{m=1..M} eta * h_m(x), where F_0 is a base score (e.g. mean / log-odds), each h_m is a regression tree, and eta is the learning rate (shrinkage).
  • Stage-wise: at iteration m fit a new tree to the negative gradient of the loss w.r.t. current predictions (functional gradient descent in function space).
  • XGBoost uses a 2nd-order Taylor expansion: per sample it uses gradient g_i = dL/dF and Hessian h_i = d^2L/dF^2; LightGBM also uses both (CatBoost uses gradients with ordered boosting).
  • Optimal leaf weight w*_j = -G_j / (H_j + lambda), where G_j, H_j are the sums of g_i, h_i in the leaf; the split gain measures the loss reduction from that closed-form leaf value.
  • Regularization is built into the objective: gamma penalizes the number of leaves, lambda (L2) and alpha (L1) penalize leaf weights — so trees are pruned by gain, not just depth.
  • Boosting reduces bias by sequentially correcting residual errors; contrast with bagging/Random Forest which reduces variance by averaging independent, fully-grown trees.

XGBoost vs LightGBM (the classic interview compare)

  • Tree growth: XGBoost grows level-wise/depth-wise by default (grow_policy='depthwise'); LightGBM grows leaf-wise (best-first, splits the leaf with max delta loss). Leaf-wise is faster and often more accurate but more prone to overfit — control via num_leaves and min_data_in_leaf. XGBoost can mimic it with grow_policy='lossguide'.
  • Both use histogram-based split finding (binning continuous features into <=255 bins, max_bin default 256) for speed; XGBoost's 'hist' tree method is the default since 2.0 and matches LightGBM's original design.
  • LightGBM speedups: GOSS (Gradient-based One-Side Sampling, keeps large-gradient samples plus a random subset of small ones) and EFB (Exclusive Feature Bundling, bundles sparse mutually-exclusive features).
  • Categorical features: LightGBM has native optimal categorical splitting (no one-hot needed); XGBoost supports native categoricals via enable_categorical=True (stable since 1.6/2.x) but historically required encoding.
  • Memory & speed: LightGBM is typically lower memory and faster on large/wide data; XGBoost is very robust, offers DART and linear boosters, GPU via device='cuda', and a broad ecosystem (Spark, Dask, external/out-of-core memory).
  • CatBoost is the third sibling worth naming: ordered boosting + ordered target statistics to fight target leakage, strong on high-cardinality categoricals with minimal tuning.

Key Hyperparameters & Tuning

  • Learning rate (eta / learning_rate) vs n_estimators: lower eta + more trees generalizes better; tune jointly and use early stopping on a validation set.
  • Tree complexity: XGBoost max_depth (3-10 typical); LightGBM num_leaves (keep < 2^max_depth) plus min_data_in_leaf / min_child_weight to prevent tiny overfit leaves.
  • Regularization: reg_lambda (L2), reg_alpha (L1), and gamma / min_split_gain raise the bar for splits.
  • Sampling for variance reduction: subsample (rows per tree) and colsample_bytree / feature_fraction (columns per tree) act like stochastic gradient boosting.
  • Imbalanced data: scale_pos_weight (binary) or class weights / custom objective; tune the decision threshold and optimize AUC / PR-AUC rather than accuracy.
  • Practical order: set high n_estimators + early stopping, fix a moderate eta, tune depth/num_leaves & min_child, then regularization & sampling; prefer Bayesian search (Optuna) over grid search.

Loss Functions & Custom Objectives

  • Regression: squared error (gradient = residual), pseudo-Huber, quantile (pinball) loss for prediction intervals, Tweedie for insurance / zero-inflated counts.
  • Classification: logistic / cross-entropy; multiclass softmax; predictions are raw log-odds (margins) — apply sigmoid/softmax to get probabilities.
  • A custom objective just needs to return per-sample gradient g and Hessian h; both libs accept an obj callable (sign convention: g = dL/dy_pred).
  • Eval metric can differ from the training objective (e.g. train logloss, early-stop on AUC); pick the eval metric that matches the business KPI.
  • Monotonic constraints (monotone_constraints) and interaction constraints let you inject domain knowledge and improve trust/robustness.
  • Probability calibration is often needed: boosted trees can be over/under-confident — use Platt scaling or isotonic regression on a holdout.

Interpretability & Diagnostics

  • Built-in feature importance: 'gain' (avg loss reduction, preferred), 'cover' (samples affected), 'weight'/'frequency' (split count) — weight is biased toward high-cardinality features.
  • SHAP (TreeSHAP) gives consistent, locally accurate per-sample attributions and is the de-facto standard for explaining GBMs; supports global summary and dependence plots.
  • Partial dependence (PDP) and ICE plots show the marginal effect of a feature; watch for misleading results under correlated features.
  • Use eval curves (train vs val metric per round) to diagnose overfitting and pick the early-stopping iteration.
  • Trees handle missing values natively (default split direction learned per node) — don't blindly impute; let the model route NaNs.
  • No feature scaling needed (tree splits are scale-invariant); monotone transforms of a single feature don't change splits.
Read the full lesson