Cheat sheet · Machine Learning

Support vector machines cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

SVMs are max-margin classifiers (and regressors) that find the optimal separating hyperplane, using the kernel trick to model nonlinear data and the C / epsilon / gamma hyperparameters to trade off margin width against errors. They excel on small-to-medium, high-dimensional datasets but scale poorly (roughly O(n^2)-O(n^3)) on very large data, where linear SVMs, GBMs, or deep nets are preferred.

Core Idea & Margin Geometry

Goal: find the hyperplane w·x + b = 0 that maximizes the margin (distance to the nearest points of each class); a larger margin yields a tighter generalization bound (smaller effective VC dimension), so it tends to generalize better.
Geometric margin of a point = y_i(w·x_i + b)/||w||. Fixing the functional margin to 1 makes maximizing the margin equivalent to minimizing (1/2)||w||^2 subject to the constraints.
Support vectors are the only training points on or inside the margin (alpha_i > 0); the boundary depends solely on them — removing any non-support-vector leaves the solution unchanged.
Decision function: f(x) = sign(w·x + b). The signed distance is a confidence-like score, NOT a calibrated probability.
SVM is a discriminative, deterministic, large-margin classifier; the primal/dual is a convex quadratic program, so there is a unique global optimum (no local minima, unlike neural nets).

Hard vs Soft Margin (Primal Optimization)

Hard margin assumes perfect linear separability — brittle and undefined if classes overlap; rarely used in practice.
Soft margin adds slack variables xi_i >= 0 to permit violations, penalized by C.
C is the regularization knob: LARGE C penalizes misclassification heavily => narrow margin, low bias / high variance (overfit). SMALL C => wider margin, more tolerance => high bias / low variance.
Equivalent hinge-loss view: minimize (1/2)||w||^2 + C * sum_i max(0, 1 - y_i(w·x_i + b)). The hinge loss is what makes the solution sparse in support vectors (it is non-differentiable at the kink, so solvers use subgradients/QP).
sklearn convention: C is the inverse of regularization strength, so SMALLER C = STRONGER regularization (opposite of the usual lambda intuition).

Dual Formulation & The Kernel Trick

The Lagrangian dual replaces w with multipliers alpha_i and depends on the data only through inner products x_i·x_j, which is what enables kernels.
Kernel trick: replace x_i·x_j with K(x_i, x_j) = phi(x_i)·phi(x_j) to operate in a high- (possibly infinite-) dimensional feature space WITHOUT ever computing phi explicitly.
Common kernels: linear (text / high-dim sparse), RBF/Gaussian K = exp(-gamma ||x - x'||^2) (default nonlinear choice), polynomial (x·x' + c)^d, sigmoid.
RBF gamma controls reach: LARGE gamma = each point's influence is local => wiggly boundary, overfit; SMALL gamma = smoother / underfit. C and gamma interact, so tune them jointly via grid/random search on a log scale.
Valid kernels must be symmetric and positive semi-definite (Mercer's condition). At the optimum the KKT conditions hold, and alpha_i > 0 only for support vectors.
Mapping to a higher-dimensional space makes data more likely to be linearly separable (Cover's theorem).

Variants: SVR, Multiclass, Probabilities

SVR (regression) uses an epsilon-insensitive tube: errors within +/-epsilon cost nothing; only points outside the tube become support vectors.
nu-SVM/nu-SVR (NuSVC/NuSVR) reparameterize C/epsilon with nu in (0,1], which bounds the fraction of support vectors and margin errors — handy when that fraction is easier to reason about.
SVM is inherently binary; multiclass is built via One-vs-Rest (n classifiers) or One-vs-One (n(n-1)/2 classifiers). sklearn's SVC/NuSVC use One-vs-One; LinearSVC uses One-vs-Rest.
Probability estimates require Platt scaling (a logistic fit on scores via internal k-fold CV); set probability=True in sklearn — it is slower and predict_proba can disagree with predict() near the boundary. Often better to threshold decision_function.
LinearSVC / linear-kernel SGD scale far better (liblinear/SAGA, ~O(n)) than kernel SVC; prefer them for large/sparse data like TF-IDF text. Note: LinearSVC defaults to squared-hinge loss and regularizes the intercept, so it is not identical to SVC(kernel='linear').
Class imbalance: use class_weight='balanced' to scale C per class, or adjust the decision threshold.

Practical Use, Complexity & Tradeoffs

ALWAYS scale/standardize features — SVMs are distance-based and sensitive to feature ranges (critical for RBF/poly). Fit the scaler inside a Pipeline to avoid leakage.
Kernel-SVM training is roughly O(n^2)-O(n^3) in samples and O(n_SV * d) at inference => impractical beyond ~100k rows; switch to LinearSVC, SGDClassifier(loss='hinge'), or trees/GBMs.
Strengths: effective in high-dimensional / small-sample regimes, robust via the max-margin objective, memory-efficient (stores only support vectors), strong theoretical grounding.
Weaknesses: slow on large data, hard to interpret (especially nonlinear kernels), no native calibrated probabilities, sensitive to C / gamma / kernel choice.
gamma='scale' = 1 / (n_features * X.var()) is a sensible default; tune C and gamma on a log grid (e.g. 1e-3 .. 1e3).
In 2026, SVMs remain a strong baseline for small tabular and text problems, but GBMs (XGBoost/LightGBM/CatBoost) and deep nets dominate large-scale and unstructured data.

Common pitfalls

Forgetting to standardize features — the single most common SVM mistake; it wrecks RBF/poly performance.
Misreading C: in sklearn a SMALLER C means MORE regularization (wider margin), the opposite of penalty-weight intuition.
Running a kernel SVM on hundreds of thousands of rows — training blows up to O(n^2)-O(n^3); use LinearSVC/SGD instead.
Confusing high gamma with smoothing — high gamma actually OVERFITS (very local influence).
Assuming SVM outputs probabilities; decision_function scores are uncalibrated and need probability=True (Platt scaling).
Tuning C and gamma independently — they interact strongly and must be searched jointly.
Calling support vectors 'the most important features' — they are training examples, not features.
Ignoring class imbalance; without class_weight the margin can be dominated by the majority class.
Assuming LinearSVC(kernel) equals SVC(kernel='linear') — different loss (squared hinge) and intercept regularization.

Interview tips

Derive why maximizing the margin equals minimizing (1/2)||w||^2, and write both the primal and dual forms.
Explain the kernel trick precisely: it computes inner products in feature space without materializing phi(x).
Articulate the bias-variance effect of C and of RBF gamma, including which direction overfits.
Define support vectors and explain why the model is sparse and depends only on them (via KKT conditions).
State Mercer's condition / what makes a valid kernel (symmetric, positive semi-definite).
Compare SVM vs logistic regression (hinge vs log loss, margin vs calibrated probability) and vs trees/GBMs (scaling, interpretability, large-n behavior).
Discuss computational complexity and when SVMs are the wrong tool (very large datasets).
Know hinge loss and how soft-margin SVM equals regularized hinge-loss minimization.
Mention SVR's epsilon-tube and multiclass strategies (OvO for SVC vs OvR for LinearSVC) when asked beyond binary classification.

Read the full lesson Test yourself — 34 question quiz