SVM kernels explained

SVM is a supervised machine learning algorithm that finds the optimal hyperplane separating two classes by maximizing the margin between them. Support vectors are the training points closest to the hyperplane that define its position. SVM works for both classification (SVC) and regression (SVR) tasks.

The kernel trick computes the dot product in a high-dimensional (even infinite-dimensional) feature space without explicitly transforming the data. A kernel function K(x, z) returns the same value as the dot product φ(x)·φ(z) in the transformed space, making it computationally efficient. This allows SVM to learn nonlinear boundaries while working entirely in the original input space.

The linear kernel is K(x, z) = x · z, the standard dot product between two feature vectors. It performs no transformation — SVM operates directly in the original input space and finds a linear hyperplane. It is preferred when data is linearly separable or when the input is already very high-dimensional and sparse, such as TF-IDF text vectors.

The RBF (Gaussian) kernel is K(x, z) = exp(−γ||x − z||²), where γ > 0 controls the width of the Gaussian. It measures similarity based on the Euclidean distance between two points: nearby points have a kernel value close to 1, while distant points approach 0. It implicitly maps data to an infinite-dimensional feature space.

Gamma controls how far the influence of a single training point reaches. A high gamma makes the kernel very narrow so each point only affects its immediate neighbors — creating a complex, jagged decision boundary that risks overfitting. A low gamma lets influence spread farther, producing a smoother, simpler boundary that may underfit.

The polynomial kernel is K(x, z) = (γ · x · z + c)^d, where d is the degree, c is a constant offset, and γ is a scaling factor. The degree controls the order of the polynomial decision boundary: d=2 gives quadratic boundaries, d=3 cubic, and so on. It is effective when pairwise or higher-order feature interactions are important, such as in image recognition tasks.

C is the regularization parameter that balances margin size against training errors. A large C penalizes misclassifications heavily, shrinking the margin and potentially overfitting the training data. A small C allows more misclassifications in exchange for a wider, more robust margin, which generally improves generalization on unseen data.

Support vectors are the training data points that lie closest to the decision hyperplane, sitting exactly on the margin boundaries. Only these points determine the position and orientation of the hyperplane — removing all other training points would leave the model unchanged. This makes SVM memory-efficient at prediction time and robust to irrelevant data points far from the boundary.

XOR data is not linearly separable — the positive and negative examples alternate in a checkered pattern so no straight hyperplane can divide them. A linear SVM can only draw flat decision surfaces, which is fundamentally insufficient for this pattern. An RBF kernel maps the data to a higher-dimensional space where the XOR pattern becomes linearly separable, allowing SVM to find a clean decision boundary.

Mercer's theorem states that a symmetric function K(x, z) is a valid kernel if and only if the Gram matrix K[i,j] = K(xᵢ, xⱼ) is positive semi-definite for every possible choice of data points. This guarantees that K corresponds to a dot product in some (possibly infinite-dimensional) feature space. It ensures the SVM optimization problem remains convex with a unique global solution.

The Gram matrix is an n×n matrix where entry K[i,j] = K(xᵢ, xⱼ) stores the kernel value between every pair of training examples. In SVM's dual formulation, training depends only on this matrix — never on the explicit (possibly infinite-dimensional) feature vectors. The matrix must be positive semi-definite for the quadratic programming optimization to have a unique global minimum.

Hard-margin SVM requires every training point to be correctly classified with no margin violations — it only works when data is perfectly linearly separable. Soft-margin SVM introduces slack variables ξᵢ ≥ 0 that allow some points to be inside or on the wrong side of the margin, with the trade-off controlled by C. In practice, real-world noisy data almost always requires soft-margin SVM.

Using the Taylor series expansion of the exponential, exp(−γ||x−z||²) can be written as an infinite sum where each term corresponds to a polynomial feature interaction of increasing order. This is equivalent to a dot product in an infinite-dimensional Hilbert space. The key insight is that we only compute the single scalar K(x, z) — never the infinite feature vector itself.

The code creates a Pipeline that first applies StandardScaler (essential since RBF uses Euclidean distance) and then fits an SVC with kernel='rbf'. The gamma='scale' option automatically sets γ = 1/(n_features × Var(X)). Wrapping in a Pipeline prevents data leakage by fitting the scaler only on training data.

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

model = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(kernel='rbf', C=1.0, gamma='scale'))
])
model.fit(X_train, y_train)

The RBF kernel computes exp(−γ||x − z||²), which depends on the squared Euclidean distance between data points. If one feature spans [0, 10000] and another spans [0, 1], the large-magnitude feature completely dominates the distance calculation and the small-scale feature is effectively ignored. StandardScaler (zero mean, unit variance) ensures each feature contributes equally to the kernel computation.

gamma='scale' sets γ = 1 / (n_features × Var(X)), automatically adapting to both the number of features and the variance of the training data. The older default gamma='auto' used only 1/n_features, ignoring data variance, which could lead to poor kernel behavior. gamma='scale' became the new default in sklearn 0.22 because it is more robust across different datasets.

The linear kernel typically outperforms RBF for text data. TF-IDF features produce very high-dimensional, sparse vectors in which the data is often already approximately linearly separable. The RBF kernel can be slow to tune and may underperform on such sparse inputs, while the linear kernel scales efficiently (using solvers like LIBLINEAR) and generalizes well in high-dimensional settings.

coef0 (also written as c or r₀) is the independent constant in the kernel formula: K(x,z) = (γ·x·z + coef0)^d for polynomial and K(x,z) = tanh(γ·x·z + coef0) for sigmoid. In the polynomial kernel it controls the influence of lower-degree terms — a coef0 of 0 means only pure interaction terms (like x₁·x₂) are captured, while a nonzero value also includes individual feature effects. The default value in sklearn is 0.

The sigmoid kernel K(x,z) = tanh(γ·x·z + c) produces a Gram matrix that is positive semi-definite only for specific parameter ranges (γ > 0 and c ≤ 0). For parameter values outside this range, the Gram matrix can have negative eigenvalues, violating Mercer's condition. This means the SVM optimization may not be convex, so convergence to the global optimum is no longer guaranteed.

Standard PCA finds linear principal components — directions of maximum variance in the original input space. Kernel PCA applies PCA in the implicit high-dimensional feature space defined by a kernel function, capturing nonlinear structure that linear PCA misses. Using the kernel trick, kernel PCA computes eigenvectors of the centered Gram matrix without ever constructing the high-dimensional feature vectors explicitly.

GridSearchCV exhaustively trains and evaluates SVM models for every combination of kernel, C, and gamma using k-fold cross-validation, then picks the combination with the best mean validation score. This prevents manual guessing and gives a statistically reliable model selection. The `best_params_` attribute reveals the winning hyperparameter combination.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    'kernel': ['linear', 'rbf', 'poly'],
    'C': [0.1, 1, 10],
    'gamma': ['scale', 'auto']
}
grid = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)
print(grid.best_params_)

The degree d in K(x,z) = (γ·x·z + c)^d controls the order of the polynomial decision boundary. A degree of 2 produces quadratic boundaries (ellipses, parabolas), degree 3 produces cubic curves, and higher degrees produce increasingly complex shapes. Greater degree gives the model more capacity to fit complex data, but also raises the risk of overfitting, especially with small training sets.

SVC uses the One-vs-One (OvO) strategy by default: for C classes it trains C(C−1)/2 binary SVMs, one per pair of classes, then takes a majority vote to predict the final label. An alternative is One-vs-Rest (OvR), which trains only C classifiers (each class against all others) and is available via LinearSVC. OvO is preferred for kernel SVC because each binary problem is smaller and faster to solve.

A string kernel measures similarity between text strings or biological sequences by counting shared substrings, subsequences, or n-grams, without converting them to fixed-length numeric vectors. Using the kernel trick, SVM can work directly with variable-length strings. Applications include spam detection, document classification, and bioinformatics tasks such as protein function prediction and DNA sequence comparison.

Very high gamma makes the RBF kernel extremely narrow — each training point influences only an infinitesimally small neighborhood around itself. The decision boundary wraps tightly around individual training examples, effectively memorizing the training set. The result is very high training accuracy but poor performance on unseen data, a textbook case of overfitting.

When the SVM primal problem is converted to its Lagrangian dual using the KKT conditions, the optimization objective and the prediction rule depend only on dot products xᵢ·xⱼ between pairs of training examples — never on individual feature vectors. Replacing each dot product with a kernel function K(xᵢ, xⱼ) = φ(xᵢ)·φ(xⱼ) implicitly performs the mapping φ without computing it. This is the rigorous mathematical reason why kernels work.

Training kernel SVM requires computing and storing the n×n Gram matrix (O(n²) space) and solving a quadratic programming problem that takes O(n²) to O(n³) time. For large datasets (n > 100,000), this becomes prohibitively slow. In contrast, linear SVM can be solved in near-linear O(n) time using specialized solvers like LIBLINEAR, making it far more scalable.

A degree-2 polynomial kernel produces smooth quadratic boundaries — shapes like ellipses and parabolas — suited for moderate nonlinearity. Degree-5 produces highly complex, wiggly boundaries that can fit intricate training patterns. Higher degree increases model capacity but makes it far more prone to overfitting, particularly when the training set is small relative to the model's complexity.

The code creates a fine mesh grid over the 2D feature space, predicts the class label at every grid point using the trained model, then uses contourf to color the regions and scatter to overlay the actual data points. This visualization clearly shows the shape of the learned decision boundary relative to training data. It only works for problems with exactly 2 features.

import numpy as np, matplotlib.pyplot as plt
from sklearn.svm import SVC

model = SVC(kernel='rbf', C=1, gamma='scale').fit(X, y)
x_min, x_max = X[:,0].min()-1, X[:,0].max()+1
y_min, y_max = X[:,1].min()-1, X[:,1].max()+1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 300),
                     np.linspace(y_min, y_max, 300))
Z = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.3)
plt.scatter(X[:,0], X[:,1], c=y, edgecolors='k')
plt.show()

You can pass a Python callable directly as the `kernel` argument to SVC — sklearn will call it as `kernel(X, Y)` to compute the Gram matrix between any two sets of examples. Alternatively, set `kernel='precomputed'` and supply the n×n Gram matrix directly as the input data. Either way, the resulting kernel matrix must be positive semi-definite for the optimization to converge to the correct solution.

from sklearn.svm import SVC
import numpy as np

def my_kernel(X, Y):
    # Squared distances between all pairs
    diff = X[:, np.newaxis, :] - Y[np.newaxis, :, :]
    sq_dist = np.sum(diff**2, axis=-1)
    return np.exp(-0.5 * sq_dist)

model = SVC(kernel=my_kernel)
model.fit(X_train, y_train)

Images have highly nonlinear class boundaries in pixel space that linear classifiers cannot capture. RBF-SVM maps handcrafted feature descriptors — such as HOG (Histogram of Oriented Gradients) or SIFT — into a space where these boundaries become linear hyperplanes. This approach achieved state-of-the-art results on benchmarks like CIFAR-10 until convolutional neural networks took over around 2012.

Gamma is the inverse width of the Gaussian kernel. High gamma → narrow kernel → each training point influences only its immediate neighborhood → complex, jagged boundary → high variance (overfitting). Low gamma → wide kernel → each training point influences a large region → simple, smooth boundary → high bias (underfitting). Selecting the right gamma via cross-validation is how you navigate this fundamental trade-off.

Compute the Gram matrix K where K[i,j] = f(xᵢ, xⱼ) for a representative sample of your data, then check whether all eigenvalues are non-negative using np.linalg.eigvalsh(K). If any eigenvalue is significantly negative, the function violates Mercer's condition and is not a valid kernel. Small negative values near zero are usually acceptable as floating-point rounding errors.

import numpy as np

def is_valid_kernel(K_func, X_sample):
    n = len(X_sample)
    K = np.array([[K_func(X_sample[i], X_sample[j])
                   for j in range(n)] for i in range(n)])
    eigenvalues = np.linalg.eigvalsh(K)
    return np.all(eigenvalues >= -1e-10)  # allow tiny floating-point errors

Mercer's theorem is closed under addition and multiplication: if K₁ and K₂ are both valid (positive semi-definite) kernels, then K₁ + K₂ and K₁ · K₂ are also valid kernels. Positive scalar multiples c·K (c > 0) are likewise valid. This property lets you design hybrid kernels — for instance, combining a linear kernel (global linear trends) with an RBF kernel (local nonlinear patterns) to capture richer structure.