Support vector machines

An SVM tries to find a decision boundary (hyperplane) that separates classes while maximizing the margin, the distance between the boundary and the nearest data points of each class. By maximizing this margin, the SVM aims for the most robust separation, which tends to generalize well to new, unseen data.

Support vectors are the training data points that lie closest to the decision boundary, exactly on or within the margin. They are the critical points that determine the position and orientation of the hyperplane; removing other points would not change the boundary, but removing support vectors would.

The margin is the distance between the separating hyperplane and the closest data points (support vectors) from each class. A larger margin generally means a more confident and generalizable classifier, which is why SVMs are called maximum-margin classifiers.

A hyperplane is the decision boundary that separates classes; it is a flat subspace one dimension lower than the feature space. In 2D it is a line, in 3D it is a plane, and in higher dimensions it generalizes to a hyperplane defined by the equation w·x + b = 0.

It represents the separating hyperplane, where w is the weight (normal) vector perpendicular to the boundary and b is the bias term that shifts it. A new point x is classified by the sign of w·x + b: positive for one class and negative for the other.

A hard margin SVM requires that all points be perfectly separated with no violations, which only works for linearly separable, noise-free data. A soft margin SVM allows some misclassifications or margin violations using slack variables, making it practical for real-world, noisy, or overlapping data.

C controls the trade-off between maximizing the margin and minimizing classification errors on the training data. A large C penalizes misclassifications heavily (narrow margin, risk of overfitting), while a small C allows more violations for a wider margin (risk of underfitting).

Increasing C makes the model penalize misclassifications more strongly, so it tries to classify training points correctly, resulting in a narrower (harder) margin. This can fit the training data better but increases the risk of overfitting to noise.

The kernel trick lets an SVM operate in a high-dimensional feature space without explicitly computing the coordinates there, by using a kernel function to compute inner products directly. This makes it possible to find non-linear boundaries efficiently, since data that is not linearly separable in the original space may become separable in the higher-dimensional space.

A non-linear kernel (like RBF or polynomial) is used when the data cannot be separated by a straight line or flat hyperplane in its original space. The kernel implicitly maps data into a higher-dimensional space where a linear separator can capture the complex, curved decision boundary in the original space.

The RBF kernel measures similarity between two points based on their distance, giving high values for close points and low values for distant ones. It is a popular default because it can model very flexible, complex non-linear boundaries and works well in many situations without prior knowledge of the data's structure.

A large gamma means each training point has a very localized influence, so the decision boundary becomes wiggly and tightly fits the training data. This increases the risk of overfitting, while a small gamma gives smoother boundaries with broader influence per point.

It computes w·x + b for the new point x, and assigns the class based on the sign of the result. If w·x + b is positive the point is on one side of the hyperplane (one class), and if negative it is on the other side (the other class).

SVM is primarily a supervised learning algorithm because it requires labeled training data to learn the boundary between classes. It is most commonly used for classification, though variants exist for regression (SVR) and outlier detection.

SVMs rely on distances and inner products between points, so features with larger numeric ranges can dominate those with smaller ranges. Scaling features to comparable ranges (e.g., mean 0, standard deviation 1) ensures each feature contributes fairly and helps the optimizer converge properly.

The SVC class (Support Vector Classifier) from sklearn.svm builds an SVM classifier. The snippet creates an RBF-kernel SVM with C=1.0, fits it to training data, and predicts labels for test data.

from sklearn.svm import SVC
model = SVC(kernel='rbf', C=1.0)
model.fit(X_train, y_train)
preds = model.predict(X_test)

A slack variable measures how much a particular data point violates the margin or is misclassified. Points on the correct side beyond the margin have zero slack, while points inside the margin or on the wrong side have positive slack proportional to the violation.

Yes, the regression variant is called Support Vector Regression (SVR). Instead of finding a boundary that separates classes, SVR fits a function such that most data points fall within a margin (an epsilon-tube) around the predicted line, ignoring errors smaller than epsilon.

SVMs are inherently binary, so multi-class problems are handled by combining multiple binary SVMs, commonly using one-vs-rest (one classifier per class) or one-vs-one (one classifier per pair of classes) strategies. The final prediction is decided by voting or by the highest decision score among these classifiers.

SVMs are effective in high-dimensional spaces, even when the number of features exceeds the number of samples, because the decision boundary depends only on the support vectors rather than all data. They are also memory-efficient since they store only these support vectors.

SVMs can be slow to train on very large datasets because the training time scales poorly (roughly quadratically to cubically) with the number of samples. They also do not directly provide probability estimates and can be sensitive to the choice of kernel and hyperparameters.

A dataset is linearly separable if there exists a straight line (in 2D) or flat hyperplane (in higher dimensions) that can perfectly separate the classes with no misclassifications. In such cases a hard-margin linear SVM can find a perfect separating boundary.

Many boundaries can separate linearly separable data, but the maximum-margin boundary leaves the most room between classes, making it least sensitive to small perturbations in the data. This typically leads to better generalization on unseen data, which is the core motivation behind SVMs.

Maximizing the margin is equivalent to minimizing the squared norm of the weight vector, ½‖w‖², subject to the constraint that all points are correctly classified (with a margin of at least 1). A smaller ‖w‖ corresponds to a larger geometric margin of 2/‖w‖.

The polynomial kernel allows the SVM to fit curved (polynomial) decision boundaries by implicitly creating feature combinations up to a chosen degree. A degree-1 polynomial kernel behaves like a linear kernel, while higher degrees capture more complex interactions but risk overfitting.

A valid kernel must correspond to a legitimate inner product in some (possibly higher-dimensional) feature space, a condition known as Mercer's condition. Intuitively, the kernel measures similarity between two inputs, and this similarity must come from a real dot product so the optimization remains well-posed.

The model with the wider margin is generally preferred because a larger margin tends to generalize better to new data and is more robust to noise. This reflects the SVM principle that, among separating boundaries, the maximum-margin one is the most reliable choice.

The magnitude of the decision function value indicates how far a point is from the decision boundary, which reflects the model's confidence. Points far from the boundary (large absolute value) are classified more confidently, while points near the boundary (value close to zero) are uncertain.

Good values for C and gamma are usually found through hyperparameter tuning, such as grid search or randomized search combined with cross-validation. You try different combinations and select the pair that gives the best validation performance, balancing bias and variance.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
params = {'C': [0.1, 1, 10], 'gamma': [0.01, 0.1, 1]}
grid = GridSearchCV(SVC(kernel='rbf'), params, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)

Nothing changes; the decision boundary stays the same. Only support vectors (the points on or within the margin) determine the hyperplane, so removing a point that is not a support vector has no effect on the trained model.

Text data is typically high-dimensional and sparse (many features such as words, most being zero per document), and SVMs handle such high-dimensional sparse data effectively. Their reliance on support vectors and margin maximization helps them generalize well even when features greatly outnumber samples.

The term C·Σξᵢ is the total penalty for margin violations and misclassifications, where ξᵢ are the slack variables and C scales how heavily these violations are punished. Minimizing it alongside ½‖w‖² balances having a wide margin against fitting the training data correctly.

The standard SVM uses the hinge loss, which is zero when a point is correctly classified beyond the margin and increases linearly as a point falls inside the margin or onto the wrong side. This loss encourages not just correct classification but classification with a safe margin of confidence.

Overfitting in an RBF SVM is often caused by gamma and/or C being too large, producing an overly complex boundary. Reducing gamma (smoother boundary) and/or reducing C (wider margin, more tolerance for errors) typically reduces overfitting and improves generalization.