Decision boundaries

A decision boundary is the surface (a line, curve, or higher-dimensional surface) in feature space that separates regions assigned to different predicted classes. On one side of the boundary the model predicts one class, and on the other side it predicts another. It is where the model is exactly undecided between classes.

In a 2D feature space, a linear classifier's decision boundary is a straight line. More generally, for a linear model it is a hyperplane: a line in 2D, a plane in 3D, and a flat hyperplane in higher dimensions. It separates the space into two half-planes.

By default the decision boundary occurs where the predicted probability equals 0.5. This is where the linear combination w·x + b equals zero, since the sigmoid of zero is 0.5. Points are classified as positive if the probability exceeds 0.5 and negative otherwise.

# boundary where sigmoid(w.x+b)=0.5 -> w.x+b=0

Two classes are linearly separable if there exists a straight line (or hyperplane in higher dimensions) that perfectly separates all points of one class from all points of the other. In that case a linear classifier can achieve zero training error. If no such line exists, the data is not linearly separable.

A single perceptron can only produce a linear decision boundary, but the XOR pattern is not linearly separable: no single straight line can separate the two XOR classes. Solving XOR requires a nonlinear boundary, which needs either multiple layers (a hidden layer) or a feature transformation. This limitation famously highlighted the need for multilayer networks.

kNN forms its decision boundary implicitly by assigning each point to the majority class among its k nearest training neighbors. This produces a piecewise, often jagged, nonlinear boundary that adapts to the local distribution of data. Smaller k gives a more complex, wiggly boundary, while larger k gives a smoother one.

Increasing k makes the decision boundary smoother and less sensitive to individual noisy points, because each prediction averages over more neighbors. This reduces variance but can increase bias, potentially missing fine structure. Very large k can oversmooth and underfit the data.

Decision trees produce axis-aligned, piecewise-rectangular decision boundaries. Each split tests one feature against a threshold, partitioning the space with cuts perpendicular to a feature axis. The combined result is a set of rectangular regions, each assigned a class.

The margin is the distance between the decision boundary (separating hyperplane) and the nearest training points of either class. A hard-margin SVM chooses the boundary that maximizes this margin, giving the widest possible gap between classes. Maximizing the margin tends to improve generalization.

Support vectors are the training points that lie closest to the decision boundary, on or within the margin. They are the critical points that determine the position and orientation of the boundary; removing other points does not change it. The boundary is defined entirely by these support vectors.

The kernel trick implicitly maps data into a higher-dimensional feature space where it may become linearly separable, computing inner products via a kernel function without ever forming the high-dimensional coordinates explicitly. A linear boundary in that higher space corresponds to a nonlinear boundary in the original input space. Common kernels include polynomial and RBF (Gaussian).

A very large gamma makes each training point's influence very local, so the decision boundary becomes highly wiggly and tightly wraps around individual points. This typically causes overfitting, fitting noise in the training data. Smaller gamma gives smoother, more general boundaries.

By transforming the inputs into polynomial features, a model that is linear in the new features becomes nonlinear in the original inputs. For example, a linear boundary in the space of (x1, x2, x1^2, x2^2) can correspond to an ellipse or other curve in the original (x1, x2) plane. This is feature engineering to gain nonlinear decision boundaries while keeping a linear learner.

X_poly = np.column_stack([x1, x2, x1**2, x2**2, x1*x2])

No, a model linear in x1 and x2 alone can only produce straight-line boundaries, so it cannot directly represent a circle. However, by adding the features x1^2 and x2^2, a linear model in the expanded space can represent the circle as x1^2 + x2^2 - r^2 = 0. This is why feature transformation enables curved boundaries.

The weight vector w is perpendicular (normal) to the decision boundary defined by w·x + b = 0. It points in the direction of increasing predicted score, i.e., toward the positive class. The bias b shifts the boundary's position relative to the origin.

Lowering the threshold makes the model classify more inputs as the positive class, effectively shifting the boundary so a larger region is labeled positive. This increases recall (catching more positives) but typically lowers precision by allowing more false positives. The threshold is a tunable trade-off independent of the learned weights.

Flexible models with high capacity (many parameters or low regularization) can bend their boundaries to fit fine details of the training data, producing complex, wiggly shapes. The risk is overfitting: the boundary captures noise rather than the true pattern, so it performs poorly on new data. Regularization or simpler models trade some flexibility for better generalization.

L2 regularization penalizes large weights, shrinking them toward zero, which produces a smoother, less extreme decision boundary that is less likely to overfit. Stronger regularization yields a simpler, more conservative boundary, while too much can underfit. It controls the bias-variance trade-off.

With one-vs-rest, each class gets its own linear score, and a point is assigned to the class with the highest score. This partitions the feature space into regions separated by piecewise-linear boundaries where two class scores are equal. The result is several connected line segments forming the boundaries between the three regions.

Hidden units with nonlinear activations (like ReLU or sigmoid) combine many simple linear pieces, and stacking layers lets the network compose these into highly flexible, nonlinear boundaries. With enough hidden units, a network can approximate almost any boundary shape. This is why deep networks can separate intricately shaped classes that linear models cannot.

In distance-based models, the feature with the larger range dominates the distance computation, so the effective decision boundary becomes stretched and biased along that feature's axis. This can distort which neighbors are 'nearest' and degrade accuracy. Feature scaling (standardization or normalization) prevents this imbalance.

A soft margin allows some training points to fall inside the margin or even be misclassified, controlled by a penalty parameter C. This is useful when data is noisy or not perfectly separable, letting the boundary ignore a few outliers for better overall generalization. A hard margin, by contrast, demands perfect separation and is sensitive to outliers.

A smaller C means a weaker penalty for misclassification, so the SVM prioritizes a wider margin and tolerates more errors. This produces a smoother, more general boundary that may underfit. A larger C forces fewer errors, giving a narrower margin and a more complex boundary that may overfit.

Many boundaries can perfectly separate the same training points, so different algorithms or different inductive biases may settle on distinct shapes (for example a tight wiggly boundary versus a wide-margin straight line). These boundaries can generalize very differently on unseen data even with identical training accuracy. This is why choosing the right model and considering margins or smoothness matters.

The bias term b shifts the decision boundary away from the origin, allowing it to not pass through the point where all features are zero. Without a bias, the boundary would be forced through the origin, limiting where it can sit. Adjusting b translates the boundary parallel to itself.

With heavy class imbalance, many algorithms push the boundary toward the minority class, since predicting the majority class minimizes overall error. This can cause the minority class to be under-predicted or ignored. Techniques like resampling, class weights, or threshold adjustment help reposition the boundary more fairly.

The signed distance from a point to the boundary is (w·x + b) divided by the norm of w, ||w||. Its magnitude measures how far the point is from the boundary, and its sign indicates which side (which class) the point lies on. Larger magnitude means the model is more confident about that point.

dist = (w @ x + b) / np.linalg.norm(w)

When Gaussian Naive Bayes assumes the same variance for a feature across classes, the quadratic terms in the log-likelihood ratio cancel, leaving a linear function of the features. This makes the resulting decision boundary a straight line or hyperplane. If variances differ between classes, the boundary becomes quadratic (curved).

A single tree gives sharp, blocky axis-aligned boundaries that can overfit, while a random forest averages many trees to produce a smoother, more robust boundary. The averaging reduces variance and softens the jagged edges of individual trees. The result generalizes better while still capturing nonlinear structure.

A common technique is to create a fine mesh grid over the feature space with numpy's meshgrid, predict the class for every grid point, and then plot the predictions as colored regions using matplotlib's contourf. Overlaying the training points on top shows how the boundary separates the classes. This works for any classifier with a predict method on two features.

xx, yy = np.meshgrid(np.linspace(x0,x1,200), np.linspace(y0,y1,200))
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.3)

A linear boundary is a straight line/hyperplane, while a quadratic boundary is curved, such as an ellipse, parabola, or hyperbola. Quadratic Discriminant Analysis (QDA) naturally produces quadratic boundaries because it allows each class its own covariance matrix. Linear Discriminant Analysis (LDA), which assumes shared covariance, gives linear boundaries.

One strategy is to transform the features (for example with polynomial or kernel mappings) so the classes become linearly separable in the new space. Another is to use an inherently nonlinear model such as a decision tree, kNN, or neural network that can fit curved boundaries directly. Both approaches let the model capture the nonlinear structure.

Points near the boundary have predicted class scores close to the threshold (e.g., probabilities near 0.5), meaning the model is nearly undecided about their class. Small changes can flip their prediction, so they carry the most uncertainty. Active learning exploits this by querying labels for boundary-near points, since they are most informative for refining the boundary.

The distance metric changes which points count as 'nearest', so it reshapes the decision boundary. Euclidean distance tends to produce smoother, more circular neighborhoods, while Manhattan distance gives boundaries with a more axis-aligned, diamond-shaped character. Choosing the metric to match the data's geometry can improve classification.