Feature crosses and interactions

A feature cross is a synthetic feature created by combining two or more raw features—typically by multiplication for numerics or by taking the Cartesian product for categoricals—so that the model can represent the joint effect that the features have together rather than only their independent additive contributions. Crosses let inherently linear models capture nonlinear interaction patterns.

A linear model is additive in its inputs and cannot represent interaction effects (the effect of one feature depending on another) unless those interactions are supplied as engineered cross features. Neural networks can in principle approximate interactions through their nonlinear hidden layers, so manual crosses are less essential—though they can still improve sample efficiency.

The Cartesian-product cross can produce up to 1,000 x 500 = 500,000 distinct categories. The practical concern is sparsity and dimensionality explosion: most combinations are rare or unseen, leading to overfitting and huge, mostly-zero one-hot vectors, which is why hashing or factorization is typically used.

The hashing trick maps the potentially enormous space of crossed categories into a fixed-size vector via a hash function, bounding memory and avoiding the need to maintain an explicit vocabulary of all combinations. Its main downside is hash collisions, where distinct crosses share a bucket, introducing noise; the collision rate is controlled by the number of buckets.

With two features the model is y = b0 + b1*x1 + b2*x2 + b3*(x1*x2). The coefficient b3 on the product term x1*x2 is the interaction coefficient: it measures how the effect (slope) of x1 on y changes per unit of x2 (and symmetrically). If b3 = 0 the features act purely additively.

y = b0 + b1*x1 + b2*x2 + b3*(x1*x2)

No. Tree ensembles capture interactions implicitly: a split on feature A followed by a split on feature B within the same path makes the prediction for A conditional on B, which is exactly an interaction. The interaction order a single tree can capture is bounded by its depth, so depth-1 stumps (additive) cannot model interactions while deeper trees can.

Friedman's H-statistic quantifies the strength of an interaction by measuring the fraction of variance in the partial dependence that is not explained by the sum of the individual (lower-order) partial dependence functions. It ranges from 0 (no interaction; the joint effect is purely additive) to 1 (the effect is driven entirely by the interaction); note that in finite samples the estimate can occasionally exceed 1 due to estimation noise.

FM learns a low-dimensional latent vector v_i for each feature and models each pairwise interaction weight as the dot product <v_i, v_j> instead of an independent free parameter. This factorization shares information across interactions, so a weight can be estimated even for feature pairs never co-observed in training, making FM robust to the extreme sparsity of crossed/one-hot data and reducing parameters from O(n^2) to O(n*k).

interaction = sum_{i<j} <v_i, v_j> * x_i * x_j

FFM gives each feature multiple latent vectors—one per 'field' (the categorical group a feature belongs to). When feature i interacts with feature j, FM uses i's vector for j's field, allowing the latent representation of a feature to differ depending on which field it is interacting with. This increases expressiveness for CTR-style data at the cost of more parameters (n*f*k).

DeepFM combines an FM component and a deep MLP component that share the same feature embeddings. The FM part explicitly models low-order (pairwise) interactions, while the deep part learns high-order, nonlinear interactions; outputs are summed before the final sigmoid. Sharing embeddings means no manual feature engineering and end-to-end training without a separately engineered wide input.

Each DCN cross layer computes x_{l+1} = x_0 * (x_l^T w_l) + b_l + x_l, where x_0 is the original input. By repeatedly multiplying by the original input, stacking L cross layers produces explicit bounded-degree feature crosses up to order L+1 in an efficient, parameter-light way, complementing a parallel deep network.

x_{l+1} = x0 * (x_l^T w_l) + b_l + x_l

DCN-v2 replaces the rank-1 weight vector w_l with a full weight matrix W_l (often low-rank factorized for efficiency), giving x_{l+1} = x_0 ⊙ (W_l x_l + b_l) + x_l. The matrix form is far more expressive—it can model richer interactions per layer—while low-rank factorization and an optional mixture-of-experts keep cost manageable.

x_{l+1} = x0 ⊙ (W_l x_l + b_l) + x_l

The wide (linear) component uses explicit crossed features to memorize specific, frequently observed feature combinations (exception rules), while the deep component uses embeddings to generalize to unseen or rare combinations via dense representations. Jointly training both balances memorization and generalization.

AutoInt projects each feature into an embedding and applies multi-head self-attention so that each feature attends to others; the attention weights determine which feature combinations interact, and stacking attention layers builds higher-order interactions. This yields interpretable, automatically learned crosses without manual feature engineering.

SHAP interaction values extend Shapley values to pairs, splitting each feature's attribution into a main (individual) effect plus pairwise interaction effects with every other feature. The interaction term for (i,j) is symmetric and, summed across j plus the main effect, recovers feature i's total SHAP value, giving a model-agnostic, locally additive breakdown of interactions.

A 2-way interaction is the joint effect of two features (e.g., x1*x2); a 3-way interaction is the joint effect of three (x1*x2*x3) that remains after accounting for all lower-order terms. Higher-order interactions are used cautiously because they multiply dimensionality and sparsity, are hard to estimate reliably, and often overfit while being difficult to interpret.

Multiplying two raw continuous values produces a single smooth term that still assumes a monotone/linear interaction, which is often too rigid. Bucketizing each feature first and taking the Cartesian product of the buckets lets a linear model assign an independent weight to each region of the joint space, capturing arbitrary nonlinear interaction shapes (e.g., learning that a specific lat-lon grid cell is special).

lat_x_lon = cross(bucketize(lat), bucketize(lon))

Centering the features before forming x1*x2 reduces the multicollinearity between the product term and its main effects, making the lower-order coefficients more stable and interpretable (they then represent effects at the mean of the other variable). Centering does not change the interaction coefficient itself or the model's overall fit, but improves numerical conditioning and interpretation.

x1c = x1 - x1.mean(); x2c = x2 - x2.mean(); inter = x1c*x2c

Under a purely additive model, the change in the outcome from increasing x1 is the same regardless of x2's value—effects simply sum. Under a multiplicative/interaction effect, the marginal effect of x1 depends on the level of x2 (the slopes differ across x2), which is precisely what an interaction term models.

In matrix-factorization-style models, the predicted affinity between, say, a user and an item is the dot product of their embedding vectors u·v = sum_k u_k v_k. Each term u_k*v_k is a multiplicative interaction between latent factors, so the dot product is effectively a sum of learned crosses between the two entities' representations.

score = sum_k u[k]*v[k]   # u·v

ReLU MLPs approximate multiplication only through compositions of piecewise-linear units, which requires many neurons and data to learn even a simple product accurately; the multiplicative form is not represented natively. Architectures like Cross Networks, FM, and DeepFM bake in explicit multiplicative terms, so they capture such interactions with far fewer parameters and less data.

L1 (Lasso) regularization is commonly used because it drives many coefficients exactly to zero, effectively selecting the small subset of crosses that matter and discarding the vast number of irrelevant, sparse combinations. This combats overfitting and reduces model size in high-dimensional crossed spaces; L1+L2 (elastic net) is also used for stability.

Crossing features multiplies the number of dimensions (e.g., enumerating all two-way crosses among n features adds up to O(n^2) columns), exponentially enlarging the feature space relative to the fixed amount of data, so most cross combinations have few or zero examples. Mitigations include hashing, factorized/embedding-based interactions (FM), bucketization, and strong L1 regularization.

A 2D PDP whose surface cannot be written as a sum of two one-dimensional functions (it twists or has ridges that depend jointly on both axes) indicates an interaction between the two features. If the features had purely additive effects, the 2D PDP would be additively separable, so a slice along one feature would be a vertically shifted copy of the slice at any other value of the second feature.

An interaction exists when the effect of one factor on the response depends on the level of another factor—equivalently, the joint effect is not equal to the sum of the individual main effects. In an ANOVA/regression framework, a nonzero coefficient on the cross-product (or factor-combination) term, beyond the main effects, signals an interaction.

You must supply hash_bucket_size, because crossed_column applies the hashing trick to map the (possibly enormous) Cartesian product of the input categorical columns into a bounded number of buckets. Choosing it trades memory for collision rate; too small causes many collisions, too large wastes memory and increases sparsity.

tf.feature_column.crossed_column(['city','dow'], hash_bucket_size=10000)

In a synergistic interaction the two features together contribute more to the prediction than the sum of their individual effects (the combination amplifies the outcome), whereas in an antagonistic/redundant interaction the joint effect is less than the sum—often because the features carry overlapping information and partially cancel. SHAP interaction values can reveal the sign and magnitude of each.

Interaction constraints specify groups of features that are allowed to appear together along a single tree path; features in different groups cannot interact, since a path may only split on features within one permitted group. They are useful for injecting domain knowledge, improving interpretability, and reducing overfitting by forbidding spurious interactions.

params['interaction_constraints'] = '[[0,1],[2,3,4]]'

With interaction_only=True the transformer omits pure powers (no a^2, b^2, c^2) and keeps only cross-product interaction terms plus the bias and originals: [1, a, b, c, a*b, a*c, b*c]. Setting interaction_only=False would additionally include a^2, b^2, and c^2.

PolynomialFeatures(degree=2, interaction_only=True, include_bias=True)

Target encoding a cross replaces each category of the crossed feature with a statistic of the target (e.g., the mean label) for rows in that combination, compressing a high-cardinality cross into a single dense numeric column. The primary risk is target leakage/overfitting, especially for rare combinations; it is mitigated with out-of-fold encoding and smoothing toward the global mean.

Simpson's paradox arises when an aggregate trend reverses within subgroups. A feature cross (e.g., treatment x group) lets the model fit a separate effect within each subgroup rather than a single pooled slope, so it can recover the correct conditional relationships that a purely additive model would average away and misread.

Separately, the model can only learn that some segments and some categories are generally more clickable (additive log-odds). The cross lets it learn segment-specific preferences (e.g., segment A loves category X but ignores category Y), which is exactly the conditional, non-additive behavior that drives clicks—greatly improving ranking quality where memorization of strong combinations matters.

Compare nested models with and without the interaction using out-of-sample/cross-validated metrics (and, for linear models, a likelihood-ratio test or the significance of the interaction coefficient). Relying on training-set fit alone is unsound because adding interaction terms almost always improves training fit while risking overfitting; held-out improvement is the decisive evidence.

Empirical and theoretical analyses (e.g., the 'Neural Collaborative Filtering vs. Matrix Factorization Revisited' work) showed that a well-tuned dot product, which directly encodes the multiplicative interaction, often matches or beats an MLP that must learn the interaction from scratch—and is cheaper to serve. The lesson is that an explicit interaction inductive bias can outperform a generic deep approximator.