Model interpretability (SHAP, LIME)

SHAP values are based on Shapley values from cooperative game theory, which fairly distribute a 'payout' (the model's prediction) among 'players' (features). This grounding guarantees a unique attribution satisfying local accuracy (efficiency), missingness, and consistency (monotonicity) — properties that purely ad-hoc methods like LIME do not provably satisfy.

Local accuracy requires that the sum of all feature SHAP values plus the base value (expected model output) exactly equals the model's prediction for that instance: f(x) = φ0 + Σφi. This makes SHAP an additive feature attribution method where attributions reconstruct the prediction.

LIME perturbs the instance to create a neighborhood of synthetic samples, weights them by proximity to the original instance, gets the black-box model's predictions on them, and fits an interpretable surrogate model (typically sparse linear regression) on this weighted local dataset. The surrogate's coefficients become the local explanation.

Exact Shapley values require evaluating the model over all 2^M feature coalitions, which is exponential in the number of features M. KernelSHAP approximates them by sampling coalitions and solving a weighted linear regression with the Shapley kernel as sample weights, recovering Shapley values as the regression coefficients without exhaustive enumeration.

TreeSHAP exploits the structure of decision trees to compute exact SHAP values in polynomial time — O(TLD^2) where T is trees, L is leaves, D is depth — rather than approximating them. This makes it dramatically faster and exact for models like XGBoost, LightGBM, and random forests.

A dependence plot shows how a single feature's SHAP value varies across its value range for every instance, revealing nonlinear effects and, via color, interaction effects with a second feature. A bar chart only shows mean absolute importance, collapsing all this directional and interaction structure into one number.

Interventional SHAP breaks feature dependencies by sampling 'missing' features from their marginal distribution (do-operator semantics), attributing only to features the model actually uses. Conditional SHAP samples from the conditional distribution given present features, spreading credit to correlated-but-unused features. The choice changes attributions substantially when features are correlated and affects whether explanations are 'true to the model' or 'true to the data'.

LIME's perturbation sampling is stochastic, so different random samples produce different local datasets and thus different surrogate coefficients, especially with too few samples or in highly nonlinear regions. This instability is a well-known LIME limitation; fixing the random seed, increasing num_samples, or tuning the kernel width mitigates it.

Both weight perturbed samples by similarity to the instance before fitting a linear surrogate. LIME uses a heuristic exponential kernel on a user-chosen distance with a tunable kernel width, while KernelSHAP uses the specific Shapley kernel weighting (favoring coalitions with very few or very many features) that makes the regression coefficients equal Shapley values. The Shapley kernel is what gives KernelSHAP its theoretical guarantees that LIME lacks.

The base value φ0 is the model's expected output over the background/reference dataset — typically the mean prediction E[f(x)]. It represents the prediction you would make with no feature information, and all SHAP values are deviations from this baseline that sum up to the actual prediction.

SHAP attributions are defined relative to a reference distribution used to simulate 'absent' features; the background dataset determines the base value and the counterfactual against which contributions are measured. A biased, tiny, or all-zeros background can produce misleading attributions, so the background should represent the population (or a meaningful reference like 'typical' instances) for the question being asked.

SHAP interaction values decompose each feature's attribution into a main effect plus pairwise interaction effects, forming a matrix per instance where off-diagonal entries quantify how much two features' joint presence changes the prediction beyond their individual effects. TreeSHAP can compute these exactly, enabling detection of feature interactions the model learned.

LIME for tabular data by default discretizes continuous features into bins and perturbs by sampling bin membership; poor binning (e.g., default quartiles on skewed features) can create perturbations far from realistic data and a surrogate that explains an unrealistic neighborhood. Disabling discretization or using domain-appropriate binning often produces more faithful explanations.

Consistency states that if a model changes so that a feature's marginal contribution increases (or stays the same) across all coalitions, that feature's SHAP value must not decrease. This is important because earlier tree feature-importance metrics like gain could violate it — assigning lower importance to a feature that actually matters more — making cross-model comparisons unreliable, whereas SHAP guarantees consistent attributions.

DeepSHAP (DeepLIFT-based) and GradientSHAP are used because they leverage gradients/backprop to approximate SHAP values far more efficiently than KernelSHAP, which would be prohibitively slow for high-dimensional image inputs. They propagate contribution scores through the network relative to a baseline, giving per-pixel attributions.

A force plot visualizes how each feature pushes the prediction away from the base value toward the final output, with red features increasing and blue features decreasing the prediction, sized by their SHAP magnitude. It provides an intuitive additive breakdown of one instance's prediction.

SHAP measures a feature's contribution to the model's predictions, not its causal effect on the real-world outcome; a feature may have high mean(|SHAP|) because it is correlated with a true cause or with the target via spurious patterns the model exploited. Without causal assumptions or interventional design, SHAP importance reflects model behavior, not data-generating causality.

Use shap.TreeExplainer on the trained model, call it on the data to get an Explanation object, then use shap.plots.beeswarm or shap.summary_plot for a global view. TreeExplainer is exact and fast for tree ensembles.

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)
shap.plots.beeswarm(shap_values)   # global summary
shap.plots.waterfall(shap_values[0])  # single prediction

LIME is inherently local — it explains one prediction via a local surrogate and has no principled aggregation to a global view. SHAP is local-first but its additive, consistent values can be soundly aggregated (e.g., mean |SHAP|, beeswarm) to yield global feature importance, bridging local and global interpretability.

Missingness states that a feature that is absent (has no value/impact) in the instance receives a SHAP value of zero. It implies attributions are only assigned to features actually present and constrains the explanation so that truly missing inputs cannot be credited with importance.

For multi-class models SHAP produces a separate set of attributions per class (a matrix of shape samples × features × classes), each explaining the contribution to that class's output (typically the margin/log-odds before softmax). You interpret feature effects on a per-class basis, since a feature can push toward one class while pushing away from another.

LIME assumes the black box is approximately linear in the local neighborhood; if the decision surface is highly curved there, a sparse linear surrogate has low local fidelity even though it is interpretable. Choosing a richer surrogate or shrinking the neighborhood improves fidelity but can reduce interpretability or stability, so the kernel width must be tuned for the locality assumption to hold.

Both treat the presence/absence of tokens (words or sub-words) as binary features: LIME perturbs by randomly removing words and fitting a linear surrogate over token presence, while SHAP (e.g., Partition or Kernel explainers with a text masker) attributes the prediction across tokens. The output highlights which words drove the classification.

They optimize different objectives: SHAP enforces game-theoretic axioms (efficiency, consistency) over all coalitions, while LIME fits a locally-weighted sparse linear model that only approximates behavior in a heuristic neighborhood. Differences in sampling distribution, weighting kernel, locality, and surrogate sparsity cause divergent attributions, especially with feature correlations or strong nonlinearity.

PartitionExplainer computes Owen values by recursively partitioning features into a hierarchical clustering and allocating credit through the hierarchy, which respects feature groupings/correlations and runs much faster than full KernelSHAP. It is the default for many text and image maskers in the SHAP library because it handles structured/correlated inputs efficiently.

Each point is one instance's SHAP value for a feature: horizontal position shows the magnitude and sign of impact on the prediction, color shows the feature's value (high vs low), and the vertical spread/density shows the distribution of effects. Together they reveal not just importance ranking but the direction and value-dependence of each feature's effect.

KernelSHAP requires many model evaluations per explained instance — roughly proportional to the number of sampled coalitions times the background size — so explaining a large dataset means an enormous number of forward passes. This makes it slow for big models or many instances, motivating model-specific explainers (TreeSHAP, DeepSHAP) or smaller background summaries (e.g., k-means).

By inspecting global SHAP importance and dependence plots, you may find a feature with implausibly high attribution (e.g., an ID, a post-outcome timestamp, or a proxy for the label) driving predictions, signaling leakage. SHAP reveals that the model relies on a feature it should not, prompting investigation and removal before deployment.

Because both rely on perturbing inputs and querying the model on out-of-distribution synthetic samples, an adversary can build a 'scaffolded' model that behaves normally on real data but detects the OOD perturbation queries and returns innocuous outputs, hiding biased behavior from the explainer (Slack et al., 'Fooling LIME and SHAP'). This shows post-hoc perturbation-based explanations can be fooled and should not be the sole audit mechanism.

For a linear model f(x)=β0+Σβi·xi with independent features, the SHAP value of feature i is βi·(xi − E[xi]) — the coefficient times the feature's deviation from its expected value. Thus SHAP generalizes the intuitive 'coefficient × centered feature' contribution while remaining exact for linear models (via LinearExplainer).

A waterfall plot lays out the additive contributions as a vertical, sorted cascade from the base value to the final prediction, making it easy to read the exact ordered magnitude and sign of each feature's push. It is clearer than the compact horizontal force plot when there are many features or you need precise per-feature values.

Because SHAP values are additive and satisfy local accuracy, the SHAP values of a sum/average of models equal the (appropriately weighted) sum of their individual SHAP values, so explanations compose linearly across ensemble members. This lets you attribute an ensemble's prediction consistently and even decompose contributions per base learner.

Summarize the background with a small representative set — e.g., shap.kmeans to cluster the data into k weighted centroids, or sample a few hundred rows — and pass that as the reference, drastically cutting model evaluations while approximating the expectation. This trades a small amount of accuracy for major speedups.

import shap
background = shap.kmeans(X_train, 50)  # 50 weighted centroids
explainer = shap.KernelExplainer(model.predict, background)
shap_values = explainer.shap_values(X_test[:100])

The kernel width controls how locally the surrogate is fit: too small and only a few perturbations get weight, yielding an unstable, overfit explanation; too large and the surrogate becomes effectively global, losing local fidelity and washing out the instance-specific signal. There is no universally optimal value, so it must be tuned (or stability checked) for the feature space, which is a known weakness of LIME.