Model Interpretability

Model interpretability is the degree to which humans can understand the internal decision-making of a machine learning model. It matters because: (1) Trust — clinicians won't use a model they don't understand. (2) Debugging — interpretability reveals why models fail on certain inputs. (3) Fairness — identify and fix biased decision patterns. (4) Regulation — GDPR's 'right to explanation' and AI regulations require interpretable decisions. (5) Scientific discovery — understand what features drive predictions in biology, climate, etc. Interpretable ≠ accurate, but both are needed for responsible deployment in high-stakes applications.

Global interpretability: understand the model's overall behavior — what features are generally important, what decision logic it has learned. Example: feature importance for a random forest showing 'age' is most important overall. Local interpretability: understand a specific prediction — why did this model predict malignant for this patient? LIME and SHAP can provide both. Local explanations are more actionable for individual decisions; global explains the model's general behavior. Many high-stakes decisions require both: understand the model globally AND explain each prediction to the affected person.

LIME (Ribeiro et al. 2016) explains any classifier's prediction on a specific input by fitting a simple interpretable model (linear regression) locally. Process: (1) Perturb the input by creating many variations (e.g., mask random words or superpixels). (2) Get black-box model predictions for all perturbations. (3) Fit a weighted linear model where perturbations closer to the original get higher weight. (4) The linear model's coefficients explain the original prediction. Pros: model-agnostic, intuitive. Cons: perturbation instability (different runs give different explanations), local linearity assumption may not hold.

import lime
from lime.lime_text import LimeTextExplainer
explainer = LimeTextExplainer(class_names=['neg','pos'])
exp = explainer.explain_instance(text, model.predict_proba, num_features=5)

SHAP uses Shapley values from cooperative game theory to fairly attribute a model's prediction to each feature. Shapley value for feature i: φ_i = Σ_{S⊆F\{i}} [|S|!(|F|-|S|-1)!/|F|!] · [f(S∪{i}) − f(S)] — the average marginal contribution of feature i across all possible subsets S. SHAP is consistent (if a feature matters more, its SHAP value is higher) and locally accurate (sum of SHAP values = prediction − baseline). TreeSHAP runs in polynomial time for trees; DeepSHAP for neural nets. SHAP is the gold standard for feature attribution and supports both local and global explanations.

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

Gradient-based saliency maps compute ∂ŷ/∂x_i — the gradient of the output with respect to each input feature/pixel. Features with high |∂ŷ/∂x_i| are most influential at that input. For images, the saliency map highlights which pixels caused the prediction. Simple to compute (one backward pass) but often noisy (gradient can be large where output is sensitive even if pixel isn't semantically important). SmoothGrad adds Gaussian noise to inputs, averages gradients over many noisy copies for cleaner maps. Guided backprop also masks negative gradients for sharper visualizations.

x.requires_grad_(True)
out = model(x)
out[0, target_class].backward()
saliency = x.grad.abs().squeeze()  # pixel importance

GradCAM (Selvaraju et al. 2017) produces class-discriminative localization maps for CNNs. It computes the gradient of class score y^c w.r.t. the last convolutional feature map A^k, then global-average-pools to get weights: α_k^c = (1/Z) Σ_{i,j} ∂y^c/∂A^k_{ij}. The heatmap is: L^c = ReLU(Σ_k α_k^c A^k). ReLU keeps only features that positively contribute to class c. The heatmap is upsampled to input size. GradCAM highlights which image regions the CNN used for classification — invaluable for debugging (e.g., detecting if the model looks at the lesion vs. the ruler in medical images).

from pytorch_grad_cam import GradCAM
from pytorch_grad_cam.utils.image import show_cam_on_image
cam = GradCAM(model=model, target_layers=[model.layer4[-1]])
grayscale_cam = cam(input_tensor=input_tensor, targets=targets)

Integrated gradients (Sundararajan et al. 2017) attributes the prediction to each input feature by integrating gradients along a straight path from a baseline x' (e.g., black image, zero vector) to the actual input x: IG_i(x) = (x_i − x'_i) · ∫_0^1 ∂F(x'+α(x−x'))/∂x_i dα. Satisfies completeness: Σ_i IG_i(x) = F(x) − F(x'). This ensures attributions sum to the actual prediction difference. Unlike vanilla gradients, IG captures the cumulative effect of increasing each feature from baseline to actual value, avoiding saturation issues (where gradient ≈ 0 even for important features at saturation).

from captum.attr import IntegratedGradients
ig = IntegratedGradients(model)
attributions = ig.attribute(input, baseline=torch.zeros_like(input), target=target_class)

Tree-based models (RF, XGBoost) compute feature importance as: (1) Impurity-based (Gini importance): total reduction in criterion from splits on feature i, weighted by samples. Fast but biased toward high-cardinality features. (2) Permutation importance: shuffle feature i in validation set, measure accuracy drop. More reliable — measures actual impact on predictions. (3) SHAP values: TreeSHAP computes exact Shapley values in polynomial time. Impurity importance is fast but misleading; permutation importance is more reliable; SHAP values are theoretically grounded and support local explanations.

from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_val, y_val, n_repeats=10)
importances = result.importances_mean

A PDP shows the marginal effect of one (or two) features on predicted outcome, averaging over all other features: PDP_j(x_j) = E_{X_{-j}}[F(x_j, X_{-j})]. For each value of x_j, it computes the average prediction when all samples have x_j set to that value. PDPs reveal: monotonic relationships (older age → higher salary?), nonlinearities, and interactions (2D PDP). Limitation: assumes feature independence — if features are correlated, marginalizing over X_{-j} creates unrealistic combinations. Individual Conditional Expectation (ICE) plots show per-instance curves instead of averages, revealing heterogeneous effects that PDPs hide.

from sklearn.inspection import PartialDependenceDisplay
PartialDependenceDisplay.from_estimator(model, X, features=[0, 1, (0,1)])

Counterfactual explanations answer: 'What would need to change for the prediction to flip?' For example: 'Your loan was denied. If your income were $5k higher, it would be approved.' Formally, find x' close to x with different prediction: argmin_{x'} ||x−x'|| s.t. f(x') = target. Methods: DiCE (Diverse Counterfactual Explanations, generates multiple diverse counterfactuals), WACHTER (gradient-based), and model-specific approaches. Key properties: actionability (change must be realistic), sparsity (few feature changes), proximity (minimal change). Counterfactuals are preferred by non-technical users as they provide recourse.

import dice_ml
data_dice = dice_ml.Data(dataframe=df, continuous_features=['age','income'], outcome_name='loan')
explainer = dice_ml.Dice(data_dice, model_dice)

A SHAP dependence plot shows SHAP value for feature j vs. its actual value, colored by a second feature to reveal interactions. x-axis: feature value; y-axis: SHAP value (contribution to prediction); color: another feature. Patterns reveal: positive/negative impact, threshold effects, and interaction effects. Example: 'SHAP for age increases up to 40, then plateaus' or 'age's effect depends on education level (color)'. Automatically selects the most interacting feature for coloring. Dependence plots are more informative than linear feature importance because they show direction and nonlinearity, and interactions, all simultaneously.

shap.dependence_plot('age', shap_values, X,
    interaction_index='income')  # auto-select interaction

Decision trees are inherently interpretable: each prediction follows a human-readable rule path from root to leaf. Example: if age>45 AND glucose>140 → diabetic. Visualizable with sklearn's plot_tree. Shallow trees (depth 3–5) are interpretable; deep trees are not. Advantage: no feature scaling needed, handles missing data, captures nonlinear relationships. Disadvantage: high variance (small data change → different tree), poor extrapolation, can overfit. In practice, trees are often used as components in ensemble methods (RF, boosting) that sacrifice direct interpretability for accuracy. Rule extraction from ensembles (rule lists, node harvesting) attempts to recover interpretability.

from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt
clf = DecisionTreeClassifier(max_depth=4)
clf.fit(X_train, y_train)
plot_tree(clf, feature_names=feature_names, filled=True)

TCAV (Testing with Concept Activation Vectors, Kim et al. 2018) measures how much human-defined concepts influence a model's predictions. Process: (1) Collect images of a concept (e.g., 'stripes'). (2) Train a linear classifier to separate concept images from random images in the model's activation space → the decision boundary is the CAV. (3) Compute CAV score: sensitivity of model output to the concept direction (directional derivative). High TCAV score = the concept positively influences predictions of class c. TCAV explains predictions in terms of human-interpretable concepts ('this was classified as zebra because of stripes, not color').

ANCHORS (Ribeiro et al. 2018) find sufficient if-then rules that locally explain predictions with high precision: 'IF income > 50k AND credit_score > 700 THEN model predicts approved (precision=0.95)'. The anchor 'anchors' the prediction — changing non-anchor features doesn't change the prediction. Rules are found with a multi-armed bandit beam search. Unlike LIME (which gives coefficients), anchors give crisp rules understandable without statistics knowledge. High precision means the rule is reliable within its coverage region. Anchors are easier for non-technical stakeholders to act on than LIME coefficients or SHAP values.

LRP (Bach et al. 2015) attributes a neural network's prediction back through layers to inputs using conservation rules: relevance R_i^l = Σ_j R_{ij}^{l→l+1}, ensuring total relevance is conserved from output to input. The propagation rule for dense layers: R_i^l = Σ_j [a_i w_{ij} / (Σ_k a_k w_{kj} + ε)] R_j^{l+1}. LRP produces pixel-wise heatmaps showing which input pixels contributed positively or negatively to the class prediction. LRP satisfies conservation (no attribution is lost), unlike vanilla gradients. Several LRP rules exist (ε-rule, β-rule, α-β-rule) with different stability/sparsity trade-offs.

Attention weights in Transformers are often used as explanations: high α_{ij} means token i attends strongly to token j. Visualizing attention heads shows linguistic patterns (dependency arcs, coreference). However, Jain & Wallace (2019) showed attention weights are not faithful explanations — they can be replaced with adversarially modified weights while preserving predictions. Wiegreffe & Pinter countered that faithfulness depends on definition. The consensus: attention is a useful exploration tool and correlates with gradients in many cases, but should not be treated as ground truth attribution. Use in combination with gradient-based methods for more reliable insight.

from transformers import pipeline
pipe = pipeline('text-classification', model='bert-base-uncased', return_all_scores=True)

Probing (diagnostic classifiers) tests what linguistic information is encoded in a model's representations. Train a simple linear or shallow classifier on top of frozen pre-trained embeddings to predict a linguistic property (POS tags, dependency depth, syntax tree depth, named entities). High probing accuracy = the property is linearly decodable from the representation. Results show BERT encodes: POS in early layers, syntax in middle, semantics in later layers. Probing reveals what the model 'knows' but not necessarily what it 'uses' for predictions — a model may encode POS without using it for downstream tasks.

from sklearn.linear_model import LogisticRegression
probe = LogisticRegression(max_iter=1000)
probe.fit(bert_embeddings_train, pos_tags_train)
print('Probing accuracy:', probe.score(bert_embeddings_val, pos_tags_val))

Intrinsic interpretability: models that are interpretable by design — linear models (coefficients = feature importance), decision trees (rule paths), GAMs (additive components visualized as curves). No additional tool needed. Post-hoc interpretability: interpret black-box models after training using external tools (LIME, SHAP, GradCAM, TCAV). Post-hoc enables using powerful models (XGBoost, neural nets) while still providing explanations. Trade-off: intrinsically interpretable models may be less accurate; post-hoc explanations may not faithfully represent the model's actual reasoning. Best practice: use intrinsic when performance allows; post-hoc when accuracy is paramount.

A SHAP summary plot shows feature importance + direction for the full dataset. Each row is a feature; each dot is one sample. x-axis = SHAP value (positive = pushes prediction up). Color = feature value (red=high, blue=low). Dot distribution shows spread of impact. Reading: 'high income (red) has high positive SHAP → high income strongly increases predicted salary'. This combines feature importance (average |SHAP|), direction, and distribution in one plot. It's the most informative single visualization for model behavior and is now standard in data science explainability reports.

shap.summary_plot(shap_values, X_test, feature_names=feature_names)
# Each row=feature, dot=sample, x=SHAP value, color=feature value

Mechanistic interpretability (MI) aims to reverse-engineer neural network computations into human-understandable algorithms. Unlike SHAP (what features matter), MI asks: what circuit (subgraph of weights and neurons) implements a specific behavior? Examples: induction heads in transformers implement pattern completion; 'curve detectors' in CNNs detect curved edges; modular arithmetic neurons perform clock arithmetic. Tools: activation patching (causal intervention on activations), probing, logit lens (reading predictions from intermediate layers). Key researchers: Anthropic's interpretability team, Chris Olah. MI is foundational for AI safety — understanding how models work is essential for ensuring they're safe.

Activation patching (Meng et al. 2022, ROME) is a causal intervention technique. Procedure: (1) Run model on a clean input, cache all activations. (2) Run model on a corrupted input (e.g., scramble subject name). (3) Patch each activation from the clean run into the corrupted run, one at a time. (4) Measure restoration of correct output. Positions where patching fully restores output are causally responsible for the behavior. This technique identified that factual knowledge in GPT is stored in specific MLP layers (mid-layers) corresponding to the subject token — enabling targeted model editing (ROME, MEMIT).

Logit lens (nostalgebraist 2020) applies the model's final unembedding matrix to intermediate residual stream activations, projecting them to vocabulary space at each layer. This shows what 'prediction' each layer is making before the final output. Early layers often show syntactic predictions; later layers converge to the final semantic prediction. For factual recall ('Paris is the capital of ___'), mid-layers already show 'France'. The logit lens reveals the progressive refinement of predictions through layers. Extended: tuned lens fine-tunes a small affine transform per layer for more accurate intermediate prediction extraction.

Fairness in ML ensures model predictions don't discriminate against protected groups (race, gender, age). Interpretability tools help detect unfairness: (1) Feature importance — if 'zip code' has high importance, it may proxy race. (2) SHAP — compare SHAP values for different demographic groups to find disparate impacts. (3) PDPs — plot model output vs. sensitive attribute to detect systematic bias. (4) Counterfactuals — 'if only the applicant's gender changed, would the prediction flip?' (demographic parity). Interpretability makes fairness auditing possible and provides evidence for regulatory compliance. Fairness and interpretability are deeply intertwined.

# Check if gender affects SHAP for predictions
import shap
shap.group_difference_plot(shap_values, X['gender']==1, X['gender']==0, X.columns)

SHAP explains predictions in terms of input features (pixel intensities, tabular columns) — granular but often not interpretable by domain experts. TCAV explains in terms of human-defined concepts (texture, shape, disease marker presence) — higher-level and more understandable to clinicians. TCAV requires concept datasets (positive examples) and works only for classification tasks in differentiable models. SHAP requires only predictions (works for any model via KernelSHAP). Concept-based explanations (TCAV, Concept Bottleneck Models) are preferred in medical imaging where 'high SHAP value at pixel (100,200)' is less useful than 'the tumor's spiculation increased the prediction'.

Concept Bottleneck Models (Koh et al. 2020) introduce an explicit intermediate layer of human-defined concepts (binary predictions) between input and final output. Model: Input → Concept predictor c(x) → Label predictor g(c). Example: tumor classifier with bottleneck concepts ('spiculation', 'calcification', 'size'). At test time: see which concepts drove the prediction; intervene by correcting wrong concept predictions ('the model wrongly predicted no calcification; force it to 1'). CBMs are interpretable by design and support test-time intervention. Drawback: requires concept labels and may sacrifice accuracy vs end-to-end models, which can use non-concept shortcuts.

Faithfulness measures whether an explanation accurately reflects the model's actual computation. A faithful explanation correctly identifies features the model truly used. Unfaithful: attention weights that can be swapped without changing predictions; gradient signs that don't match actual feature importance. Methods to evaluate faithfulness: (1) Deletion/insertion tests — remove highest-attribution features; if prediction drops, attribution was faithful. (2) Sufficiency — do top-k attributed features alone give the same prediction? (3) Roar (RemOve And Retrain) — retrain without top-k features; faithfulness requires large performance drop. Faithfulness ≠ plausibility (human agreement) — a faithful explanation may not align with human intuition.

SHAP interaction values decompose each SHAP value φ_i into main effects Φ_{ii} and pairwise interaction effects Φ_{ij} (i≠j): φ_i = Σ_j Φ_{ij}. Interaction values satisfy: Σ_j Φ_{ij} = φ_i and Φ_{ij} = Φ_{ji}. They reveal how two features jointly affect predictions beyond their individual effects. Example: 'age and income interact — high age + high income gives much higher salary prediction than the sum of their individual effects'. TreeSHAP computes exact interaction values efficiently. Interaction heatmap visualizes all pairwise interactions, identifying the strongest joint feature effects.

shap_interaction = explainer.shap_interaction_values(X_test)
# shap_interaction[sample, feature_i, feature_j] = interaction effect

Model debugging uses interpretability to find and fix model errors. Common patterns: (1) Shortcut learning — SHAP shows background pixels have high importance in medical images (model uses scanner artifact not tumor). Fix: remove confounders. (2) Label leakage — highly important feature correlated with target through a leaky path. (3) Distribution shift — important features shift between train and deployment. Tools: SHAP waterfall plots per error case, GradCAM on misclassified images, confusion matrix + LIME on hard examples, data maps (correctness + confidence plots). Systematic debugging: cluster errors by SHAP, find common patterns in failure modes.

A surrogate model is a simple, interpretable model trained to approximate the predictions of a complex black-box model f(x). Global surrogate: train a decision tree or linear model on {x, f(x)} pairs. The surrogate's coefficients/rules approximate f's behavior globally. LIME uses local surrogates (linear model in a neighborhood). Fidelity of the surrogate (how well it approximates f) must be verified. A surrogate with low fidelity misleads about the black-box's actual behavior. Surrogates are useful for regulatory reporting (required to be interpretable) but may hide critical nuances of the black-box decision boundary.

from sklearn.tree import DecisionTreeClassifier
# Global surrogate
black_box_preds = black_box.predict(X)
surrogate = DecisionTreeClassifier(max_depth=5)
surrogate.fit(X, black_box_preds)  # approximate black-box

Spurious correlations occur when a model learns to predict from confounders that correlate with the label in training but not in deployment. Famous example: NLP models using 'not' keyword to predict negation, or 'because' to predict entailment — pattern matching, not reasoning. In medical AI: pneumonia models using hospital equipment in X-rays as shortcuts. SHAP/GradCAM can reveal: 'the model predicts pneumonia based on presence of metal clip (hospital confounder), not lung texture'. Causal analysis and distribution shift testing confirm spurious learning. Fix: data augmentation, adversarial training, causal models, or removing confounders from images.

AOPC (Area Over the Perturbation Curve) measures faithfulness of feature attribution methods. Procedure: (1) Order features by attributed importance (highest first). (2) Iteratively remove (mask/replace) features in order. (3) Measure prediction drop at each step. (4) AOPC = area under the degradation curve. High AOPC = removing important features quickly degrades prediction → attribution is faithful. Used to compare LIME vs SHAP vs GradCAM. A good explanation should identify features that when removed immediately degrade the prediction, not features irrelevant to the model. AOPC is one of several faithfulness benchmarks (insertion test is the complementary metric).

More interpretable models (linear regression, decision stumps, GAMs) are often less accurate on complex tasks; more accurate models (deep nets, GBTs) are often less interpretable. This trade-off exists because: (1) Complex models can learn high-order interactions and nonlinearities; interpretable models often can't. (2) Forcing interpretable structure constrains model capacity. However, this trade-off is not always large: (a) In low-dimensional, near-linear problems, interpretable models match black boxes. (b) Post-hoc methods provide explanations for black boxes, approximating interpretability. (c) Inherently interpretable models (ebm, sparse networks) can approach black-box accuracy on tabular data.

EBMs (Nori et al. 2019, Microsoft) are interpretable GAMs (Generalized Additive Models): ŷ = f_1(x_1) + f_2(x_2) + ... + f_{ij}(x_i, x_j). Each term is a 1D or 2D shape function learned by boosting. Shape functions f_i are visualized as curves, revealing nonlinear effects and interactions. EBMs match or beat gradient boosted trees on tabular data, disproving that interpretability requires accuracy sacrifice. Each feature's contribution is explicit and visualizable. Pairwise interactions f_{ij} are automatically detected. EBMs are the state-of-the-art interpretable model for tabular data and recommended when interpretability is required without sacrificing accuracy.

from interpret.glassbox import ExplainableBoostingClassifier
ebm = ExplainableBoostingClassifier()
ebm.fit(X_train, y_train)
ebm_global = ebm.explain_global()
show(ebm_global)  # interactive visualization

When a model makes multiple interdependent predictions (e.g., object detection with class + box + mask), attribution is complex: which input pixels caused the class prediction? The box prediction? SHAP for multi-output models attributes independently per output. For SHAP and GradCAM in multi-class settings, each class has its own attribution — GradCAM^c shows which pixels matter for class c specifically. The problem worsens with long reasoning chains: in LLMs doing arithmetic, the 'explanation' must attribute across many token steps, not just input tokens. Mechanistic interpretability traces circuits through all intermediate computations, but is expensive and task-specific.

GDPR (Article 22) gives EU citizens the right not to be subject to solely automated decisions with legal/significant effects, and requires 'meaningful information about the logic involved'. The EU AI Act (2024) classifies high-risk AI (credit, hiring, medical, law enforcement) as requiring transparency, documentation, and human oversight. This drives demand for: post-hoc explanations per prediction (LIME, SHAP), model cards documenting performance across groups, fairness audits, and human review processes. Technical explainability (SHAP values) must be translated into user-understandable explanations for compliance. This regulatory pressure is one of the main industrial drivers of interpretability research.

Activation maximization finds the input x* that maximally activates a neuron or class: x* = argmax_x f_i(x). Starting from random noise or a natural image, gradient ascent maximizes the target activation. For early CNN neurons: generates simple patterns (edges, colors). For later neurons: generates textures, object parts. For softmax class neurons: generates class-prototypical images (often bizarre). Regularization is needed to keep x* natural: total variation, L2 on pixel values, or Gaussian blur at each step. Activation maximization visualizes 'what concept each neuron detects' — Olah et al.'s circuits work uses this extensively to understand feature detectors.

x = torch.randn(1,3,224,224, requires_grad=True)
opt = torch.optim.Adam([x], lr=0.01)
for _ in range(200):
    loss = -model(x)[0, target_neuron]
    opt.zero_grad(); loss.backward(); opt.step()

SmoothGrad (Smilkov et al. 2017) averages gradients over n noisy copies of the input: SmoothGrad(x) = (1/n) Σ_k ∇_x F(x + δ_k), δ_k ~ N(0,σ²I). Averaging reduces gradient noise that makes vanilla saliency maps look patchy. SIG combines SmoothGrad + Integrated Gradients: average IG over multiple noise realizations. The noise level σ is a hyperparameter (typically 10–20% of input range). Too little noise = still noisy; too much = attribution diffuses. SmoothGrad produces visually cleaner saliency maps and correlates better with human annotations of important image regions.

def smooth_grad(model, x, target, n=50, sigma=0.15):
    grads = []
    for _ in range(n):
        noise = torch.randn_like(x) * sigma
        x_noisy = (x + noise).requires_grad_(True)
        model(x_noisy)[0, target].backward()
        grads.append(x_noisy.grad.clone())
    return torch.stack(grads).mean(0)

Model cards (Mitchell et al. 2019, Google) are short documents accompanying ML models, documenting: intended use, training data summary, evaluation results across demographic groups, limitations, and ethical considerations. Datasheets for datasets document: motivation, composition, collection process, recommended uses, biases. These tools increase transparency for deployers, regulators, and affected users. HuggingFace model hub requires model cards. Model cards are not technical explanations (like SHAP) but are essential for institutional accountability and audit trails. They translate ML technical details into policy-relevant information and are now increasingly required by law (EU AI Act).

A SHAP waterfall plot explains a single prediction by showing how each feature's SHAP value pushes the model output from the baseline (expected prediction E[f(x)]) to the actual prediction f(x). Features are sorted by magnitude; positive SHAP values (red arrows) push prediction up; negative (blue) push down. The sum of all SHAP values + baseline = final prediction. It's the most informative single-instance explanation: 'this loan was predicted as high-risk because income=-0.3, debt=+0.5, age=-0.1, starting from baseline 0.35 → final 0.62'. More informative than a bar chart because it shows direction and magnitude together.

shap.plots.waterfall(shap_values[0])  # explain one sample

Feature ablation (or ablation study) removes or masks one feature at a time and measures the change in model performance. Features with high performance drop are important; those with negligible impact are redundant. For images, 'occlusion sensitivity' slides a patch across the image and measures prediction change — regions where occlusion drops confidence most are the most important. Feature ablation is model-agnostic and easy to implement, but requires many forward passes (O(n_features) for tabular, O(image_size) for images). It measures marginal importance — unlike SHAP which accounts for feature interactions.

def ablate_feature(model, X, feature_idx, baseline=0):
    X_ablated = X.copy()
    X_ablated[:, feature_idx] = baseline
    return model.predict(X) - model.predict(X_ablated)

Faithfulness: does the explanation accurately reflect the model's actual computation? A faithful explanation correctly identifies features the model truly used. Plausibility: does the explanation align with human intuition and domain knowledge? An explanation can be plausible (humans agree) but unfaithful (model used different features). Or faithful but implausible (model used spurious features humans find irrelevant). The ideal explanation is both. Jain & Wallace (2019) showed attention weights can be unfaithful; DeYoung et al. (ERASER benchmark) systematically evaluates faithfulness. Measuring faithfulness requires perturbation tests (deletion/insertion) not just human agreement.

Human evaluation assesses whether explanations help humans make better decisions, understand model behavior, or detect errors. Methods: (1) User study — show humans explanations + predictions, measure decision accuracy or trust calibration. (2) Simulation — can humans predict model behavior from the explanation? (3) Forward prediction — do explanations help users anticipate correct vs. incorrect model predictions? (4) Counterfactual task — can users identify what change would flip the prediction? A high-quality explanation should make human-AI collaboration more effective, not just look visually appealing or satisfy technical faithfulness metrics.

Rationalization selects a subset of input text (rationale) that is sufficient to reproduce the model's prediction, presented as the explanation. Methods: (1) Extractive rationale — select contiguous or non-contiguous spans from input (LIME can do this via superpixel-style chunking of text). (2) Abstractive rationale — generate a free-text explanation using a language model. The ERASER benchmark evaluates whether rationales are comprehensive (contain all needed information) and sufficient (model can reproduce prediction from rationale alone). Rationales are more human-readable than feature weights but can be generated to be plausible without being faithful.

Transparency means the model's workings are directly understandable before prediction: decision trees, linear models, rule lists, GAMs. No additional tool needed — the model IS the explanation. Post-hoc explainability generates explanations after a black-box prediction using external tools (LIME, SHAP, GradCAM). Post-hoc explanations are approximations that may not perfectly reflect model behavior (faithfulness issue). For high-stakes applications (medical diagnosis, criminal sentencing, loan decisions), transparency may be legally required — post-hoc explanations for black-boxes may not satisfy 'meaningful information about the logic'. The FDA and EU AI Act lean toward inherently transparent models for certain use cases.

A SHAP force plot (Lundberg & Lee) visualizes contributions of features to a single prediction as horizontal arrows stacked on a baseline. Red arrows push the prediction rightward (increase output); blue push leftward (decrease). The length of each arrow equals the feature's absolute SHAP value. The final prediction is where all arrows land. Multiple force plots stacked vertically (rotated 90°) create a 'SHAP heatmap' for a dataset — showing patterns across samples. Interactive force plots (using shap.plots.force or the JS visualization) allow hovering over each feature to see its exact value and contribution.

shap.plots.force(shap_values[0])  # interactive force plot for one sample

Concept Bottleneck Models (Koh et al. 2020) provide explanations in terms of human-interpretable concepts (not raw features). They introduce a bottleneck layer of concept predictions c(x) before the final classifier. Preferred when: domain experts have predefined concepts (radiology has 'mass', 'calcification', 'spiculation'); regulatory requirements demand human-understandable explanations; test-time intervention is needed (correct a wrong concept prediction to fix the output). CBMs may sacrifice accuracy (concept bottleneck constrains the model) but provide richer, more actionable explanations than feature-level SHAP for domain experts who think in concepts.

Influence functions (Koh & Liang 2017) estimate how much removing or upweighting a single training example x_i would change the model's prediction on a test example x_t. Using the implicit function theorem: Influence(x_i, x_t) ≈ −(1/n) ∇_θ L(x_t)^T H^{-1} ∇_θ L(x_i), where H is the Hessian of the training loss. High influence = that training example strongly shaped this prediction. Applications: (1) Find mislabeled training examples (high influence on wrong predictions). (2) Data poisoning detection. (3) Understand dataset artifacts. Challenge: computing H^{-1} is expensive (iterative methods or stochastic approximations are needed for large models).

Completeness is the key axiom of integrated gradients: the sum of all attribution scores exactly equals the difference between prediction at input x and prediction at baseline x': Σ_i IG_i(x) = F(x) − F(x'). This ensures no attribution is lost or created — the total 'credit' distributed equals the total prediction change from baseline to input. This is satisfied by Shapley values (SHAP) and integrated gradients, but NOT by vanilla gradients (which don't sum to prediction) or gradient × input (which has a different guarantee). Completeness is fundamental for fair credit attribution across features.

# Verify completeness
ig_sum = attributions.sum()
pred_diff = model(input).squeeze() - model(baseline).squeeze()
assert abs(ig_sum - pred_diff) < 1e-4, 'Completeness violated'

Concept drift occurs when the statistical relationship between features and target changes over time (e.g., COVID changed what 'fever' meant for flu prediction). Interpretability detects drift by monitoring: (1) SHAP value distributions — if feature importance rankings shift, the model may be using different features. (2) Feature importance change alerts. (3) Prediction confidence distribution shift (MMD test on prediction distributions). (4) Error analysis on recent data vs old data. If SHAP shows 'income' becomes less important and 'location' becomes more important over time, the underlying population or economic behavior has changed and the model needs retraining.

XAI encompasses all methods to make ML models understandable. Current landscape: feature attribution (SHAP, IG, LIME), concept-based (TCAV, CBM), example-based (influence functions, prototypes), rule extraction (anchors, decision trees), mechanistic interpretability. Key open challenges: (1) Evaluation — no ground-truth for 'correct' explanations; faithfulness vs plausibility metrics disagree. (2) Scalability — SHAP exact O(2^n), mechanistic interp for LLMs is expensive. (3) Robustness — small input perturbations can drastically change SHAP values (adversarial explanations). (4) Actionability — technically faithful explanations aren't always actionable for end users. (5) Causal vs. associative — most explanations show correlation not causation.