Bias, fairness and explainability

Disparate treatment is intentional discrimination where a protected attribute is used directly (or via a deliberate proxy) in the decision, whereas disparate impact is a facially neutral policy that produces disproportionately adverse outcomes for a protected group regardless of intent. Removing the protected attribute (fairness through unawareness) can eliminate disparate treatment but not disparate impact, because correlated proxies still encode the attribute.

Demographic parity requires the positive prediction rate to be equal across groups: P(Ŷ=1 | A=a) = P(Ŷ=1 | A=b). Its main limitation is that it ignores the true label, so it can force equal acceptance even when the base rates legitimately differ, harming accuracy and potentially requiring less-qualified members of one group to be selected.

Equalized odds requires both the true positive rate and the false positive rate to be equal across groups (Ŷ ⫫ A | Y). Equal opportunity is the relaxation that requires only equality of true positive rates (i.e., equal TPR for the Y=1 subgroup), so it constrains errors only among those who actually qualify.

It states that, except in trivial cases (equal base rates across groups or a perfect classifier), you cannot simultaneously satisfy calibration (predictive parity), equalized false positive rates, and equalized false negative rates. When group base rates differ, these criteria are mathematically incompatible, so practitioners must choose which fairness notion to prioritize.

The four-fifths rule (US EEOC guideline) flags adverse impact when the selection rate for a protected group is less than 80% of the rate for the most-selected group. It operationalizes disparate impact via the ratio of demographic-parity (selection) rates and serves as a practical screening threshold rather than a proof of discrimination.

Removing the protected attribute does not remove its influence because other features act as proxies (e.g., zip code for race, name for gender), allowing the model to reconstruct and use the attribute implicitly. It only prevents direct, explicit use; it cannot prevent disparate impact arising from redundant encodings in correlated features.

A predictor is counterfactually fair if its prediction for an individual would be unchanged in a counterfactual world where that individual belonged to a different protected group, holding the non-descendants of the attribute fixed. It is defined on a causal graph: the decision must not depend on the protected attribute or any variable causally descended from it.

Pre-processing modifies the training data (e.g., reweighing, resampling, learning fair representations) before model fitting. In-processing changes the learning algorithm itself (e.g., fairness constraints or adversarial debiasing in the objective). Post-processing adjusts a trained model's outputs (e.g., group-specific thresholds, reject-option classification) without retraining; it is useful when the model is fixed but requires access to the protected attribute at decision time.

It trains the predictor jointly with an adversary that tries to recover the protected attribute from the predictor's output (or representation). The predictor minimizes its task loss while maximizing the adversary's error, so at convergence the prediction carries little information about the protected attribute, approximating demographic parity or equalized odds depending on the adversary's inputs.

Reweighing assigns instance weights so that, under the weighted distribution, the protected attribute and the class label become statistically independent. Weights are higher for (group, label) combinations that are under-represented relative to the product of their marginals, compensating for sampling/label imbalance without altering feature values.

w = (P(A=a)*P(Y=y)) / P(A=a, Y=y)  # weight for each (group a, label y) cell

Yes. The Shapley value is the unique attribution that simultaneously satisfies efficiency (attributions sum to the prediction minus baseline), symmetry, dummy (zero contribution gets zero), and additivity/linearity. SHAP builds on this game-theoretic uniqueness, which is what distinguishes it from heuristic importance measures.

Computing Shapley values requires evaluating the model on feature coalitions, which means imputing the 'absent' features. With correlated features, the common interventional/marginal approach evaluates off-manifold (unrealistic) inputs, while the conditional approach can spread credit to correlated-but-unused features. The choice changes attributions, and neither perfectly separates causation from correlation.

LIME perturbs the instance to create a neighborhood of synthetic samples, weights them by proximity to the instance, queries the black-box model for their predictions, and fits an interpretable surrogate (usually sparse linear) on this weighted local data. Its core assumption is that the decision boundary is approximately linear within that local neighborhood, so the explanation is only locally faithful.

Local explanations attribute a single prediction to feature contributions for one specific instance (e.g., a SHAP force plot or LIME). Global explanations describe the model's overall behavior across the dataset (e.g., mean |SHAP| feature importance, partial dependence plots, surrogate trees). Aggregating consistent local explanations is one route to global insight.

PDPs marginalize one feature by averaging predictions while substituting all observed values of it, which creates unrealistic (off-manifold) combinations when features are correlated, biasing the curve. Accumulated Local Effects (ALE) plots fix this by averaging local changes in prediction within small conditional intervals, avoiding extrapolation into low-density regions.

Integrated Gradients integrates the model's gradients along a straight-line path from a baseline input to the actual input. It satisfies completeness (attributions sum to the output difference between input and baseline) plus the sensitivity and implementation-invariance axioms. Plain gradients violate sensitivity because they can saturate to zero, while some attribution methods (e.g., DeepLIFT, LRP) can violate implementation invariance.

Grad-CAM produces a coarse class-discriminative localization heatmap for CNNs by weighting the feature maps of a (typically last) convolutional layer by the global-average-pooled gradients of the target class score with respect to those maps, then ReLU-ing the result. It highlights image regions most responsible for a given class prediction without requiring architectural changes.

Not reliably. Research (Jain & Wallace; Wiegreffe & Pinter) shows attention weights are often not faithful: alternative attention distributions can yield the same prediction, and attention correlates weakly with gradient-based importance. Attention can be plausible to humans but should not be treated as a guaranteed causal explanation of model behavior.

Faithfulness measures how accurately an explanation reflects the model's actual reasoning/computation (e.g., do removed 'important' features actually change the output). Plausibility measures how convincing or agreeable the explanation is to humans. An explanation can be highly plausible yet unfaithful, which is dangerous because it builds misplaced trust.

A counterfactual explanation states the smallest change to an instance's features that would flip the model's decision (e.g., 'income +$5k would change the loan denial to approval'). Good counterfactuals are actionable (change mutable features), proximal (minimal change), realistic/on-manifold, and ideally diverse, offering recourse without exposing the full model.

Anchors (Ribeiro et al.) are high-precision IF-THEN rules identifying a sufficient subset of feature conditions that 'anchor' the prediction: as long as those conditions hold, the model returns the same prediction with high probability regardless of other features. They offer clearer coverage/precision guarantees than LIME's local linear fits.

Impurity-based importance is biased toward high-cardinality and continuous features because they offer more split points and can reduce impurity by chance, even when uninformative. It is also computed on training data and can reward overfitting. Permutation importance on held-out data is a more reliable, model-agnostic alternative.

Permuting one feature at a time breaks correlations and creates unrealistic inputs, so importance can be split or distorted among correlated features and evaluated off-manifold. Mitigations include grouping/permuting correlated features together, using conditional permutation importance, or clustering features before measuring importance.

Sampling (representation) bias arises when the training data does not represent the deployment population (some groups under-sampled). Measurement/label bias arises when the recorded features or labels are flawed proxies of the true target (e.g., arrests as a proxy for crime). Aggregation bias arises when one model is fit across distinct subgroups that should be modeled separately, masking group-specific patterns.

A model or metric can look fair (or favorable) in aggregate while showing the opposite pattern within every subgroup, because a confounding variable correlated with group membership reverses the relationship upon disaggregation. This is why fairness audits should be stratified by relevant subgroups rather than relying on pooled statistics.

A feedback loop occurs when a model's predictions influence the future data it is trained on (e.g., predictive policing sends officers to predicted hotspots, generating more recorded incidents there, reinforcing the prediction). This can amplify initial biases over time, entrenching disparities even if the original model was only mildly biased.

Predictive parity requires that, given a positive prediction, the probability of being truly positive (PPV) is equal across groups; well-calibrated scores extend this to all score values. By the impossibility theorem, when base rates differ across groups, satisfying calibration/predictive parity forces unequal false positive and false negative rates between groups.

Model cards (Mitchell et al.) document a model's intended use, performance disaggregated by relevant subgroups, training/eval data, limitations, and ethical considerations. Datasheets for datasets (Gebru et al.) document a dataset's motivation, composition, collection process, preprocessing, and recommended uses. Both improve transparency, support informed adoption, and surface fairness risks before deployment.

Intrinsically interpretable models (linear/logistic regression, shallow decision trees, GAMs, rule lists) are transparent by construction. Post-hoc methods (SHAP, LIME, Grad-CAM) approximate explanations for an already-trained black box. The key trade-off: intrinsic models guarantee faithfulness but may sacrifice accuracy on complex tasks, while post-hoc explanations enable high-capacity models but are only approximations that can be unfaithful.

TCAV explains a model in terms of human-defined high-level concepts rather than raw features. It learns a Concept Activation Vector as the normal to a linear classifier separating concept vs non-concept examples in an activation space, then uses directional derivatives to quantify how sensitive a class prediction is to that concept (the TCAV score = fraction of inputs positively influenced).

ECE bins predictions by confidence, then averages the absolute gap between average confidence and empirical accuracy in each bin, weighted by bin size. A model can be well-calibrated overall yet miscalibrated within a subgroup, so low aggregate ECE does not guarantee calibration parity across protected groups; per-group ECE must be checked.

import numpy as np
def ece(probs, labels, n_bins=10):
    bins = np.linspace(0,1,n_bins+1)
    e = 0.0
    for lo, hi in zip(bins[:-1], bins[1:]):
        m = (probs > lo) & (probs <= hi)
        if m.sum() == 0: continue
        acc = labels[m].mean(); conf = probs[m].mean()
        e += (m.mean()) * abs(acc - conf)
    return e

Predictive equality requires equal false positive rates across groups (P(Ŷ=1 | Y=0) equal for all groups). It is the error-rate-balance complement to equal opportunity (which equalizes TPR); satisfying both predictive equality and equal opportunity together is exactly equalized odds.

In static embeddings, the Word Embedding Association Test (WEAT) measures differential cosine-similarity associations between target sets (e.g., gender terms) and attribute sets (e.g., career vs family) analogous to the human IAT. For LLMs, benchmarks like StereoSet, CrowS-Pairs, and BBQ compare model likelihoods/answers on stereotype vs anti-stereotype or ambiguous-context pairs to quantify stereotypical preference.

RLHF aligns outputs to human-preference labels, which can lower toxic or stereotyped completions; however, annotator demographics and guideline framing inject their own biases, and the model may learn to suppress surface markers while retaining biased representations (sycophancy, uneven refusal rates, or 'biased toward silence' for some groups). Evaluation must therefore go beyond toxicity scores to disaggregated behavioral and refusal-rate audits.