Explainable AI

Key ideas

• Why explain: to trust and justify decisions (medicine, credit), to debug models that learned a shortcut, to check fairness, and to meet regulation — the GDPR's right to explanation concerns local explanations of individual decisions¹. They matter most when what we want (fairness, safety) is not fully captured by the training objective, since a model can meet the objective with tricks its designers never intended¹.
• Interpretability vs explainability: interpretability is understanding how the model works; explainability is understanding how it produced a given result¹. A model can be interpretable by design (intrinsic: sparse linear model, short tree) or explained post-hoc after training².
• Model-agnostic methods only query inputs and outputs; model-specific ones read internals (gradients, activations). Global methods describe average behaviour over a dataset; local methods explain one prediction².

• PDP averages the prediction over the data with the feature of interest fixed; each row's curve is an ICE line, and the PDP is their mean³.
• Permutation importance: shuffle one column $K$ times on held-out data and average the score drop; it measures how much this model relies on the feature⁴.
• LIME samples perturbed inputs around $x$, gets the black box's predictions, weights them by closeness to $x$ and fits a sparse linear model. For images its features are super-pixels switched on or greyed out². Such explanations can expose shortcuts, e.g. a wolf-vs-husky classifier that relied on snow in the background².
• Shapley values / SHAP average a feature's marginal contribution over all coalitions of the other features. They are the only additive attribution with local accuracy, missingness and consistency; they explain the gap between $f(x)$ and the base value $\mathbb{E}[f(X)]$⁵.
• Gradient saliency (vanilla gradient) differentiates the class score with respect to the pixels — a first-order Taylor view of the network². For RGB images Simonyan et al. keep the largest absolute gradient over the three channels, and they work with the unnormalised score $S_c$ rather than the softmax probability, which can also rise when other classes' scores fall⁶. SmoothGrad averages such maps over $n \approx 50$ noisy copies with noise level 10–20% of the pixel range².
• Grad-CAM averages the gradients of the pre-softmax score $y^c$ over each feature map of (usually) the last convolutional layer, which still keeps the spatial layout, weights the maps with these averages, sums them and applies a ReLU to keep evidence for the class; the coarse map is upscaled and laid over the image².
• Integrated gradients (IG) fixes the saturation problem of gradients: a flat region gives zero gradient even for an important input. It integrates gradients along the straight path from a baseline $x'$ (black image, zero embedding) to $x$ and satisfies sensitivity, implementation invariance and completeness⁷. Check that the attributions sum to $F(x) - F(x')$ and raise the number of steps (20–300 is typical) if not⁷.
• Occlusion is perturbation-based: slide a grey patch over the image and record the drop of the class score. It is model-agnostic², but needs one forward pass per patch position.
• Counterfactual explanations state the smallest change that would give the desired outcome ("with income 64k the loan would be approved"). Wachter et al. minimise a prediction loss plus an L1 distance in which each feature's change is divided by its median absolute deviation (MAD), which is robust to outliers and puts features on one scale²; the L1 form tends to change only a few features.
• Evaluating explanations: plausibility is how convincing an explanation looks to people, faithfulness is whether it reflects the model's actual reasoning⁸. Sanity checks randomise the weights or the labels: Guided Backprop and Guided Grad-CAM barely change, behaving like edge detectors, while vanilla gradients and Grad-CAM pass². Attention weights are often read as a built-in explanation, but work on text classifiers has cast doubt on how faithful they are⁸.

Formulas

Permutation importance with reference score $s$ and shuffled scores $s_{k,j}$⁴: $i_j = s - \frac{1}{K}\sum_{k=1}^{K} s_{k,j}$

Partial dependence (brute force over $n$ rows)³: $\mathrm{pd}_{X_S}(x_S) \approx \frac{1}{n}\sum_{i=1}^{n} f(x_S, x_C^{(i)})$

LIME: $\xi(x) = \operatorname*{arg\,min}_{g \in G} L(f, g, \pi_x) + \Omega(g)$, where $\pi_x$ is an exponential proximity kernel such as $\exp(-D(x, z)^2/\sigma^2)$ whose width $\sigma$ sets the neighbourhood².

Shapley value of feature $i$ among $M$ features, and local accuracy⁵: $\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!\,(M - |S| - 1)!}{M!}\,\bigl[v(S \cup \{i\}) - v(S)\bigr], \qquad \sum_{i=1}^{M} \phi_i = v(F) - v(\emptyset)$

For SHAP, $v(F) - v(\emptyset) = f(x) - \mathbb{E}[f(X)]$. Linear SHAP (independent features): $\phi_i = w_i(x_i - \mathbb{E}[x_i])$⁵.

Saliency of pixel $(i, j)$: $M_{ij} = \max_k \lvert \partial S_c / \partial I_{ijk} \rvert$ over channels $k$⁶. SmoothGrad: $\hat{M}_c(x) = \frac{1}{n}\sum_{k=1}^{n} M_c(x + \mathcal{N}(0, \sigma^2))$².

Grad-CAM²: $\alpha_k^c = \frac{1}{Z}\sum_{i}\sum_{j}\frac{\partial y^c}{\partial A^k_{ij}}, \qquad L^c = \mathrm{ReLU}\Bigl(\sum_k \alpha_k^c A^k\Bigr)$

Integrated gradients and completeness⁷: $\mathrm{IG}_i(x) = (x_i - x'_i)\int_0^1 \frac{\partial F(x' + \alpha(x - x'))}{\partial x_i}\,d\alpha, \qquad \sum_i \mathrm{IG}_i(x) = F(x) - F(x')$

Counterfactual: $\min_{x'} \lambda\,(f(x') - y')^2 + \sum_k |x_k - x'_k| / \mathrm{MAD}_k$².

Worked example

Shapley values of $f(x_1, x_2) = x_1 + 2x_2 + x_1x_2$ at $x = (1, 2)$, with absent features set to the baseline 0. Coalition values: $v(\emptyset) = 0$, $v(\{1\}) = 1$, $v(\{2\}) = 4$, $v(\{1, 2\}) = 7$. With two features each coalition order has weight $\frac{1}{2}$: $\phi_1 = \frac{1}{2}[(1 - 0) + (7 - 4)] = 2$ and $\phi_2 = \frac{1}{2}[(4 - 0) + (7 - 1)] = 5$. They sum to $f(x) - f(0) = 7$: the interaction term $x_1x_2 = 2$ is split equally.

Occlusion cost: a $k \times k$ patch with stride $s$ on an $H \times H$ image gives $((H - k)/s + 1)^2$ positions, e.g. $((32 - 8)/4 + 1)^2 = 49$ forward passes.

In code

import torch
F = lambda z: z[..., 0] ** 2 * z[..., 1]           # toy model F(x1, x2) = x1^2 * x2
x, x0 = torch.tensor([2.0, 3.0]), torch.zeros(2)   # input and baseline
xg = x.clone().requires_grad_(True)
F(xg).backward()
print(xg.grad)                                     # tensor([12.,  4.])  vanilla gradient
m = 300                                            # Riemann steps
alphas = torch.arange(1, m + 1).view(-1, 1) / m
path = (x0 + alphas * (x - x0)).requires_grad_(True)
F(path).sum().backward()
ig = (x - x0) * path.grad.mean(dim=0)              # integrated gradients
print(ig, ig.sum())                                # tensor([8.0400, 4.0200]) tensor(12.0601)

The exact values are $\mathrm{IG} = (8, 4)$, summing to $F(x) - F(x') = 12$; the gradient $(12, 4)$ does not add up to the output⁷.

Common pitfalls

• A pretty map is not a faithful one: run sanity checks², or erase the top-ranked inputs and check that the decision changes⁸. High test accuracy on data from the same collection does not rule out a Clever Hans shortcut such as a watermark, because the test set shares it^1,2.
• Permutation importance shows reliance, not direction or causality; correlated features hide each other's importance, since the model still sees the copy⁴.
• A flat PDP can hide opposite effects in different rows — look at ICE lines; with correlated features PDPs evaluate impossible inputs³.
• SHAP values are measured from the base value $\mathbb{E}[f(X)]$, not from zero: a negative weight with a below-average input gives a positive $\phi$⁵.
• LIME's kernel width defines "local": with a very wide kernel, distant samples weigh as much as close ones, so the surrogate follows the global trend, which can even have the opposite sign to the local effect².
• Plain gradients miss saturated features; IG fixes this, but any pixel equal to the baseline (black pixels with a black baseline) gets exactly zero⁷.
• Grad-CAM averages (not sums) the gradients, uses the pre-softmax score, keeps only positive evidence through the ReLU, and gives a map as coarse as the feature maps².
• Attention weights are not explanations by default; check their faithfulness like any other attribution⁸.