Model Interpretation & Explainability
421 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Model Interpretation & Explainability ## 🎯 Learning Objectives - Explain why model interpretability matters (regulatory, ethical, debugging) - Use SHAP values to explain individual predictions - Generate partial dependence plots for feature effects - Compare global and local interpretability methods ## 📖 Core Co...
Model Interpretation & Explainability
🎯 Learning Objectives
- Explain why model interpretability matters (regulatory, ethical, debugging)
- Use SHAP values to explain individual predictions
- Generate partial dependence plots for feature effects
- Compare global and local interpretability methods
📖 Core Content
6.1 Why Interpretability Matters
| Reason | Example |
|---|---|
| Regulatory | GDPR "right to explanation" for automated decisions |
| Debugging | Model relies on spurious correlations (e.g., "horses" identified by watermark) |
| Trust | Doctors won't use a black-box diagnostic model |
| Fairness | Detect if model discriminates against protected groups |
| Improvement | Understanding failures guides feature engineering |
6.2 Interpretability Methods
(Diagram)
6.3 SHAP Values
SHAP (SHapley Additive exPlanations) uses game theory to fairly distribute the prediction among features:
python# runnable # Note: Requires shap library # import shap # from sklearn.ensemble import RandomForestClassifier # from sklearn.datasets import load_iris # # iris = load_iris() # model = RandomForestClassifier() # model.fit(iris.data, iris.target) # # # Create SHAP explainer # explainer = shap.TreeExplainer(model) # shap_values = explainer.shap_values(iris.data) # # # Summary plot # shap.summary_plot(shap_values, iris.data, feature_names=iris.feature_names) # # # Force plot for a single prediction # shap.force_plot(explainer.expected_value[0], shap_values[0][0], iris.data[0])
6.4 Partial Dependence Plots
python# runnable from sklearn.inspection import partial_dependence, PartialDependenceDisplay from sklearn.ensemble import RandomForestRegressor from sklearn.datasets import load_diabetes import matplotlib.pyplot as plt diabetes = load_diabetes() model = RandomForestRegressor() model.fit(diabetes.data, diabetes.target) # PDP for the first two features PartialDependenceDisplay.from_estimator( model, diabetes.data, [0, 1], feature_names=diabetes.feature_names, grid_resolution=20 ) plt.show()
📝 Practice Questions
Q1: What's the difference between global and local interpretability?Global: Explains the entire model behavior. "Which features are most important overall?" Methods: feature importance, partial dependence plots. Local: Explains a single prediction. "Why did the model deny this specific loan?" Methods: SHAP, LIME. Both are needed for complete understanding. Q2: How does SHAP ensure fair allocation of feature contributions?SHAP uses Shapley values from cooperative game theory. Each feature is a "player" in a coalition (the model). The Shapley value is the average marginal contribution of a feature across all possible feature subsets. This ensures: (1) efficiency (predictions sum to contributions), (2) symmetry (equal contributions for equal features), (3) linearity, (4) null player (zero for unused features). Q3: What does a steep partial dependence plot indicate?A steep PDP indicates the feature has a strong effect on predictions. A flat PDP means the feature has little to no effect. The shape reveals the relationship: linear, U-shaped, threshold effect, etc. Example: age→income shows steep increase up to ~50 years, then plateau — meaningful insight for the model's decision logic. Join Discord PreviousHyperparameter TuningNextFeature Engineering