Decision Trees

🎯 Learning Objectives

• Explain how decision trees recursively partition the feature space
• Compute entropy, Gini impurity, and information gain
• Understand when to stop splitting and how to prune trees
• Implement decision tree classifiers and regressors with sklearn
• Compare decision trees with other algorithms

📋 Prerequisites

• Basic probability — entropy requires understanding probability distributions
• Logarithms — for information theory measures
• Classification concepts — predicting discrete classes

📖 Core Content

6.1 Intuition: The 20 Questions Game

• Each internal node asks a question about a feature (e.g., "Is age > 30?")
• Each branch represents an answer (Yes/No)
• Each leaf node gives the final prediction (Diagram) The key question is: which feature should be the root? The answer is the feature that best separates the classes — the one that gives the purest subsets.

6.2 Formal Definition

• Each internal node tests a feature $x_j$ against a threshold $t$: $x_j \leq t$
• Each leaf predicts a class (classification) or value (regression)
• The tree is built top-down, greedy — at each node, pick the best split Notation:
• $S$: set of training examples at the current node
• $p_c$: proportion of examples in class $c$ at the node
• $C$: number of classes

6.3 Split Quality Metrics

• Entropy = 0 when all examples are the same class (pure)
• Entropy = $\log_2(C)$ when classes are perfectly mixed Information Gain:

• Gini = 0 when all examples are the same class (pure)
• Gini is maximized when classes are evenly split

6.4 Worked Example 1: Computing Entropy and IG by Hand

• P(Yes) = 4/5, P(No) = 1/5
• $H(S) = -\frac{4}{5}\log_2(\frac{4}{5}) - \frac{1}{5}\log_2(\frac{1}{5})$
• $= -0.8(-0.322) - 0.2(-2.322) = 0.258 + 0.464 = 0.722$ Step 2: Compute IG for splitting on "Outlook". Split values:
• Sunny: {No, Yes} → 2 examples: H = -0.5log₂(0.5) - 0.5log₂(0.5) = 1.0
• Overcast: {Yes} → 1 example: H = 0 (pure)
• Rainy: {Yes, Yes} → 2 examples: H = 0 (pure)

• Hot: {No, Yes} → H = 1.0
• Mild: {Yes} → H = 0
• Cool: {Yes} → H = 0

6.5 Worked Example 2: Gini Impurity

• p₁ = 4/5, p₂ = 1/5
• $G = 1 - (4/5)^2 - (1/5)^2 = 1 - 0.64 - 0.04 = 0.32$ After splitting on Outlook:
• Sunny (2): p₁=0.5, p₂=0.5 → G = 0.5
• Overcast (1): p₁=1, p₂=0 → G = 0
• Rainy (2): p₁=1, p₂=0 → G = 0 Weighted Gini after split: $\frac{2}{5}(0.5) + \frac{1}{5}(0) + \frac{2}{5}(0) = 0.2$ Gini reduction: 0.32 - 0.2 = 0.12.

6.6 Pruning: Preventing Overfitting

• Max depth: stop at depth 3-5
• Min samples per leaf: require ≥ 5-10 examples per leaf
• Min impurity decrease: only split if IG > threshold Post-pruning (cost-complexity pruning):
• Grow a full tree
• Prune branches that don't significantly improve validation performance
• Use cost-complexity parameter α: penalize trees with many leaves

6.7 Decision Tree Regressors

6.8 Python Implementation

python
Run Inline
# runnable
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor, plot_tree
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Train with pruning
dt = DecisionTreeClassifier(
    criterion='gini',
    max_depth=3,          # Pre-pruning
    min_samples_leaf=5,    # Pre-pruning
    random_state=42
)
dt.fit(X_train, y_train)
print(f"Train accuracy: {dt.score(X_train, y_train):.3f}")
print(f"Test accuracy:  {dt.score(X_test, y_test):.3f}")
print(f"Tree depth: {dt.get_depth()}")
print(f"Number of leaves: {dt.get_n_leaves()}")
# Feature importance
for name, imp in zip(iris.feature_names, dt.feature_importances_):
    print(f"  {name}: {imp:.3f}")
# Visualize the tree (text for now)
text_representation = plot_tree(dt, feature_names=iris.feature_names,
                                 class_names=iris.target_names, filled=True)
plt.show()

6.9 When to Use / Not Use

📐 Key Formulas / Concepts

⚠️ Common Pitfalls

Pitfall 1: Growing Trees Too Deep

Pitfall 2: Ignoring Feature Importance Drift

Pitfall 3: Unstable Decision Boundaries

📝 Practice Questions

Q1: Calculate entropy when p(Yes)=0.9, p(No)=0.1

$H = -0.9\log_2(0.9) - 0.1\log_2(0.1)$ $= -0.9(-0.152) - 0.1(-3.322)$ $= 0.137 + 0.332 = 0.469$

The set is mostly pure (low entropy). Compare to pure (H=0) and perfectly mixed (H=1.0 for binary). Q2: When would entropy and Gini choose different splits?

Gini and entropy are usually very similar. Entropy tends to favor splits that create more balanced children, while Gini favors splits that isolate the largest class into one child. In practice, the difference is negligible — both give similar performance. Q3: A node has samples [5 Red, 5 Blue]. Compute Gini.

$G = 1 - (5/10)^2 - (5/10)^2 = 1 - 0.25 - 0.25 = 0.5$

This is the maximum Gini for binary classification (perfectly mixed). Q4: What does a feature importance of 0 for petal width mean?

The decision tree never used "petal width" for splitting — it's not useful for separating classes given the other features. This doesn't mean "petal width is irrelevant in general" — it might be redundant with other features. Q5: Why are decision trees called "white box" models?

Because every decision is explicit and can be understood by humans. You can trace any prediction through the tree: "Age > 30 → Yes, Income > 50k → No → Predict: Won't Buy." This is contrast to "black box" models (neural networks, ensemble methods) where the decision process is opaque. Q6: Your tree has 100% training accuracy but 60% test accuracy. What's wrong?

Overfitting — the tree is too deep. Solutions:

1. Reduce max_depth
2. Increase min_samples_leaf
3. Use cost-complexity pruning
4. Use cross-validation to find optimal parameters

A depth-10 tree on 100 examples will almost certainly overfit. Q7: How does a decision tree handle missing values?

sklearn's DecisionTreeClassifier does not handle missing values — you must impute them first. Some implementations (like C4.5) can split on missing values by sending examples proportionally down branches. Common approaches: impute with mean/median/mode, or create an "unknown" category. Q8: For a regression tree, what value does a leaf predict?

The mean of all target values in that leaf. For example, if a leaf contains houses with prices [$200k,$250k, $300k], it predicts$250k. The split criterion is variance reduction — we want leaves with low target variance. Q9: Compare information gain vs gain ratio.

Problem: Information gain favors features with many values (e.g., "Student ID" would perfectly split every example but is useless).

Gain Ratio normalizes by split entropy: $GR = \frac{IG(S, f)}{H(Split)}$

This penalizes features that split data into many tiny subsets. Gain Ratio is the default in C4.5 (the successor to ID3). Q10: How do you handle continuous features in decision trees?

For a continuous feature, sort the values, then try splits at midpoints between consecutive sorted values. For example, if ages are [18, 25, 30, 40], try thresholds at 21.5, 27.5, 35.0. Choose the threshold with the highest information gain. This is O(m log m) per feature per node. Q11: What is cost-complexity pruning?

It adds a penalty for tree size (number of leaves): $Cost = Error(T) + \alpha \cdot |T|$

Where $|T|$ is the number of leaf nodes and $\alpha \geq 0$ controls the penalty. Higher α → simpler tree. sklearn uses ccp_alpha — use cross-validation to find the optimal α. Q12: Implement a decision tree classifier for the iris dataset and visualize the tree.

python
Run Inline
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=42)

dt = DecisionTreeClassifier(max_depth=3, random_state=42)
dt.fit(X_train, y_train)

print(export_text(dt, feature_names=iris.feature_names))
print(f"Test accuracy: {dt.score(X_test, y_test):.3f}")

🔗 Cross-References

• Next Topic: Ensemble Methods: Bagging & Random Forest — combining trees
• Related: Decision Tree Pruning — deeper on post-pruning
• Related: Gradient Boosting — boosting trees
• External: IITM BSCS2004 Week 6, Hands-On ML Ch. 6, ISLR Ch. 8 Join Discord Previousk-Nearest NeighborsNextBagging & Random Forest