Neural Networks: Perceptron to MLP

🎯 Learning Objectives

• Implement a single perceptron and understand its limitations
• Build a multi-layer perceptron (MLP) for non-linear problems
• Derive and implement backpropagation
• Choose appropriate activation functions

📖 Core Content

3.1 Intuition: Building a Brain-like Computing System

3.2 The Perceptron (1958)

3.3 Multi-Layer Perceptron (MLP)

3.4 Activation Functions

3.5 Backpropagation

1. Forward pass: Compute predictions
2. Compute loss: $\mathcal{L}(\hat{y}, y)$
3. Backward pass:
   • $\frac{\partial \mathcal{L}}{\partial W^{(L)}} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z^{(L)}} \cdot \frac{\partial z^{(L)}}{\partial W^{(L)}}$
   • Propagate error backward through layers

python
Run Inline
# runnable
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
mlp = MLPClassifier(
    hidden_layer_sizes=(64, 32),
    activation='relu',
    solver='adam',
    learning_rate='adaptive',
    max_iter=500,
    early_stopping=True,
    random_state=42
)
mlp.fit(X_train, y_train)
print(f"MLP test accuracy: {mlp.score(X_test, y_test):.3f}")
print(f"Loss curve: {mlp.loss_curve_[:5]}")

📝 Practice Questions

Q1: Why can't a single perceptron learn XOR?

XOR is not linearly separable. A single perceptron creates a linear decision boundary (line in 2D). XOR requires a non-linear boundary — you can't draw a single straight line that separates (0,0) and (1,1) from (0,1) and (1,0). An MLP with at least one hidden layer can learn XOR. Q2: What problem does ReLU solve compared to sigmoid?

Vanishing gradient: Sigmoid's gradient saturates near 0 and 1 (gradient → 0). Deep networks with sigmoid have gradients that vanish to 0 in early layers. ReLU's gradient is 1 for positive inputs and 0 for negative — it doesn't saturate for positive values, allowing gradients to flow through deep networks. Q3: What is the "dying ReLU" problem?

If a ReLU neuron gets a large negative bias update, its output becomes 0 for all inputs, and its gradient is 0 (can't recover). The neuron is "dead." Solutions: use Leaky ReLU (small positive slope for negative inputs) or ELU. Q4: How does backpropagation use the chain rule?

For a 2-layer network: $\mathcal{L} = \mathcal{L}(\sigma(W_2 \sigma(W_1 x + b_1) + b_2), y)$

The chain rule gives: $\frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_2} \cdot \frac{\partial a_2}{\partial z_2} \cdot \frac{\partial z_2}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial W_1}$

Each term is a Jacobian matrix. The key insight: many terms are reused — the error at layer l depends on the error at layer l+1, so we compute backward from output to input. Join Discord PreviousNaive BayesNextKernel Methods