Neural Sync Active

Neural Networks: Perceptron to MLP

Registry Synced

Neural Networks: Perceptron to MLP

600 words

3 min read

Reading compass

Now · 🎯 Learning Objectives

Neural Networks: Perceptron to MLP

🎯 Learning Objectives

Implement a single perceptron and understand its limitations
Build a multi-layer perceptron (MLP) for non-linear problems
Derive and implement backpropagation
Choose appropriate activation functions

📖 Core Content

3.1 Intuition: Building a Brain-like Computing System

A neural network is a chain of simple computing units (neurons), each taking inputs, performing a weighted sum, applying a non-linear activation, and passing the result forward. Stacking these layers lets the network learn increasingly abstract features — first edges, then shapes, then objects.

3.2 The Perceptron (1958)

\hat{y} = \begin{cases} 1 & \text{if } w^T x + b > 0 \\ 0 & \text{otherwise} \end{cases}

Limitation: A single perceptron can only learn linearly separable functions — it can't learn XOR. This was shown by Minsky & Papert (1969) and caused the first "AI winter."

3.3 Multi-Layer Perceptron (MLP)

An MLP adds hidden layers with non-linear activation functions:

a^{(1)} = \sigma(W^{(1)} x + b^{(1)})

a^{(2)} = \sigma(W^{(2)} a^{(1)} + b^{(2)})

\hat{y} = \text{softmax}(W^{(3)} a^{(2)} + b^{(3)})

(Diagram)

3.4 Activation Functions

Function	Formula	Range	Use Case
Sigmoid	$\sigma(x) = 1/(1+e^{-x})$	(0, 1)	Output for binary classification
Tanh	$\tanh(x) = (e^x-e^{-x})/(e^x+e^{-x})$	(-1, 1)	Hidden layers (zero-centered)
ReLU	$\max(0, x)$	[0, ∞)	Default for hidden layers
Leaky ReLU	$\max(0.01x, x)$	(-∞, ∞)	Fixes "dying ReLU"

3.5 Backpropagation

Backpropagation computes gradients via the chain rule, from output back to input:

Forward pass: Compute predictions
Compute loss: $\mathcal{L}(\hat{y}, y)$
Backward pass:
- $\frac{\partial \mathcal{L}}{\partial W^{(L)}} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z^{(L)}} \cdot \frac{\partial z^{(L)}}{\partial W^{(L)}}$
- Propagate error backward through layers


python
# runnable
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
mlp = MLPClassifier(
    hidden_layer_sizes=(64, 32),
    activation='relu',
    solver='adam',
    learning_rate='adaptive',
    max_iter=500,
    early_stopping=True,
    random_state=42
)
mlp.fit(X_train, y_train)
print(f"MLP test accuracy: {mlp.score(X_test, y_test):.3f}")
print(f"Loss curve: {mlp.loss_curve_[:5]}")

📝 Practice Questions

Q1: Why can't a single perceptron learn XOR?

XOR is not linearly separable. A single perceptron creates a linear decision boundary (line in 2D). XOR requires a non-linear boundary — you can't draw a single straight line that separates (0,0) and (1,1) from (0,1) and (1,0). An MLP with at least one hidden layer can learn XOR. Q2: What problem does ReLU solve compared to sigmoid?

Vanishing gradient: Sigmoid's gradient saturates near 0 and 1 (gradient → 0). Deep networks with sigmoid have gradients that vanish to 0 in early layers. ReLU's gradient is 1 for positive inputs and 0 for negative — it doesn't saturate for positive values, allowing gradients to flow through deep networks. Q3: What is the "dying ReLU" problem?

If a ReLU neuron gets a large negative bias update, its output becomes 0 for all inputs, and its gradient is 0 (can't recover). The neuron is "dead." Solutions: use Leaky ReLU (small positive slope for negative inputs) or ELU. Q4: How does backpropagation use the chain rule?

For a 2-layer network: $\mathcal{L} = \mathcal{L}(\sigma(W_2 \sigma(W_1 x + b_1) + b_2), y)$

The chain rule gives: $\frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_2} \cdot \frac{\partial a_2}{\partial z_2} \cdot \frac{\partial z_2}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial W_1}$

Each term is a Jacobian matrix. The key insight: many terms are reused — the error at layer l depends on the error at layer l+1, so we compute backward from output to input. Join Discord PreviousNaive BayesNextKernel Methods