🧬 Deep Learning History & The Perceptron

982 words

5 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# 🧬 Deep Learning History & The Perceptron ## 1. 🎯 Learning Objectives By the end of this topic, you will be able to: - Trace the history of neural networks from McCulloch-Pitts (1943) to modern deep learning - Explain the McCulloch-Pitts neuron model and its limitations - Implement Rosenblatt's perceptron and exp...

🧬 Deep Learning History & The Perceptron

1. 🎯 Learning Objectives

By the end of this topic, you will be able to:

Trace the history of neural networks from McCulloch-Pitts (1943) to modern deep learning
Explain the McCulloch-Pitts neuron model and its limitations
Implement Rosenblatt's perceptron and explain the perceptron learning algorithm
State the Perceptron Convergence Theorem
Explain why the XOR problem caused the first AI winter
Compare biological neurons to artificial neurons

2. 📋 Prerequisites

Prerequisite	Course	Why It Matters
Linear algebra basics	BSMA1001	Vector operations, dot products
Binary classification	BSCS3003	What perceptrons do
Gradient descent intuition	BSMA1002	How perceptrons learn

3. 📖 Core Content

3.1 Intuition: Learning from Biology

Your brain has about 86 billion neurons, each connected to thousands of others. When a neuron receives enough input from its neighbors, it "fires" — sending an electrical signal down its axon to other neurons. Artificial neural networks are a vast simplification of this: they take weighted sums of inputs and apply an activation function to decide whether to "fire." The history of deep learning is a story of ideas that were decades ahead of their time, waiting for data and computation to catch up.

3.2 The McCulloch-Pitts Neuron (1943)

Warren McCulloch (neuroscientist) and Walter Pitts (logician) created the first mathematical model of a neuron in 1943:

y = f\left(\sum_{i=1}^n w_i x_i - \theta\right)

Where:

$x_i$ : Binary inputs (0 or 1)
$w_i$ : Weights (excitatory = +1, inhibitory = -1)
$\theta$ : Threshold
$f$ : Step function (outputs 1 if sum ≥ θ, else 0) Key insight: The MP neuron is a logical threshold unit. With appropriate weights and thresholds, it can compute AND, OR, NOT — any logical function. AND Gate:
Inputs: x1, x2 ∈ {0,1}
Weights: w1 = 1, w2 = 1
Threshold: θ = 2
Output: 1 only when x1 = 1 AND x2 = 1 OR Gate:
Weights: w1 = 1, w2 = 1
Threshold: θ = 1
Output: 1 when x1 = 1 OR x2 = 1 Limitation: Weights are fixed — there is no learning algorithm.

3.3 Rosenblatt's Perceptron (1957)

Frank Rosenblatt introduced the perceptron, which could learn its weights from data. The perceptron takes real-valued inputs, applies weights, sums them, and outputs through a step function. Mathematical formulation:

\hat{y} = \begin{cases} 1 & \text{if } \mathbf{w} \cdot \mathbf{x} + b > 0 \\ 0 & \text{otherwise} \end{cases}

Perceptron Learning Rule: For each misclassified example

\mathbf{x}^{(i)}

If $\hat{y} = 0$ but $y^{(i)} = 1$ : $\mathbf{w} \leftarrow \mathbf{w} + \eta \cdot \mathbf{x}^{(i)}$ , $b \leftarrow b + \eta$
If $\hat{y} = 1$ but $y^{(i)} = 0$ : $\mathbf{w} \leftarrow \mathbf{w} - \eta \cdot \mathbf{x}^{(i)}$ , $b \leftarrow b - \eta$ Where $\eta$ is the learning rate.

3.4 Perceptron Convergence Theorem

Theorem: If the training data is linearly separable, the perceptron learning algorithm will converge to a solution (zero training errors) in a finite number of steps. Proof sketch:

Assume there exists a weight vector $\mathbf{w}^*$ that correctly classifies all points with margin $\gamma > 0$
Show that after each update, $\mathbf{w}^{(k)}$ gets "closer" to $\mathbf{w}^*$ in terms of cosine similarity
Bound the number of updates by $(R/\gamma)^2$ where R is the radius of the data

3.5 The XOR Problem and the First AI Winter

In 1969, Minsky and Papert published "Perceptrons," proving that a single perceptron cannot learn the XOR function:

x1	x2	XOR
0	0	0
0	1	1
1	0	1
1	1	0

XOR is not linearly separable — no single straight line can separate the two classes. (Diagram) This limitation, combined with exaggerated claims about perceptrons, led to the first AI winter — funding dried up, and neural network research stagnated for nearly a decade.

3.6 The Solution: Multi-Layer Perceptrons

The solution to XOR requires at least two layers of neurons:

First layer: computes two separating lines
Second layer: combines them logically XOR(x1, x2) = (x1 AND NOT x2) OR (NOT x1 AND x2) This requires hidden layers — leading to the Multi-Layer Perceptron (MLP).

4. 📐 Key Formulas / Concepts

Concept	Formula	Notes
MP Neuron	$y = f(\sum w_i x_i - \theta)$	Fixed weights, logical
Perceptron	$y = f(\mathbf{w} \cdot \mathbf{x} + b)$	Learns weights
Perceptron update	$\mathbf{w} \leftarrow \mathbf{w} + \eta \cdot y \cdot \mathbf{x}$	For misclassified only
Convergence bound	$(R/\gamma)^2$ steps	For separable data

5. ⚠️ Common Pitfalls

Pitfall 1: Confusing MP Neuron with Perceptron

The mistake: Treating them as the same thing. Correct approach: MP neuron has fixed weights (no learning). Perceptron has learnable weights via the perceptron learning rule.

Pitfall 2: Thinking Perceptrons Are Useless

The mistake: "Perceptrons can't learn XOR, so they're useless." Correct approach: Perceptrons work perfectly for linearly separable problems. Many real-world problems (especially with enough features) are linearly separable.

Pitfall 3: Forgetting the Bias Term

The mistake: Using perceptron without bias

b

. Correct approach: Without bias, the decision boundary always passes through the origin, severely limiting the perceptron's capacity.

6. 📝 Practice Questions

Q1: Show that a single perceptron cannot learn XOR.

Answer: XOR requires a non-linear decision boundary. A perceptron draws a linear boundary (hyperplane). The XOR truth table has points (0,0) and (1,1) on one side (class 0) and (0,1) and (1,0) on the other (class 1) — no single line can separate these. Q2: Trace the perceptron learning rule on AND gate data.

Answer: Initialize w=[0,0], b=0, η=1. Example (0,0): ŷ=0, y=0 → correct. (0,1): ŷ=0, y=0 → correct. (1,0): ŷ=0, y=0 → correct. (1,1): ŷ=0, y=1 → misclassified! Update: w=[1,1], b=1. Repeat until all correct. Converges quickly. Q3: What is the significance of the Perceptron Convergence Theorem?

Answer: It guarantees that if the data is linearly separable, the perceptron will find a separating hyperplane in finite time. This was a powerful theoretical result that gave legitimacy to neural network research. Join Discord NextActivation Functions