🔙 Backpropagation

538 words

3 min read

Visual companion

Python

Type and operator map

Python Week 1: the first filter for runtime behavior

Open gallery Related surface

View

Revision summary

What this note is really saying

Short form

# 🔙 Backpropagation ## 1. 🎯 Learning Objectives - Apply the chain rule to compute gradients in a neural network - Trace backpropagation through a 3-node network by hand - Explain the vanishing gradient problem - Implement backpropagation for a 2-layer MLP ## 2.

🔙 Backpropagation

1. 🎯 Learning Objectives

Apply the chain rule to compute gradients in a neural network
Trace backpropagation through a 3-node network by hand
Explain the vanishing gradient problem
Implement backpropagation for a 2-layer MLP

2. 📖 Core Content

3.1 Intuition: Reverse-Mode Differentiation

Forward propagation computes the output. Backpropagation computes the gradient of the loss with respect to each parameter by applying the chain rule from the output back to the input. Each parameter's gradient tells us: "if we tweak this weight, how much does the loss change?" This is what we need for gradient descent.

3.2 Computational Graph for a 3-Node Network

Consider a minimal network: input x, one hidden neuron with sigmoid, output neuron with sigmoid. (Diagram)

3.3 Forward Pass

z_1 = w_1 \cdot x

h = \sigma(z_1) = \frac{1}{1+e^{-z_1}}

z_2 = w_2 \cdot h

\hat{y} = \sigma(z_2)

L = \frac{1}{2}(y - \hat{y})^2

3.4 Backward Pass (Gradient Computation)

We want

\frac{\partial L}{\partial w_1}

and

\frac{\partial L}{\partial w_2}

. Chain rule for w₂:

\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_2} \cdot \frac{\partial z_2}{\partial w_2}

Step 1:

\frac{\partial L}{\partial \hat{y}} = \hat{y} - y

(since

L = \frac{1}{2}(y-\hat{y})^2

) Step 2:

\frac{\partial \hat{y}}{\partial z_2} = \sigma(z_2)(1-\sigma(z_2)) = \hat{y}(1-\hat{y})

Step 3:

\frac{\partial z_2}{\partial w_2} = h

Product:

\frac{\partial L}{\partial w_2} = (\hat{y} - y) \cdot \hat{y}(1-\hat{y}) \cdot h

Chain rule for w₁:

\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_2} \cdot \frac{\partial z_2}{\partial h} \cdot \frac{\partial h}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1}

Step 4:

\frac{\partial z_2}{\partial h} = w_2

Step 5:

\frac{\partial h}{\partial z_1} = \sigma(z_1)(1-\sigma(z_1)) = h(1-h)

Step 6:

\frac{\partial z_1}{\partial w_1} = x

Product:

\frac{\partial L}{\partial w_1} = (\hat{y} - y) \cdot \hat{y}(1-\hat{y}) \cdot w_2 \cdot h(1-h) \cdot x

3.5 Numerical Example

Let x=1, y=0 (target), w₁=0.5, w₂=0.8. Forward: z₁ = 0.5 × 1 = 0.5 h = σ(0.5) = 0.622 z₂ = 0.8 × 0.622 = 0.498 ŷ = σ(0.498) = 0.622 L = 0.5 × (0 - 0.622)² = 0.193 Backward: ∂L/∂ŷ = 0.622 - 0 = 0.622 ∂ŷ/∂z₂ = 0.622 × 0.378 = 0.235 ∂z₂/∂w₂ = h = 0.622 ∂L/∂w₂ = 0.622 × 0.235 × 0.622 = 0.0909 ∂z₂/∂h = w₂ = 0.8 ∂h/∂z₁ = 0.622 × 0.378 = 0.235 ∂z₁/∂w₁ = x = 1 ∂L/∂w₁ = 0.622 × 0.235 × 0.8 × 0.235 × 1 = 0.0275 Update (η=0.5): w₂_new = 0.8 - 0.5 × 0.0909 = 0.755 w₁_new = 0.5 - 0.5 × 0.0275 = 0.486

3.6 Vanishing Gradient

Notice that ∂L/∂w₁ = (something) × w₂ × h(1-h) × x. The sigmoid derivative h(1-h) is at most 0.25 and near 0 for large |z|. In deep networks, multiplying many such small gradients causes the gradient to "vanish" in early layers — they barely learn. This is why ReLU (derivative = 1 for positive inputs) is preferred in deep networks.

4. 📝 Practice Questions

Q1: For the 3-node network above with x=0.5, y=1, w₁=1, w₂=2, compute ∂L/∂w₂.

Answer: Forward: z₁=1×0.5=0.5, h=σ(0.5)=0.622, z₂=2×0.622=1.244, ŷ=σ(1.244)=0.776, L=0.5×(1-0.776)²=0.025. Backward: ∂L/∂ŷ=ŷ-y=0.776-1=-0.224, ∂ŷ/∂z₂=0.776×0.224=0.174, ∂z₂/∂w₂=h=0.622. ∂L/∂w₂=(-0.224)×0.174×0.622=-0.0242. Join Discord PreviousForward Prop & LossNextOptimization Methods