Quiz 2
Registry Synced

🔙 Backpropagation

538 words
3 min read

Reading compass

Now · 1. 🎯 Learning Objectives

🔙 Backpropagation

1. 🎯 Learning Objectives

  • Apply the chain rule to compute gradients in a neural network
  • Trace backpropagation through a 3-node network by hand
  • Explain the vanishing gradient problem
  • Implement backpropagation for a 2-layer MLP

2. 📖 Core Content

3.1 Intuition: Reverse-Mode Differentiation

Forward propagation computes the output. Backpropagation computes the gradient of the loss with respect to each parameter by applying the chain rule from the output back to the input. Each parameter's gradient tells us: "if we tweak this weight, how much does the loss change?" This is what we need for gradient descent.

3.2 Computational Graph for a 3-Node Network

Consider a minimal network: input x, one hidden neuron with sigmoid, output neuron with sigmoid. (Diagram)

3.3 Forward Pass

z1=w1⋅xz_1 = w_1 \cdot x h=σ(z1)=11+e−z1h = \sigma(z_1) = \frac{1}{1+e^{-z_1}} z2=w2⋅hz_2 = w_2 \cdot h y^=σ(z2)\hat{y} = \sigma(z_2) L=12(y−y^)2L = \frac{1}{2}(y - \hat{y})^2

3.4 Backward Pass (Gradient Computation)

We want ∂L∂w1\frac{\partial L}{\partial w_1} and ∂L∂w2\frac{\partial L}{\partial w_2}. Chain rule for w₂:
∂L∂w2=∂L∂y^⋅∂y^∂z2⋅∂z2∂w2\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_2} \cdot \frac{\partial z_2}{\partial w_2}
Step 1: ∂L∂y^=y^−y\frac{\partial L}{\partial \hat{y}} = \hat{y} - y (since L=12(y−y^)2L = \frac{1}{2}(y-\hat{y})^2) Step 2: ∂y^∂z2=σ(z2)(1−σ(z2))=y^(1−y^)\frac{\partial \hat{y}}{\partial z_2} = \sigma(z_2)(1-\sigma(z_2)) = \hat{y}(1-\hat{y}) Step 3: ∂z2∂w2=h\frac{\partial z_2}{\partial w_2} = h Product: ∂L∂w2=(y^−y)⋅y^(1−y^)⋅h\frac{\partial L}{\partial w_2} = (\hat{y} - y) \cdot \hat{y}(1-\hat{y}) \cdot h Chain rule for w₁:
∂L∂w1=∂L∂y^⋅∂y^∂z2⋅∂z2∂h⋅∂h∂z1⋅∂z1∂w1\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_2} \cdot \frac{\partial z_2}{\partial h} \cdot \frac{\partial h}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1}
Step 4: ∂z2∂h=w2\frac{\partial z_2}{\partial h} = w_2 Step 5: ∂h∂z1=σ(z1)(1−σ(z1))=h(1−h)\frac{\partial h}{\partial z_1} = \sigma(z_1)(1-\sigma(z_1)) = h(1-h) Step 6: ∂z1∂w1=x\frac{\partial z_1}{\partial w_1} = x Product: ∂L∂w1=(y^−y)⋅y^(1−y^)⋅w2⋅h(1−h)⋅x\frac{\partial L}{\partial w_1} = (\hat{y} - y) \cdot \hat{y}(1-\hat{y}) \cdot w_2 \cdot h(1-h) \cdot x

3.5 Numerical Example

Let x=1, y=0 (target), w₁=0.5, w₂=0.8. Forward: z₁ = 0.5 × 1 = 0.5 h = σ(0.5) = 0.622 z₂ = 0.8 × 0.622 = 0.498 ŷ = σ(0.498) = 0.622 L = 0.5 × (0 - 0.622)² = 0.193 Backward: ∂L/∂ŷ = 0.622 - 0 = 0.622 ∂ŷ/∂z₂ = 0.622 × 0.378 = 0.235 ∂z₂/∂w₂ = h = 0.622 ∂L/∂w₂ = 0.622 × 0.235 × 0.622 = 0.0909 ∂z₂/∂h = w₂ = 0.8 ∂h/∂z₁ = 0.622 × 0.378 = 0.235 ∂z₁/∂w₁ = x = 1 ∂L/∂w₁ = 0.622 × 0.235 × 0.8 × 0.235 × 1 = 0.0275 Update (η=0.5): w₂_new = 0.8 - 0.5 × 0.0909 = 0.755 w₁_new = 0.5 - 0.5 × 0.0275 = 0.486

3.6 Vanishing Gradient

Notice that ∂L/∂w₁ = (something) × w₂ × h(1-h) × x. The sigmoid derivative h(1-h) is at most 0.25 and near 0 for large |z|. In deep networks, multiplying many such small gradients causes the gradient to "vanish" in early layers — they barely learn. This is why ReLU (derivative = 1 for positive inputs) is preferred in deep networks.

4. 📝 Practice Questions

Q1: For the 3-node network above with x=0.5, y=1, w₁=1, w₂=2, compute ∂L/∂w₂.
Answer: Forward: z₁=1×0.5=0.5, h=σ(0.5)=0.622, z₂=2×0.622=1.244, ŷ=σ(1.244)=0.776, L=0.5×(1-0.776)²=0.025. Backward: ∂L/∂ŷ=ŷ-y=0.776-1=-0.224, ∂ŷ/∂z₂=0.776×0.224=0.174, ∂z₂/∂w₂=h=0.622. ∂L/∂w₂=(-0.224)×0.174×0.622=-0.0242. Join Discord PreviousForward Prop & LossNextOptimization Methods
Document outline

Keep your place and jump directly to a heading.

Table of Contents
System Normal // Awaiting Context

Intelligence Hub

Navigate the knowledge graph to generate context. The Hub adapts dynamically to surface backlinks, related notes, and metadata insights.

Open library →Launch Assessment Engine