Quiz 2

➡️ Forward Propagation & Loss Functions

277 words
1 min read
Python Week 1: the first filter for runtime behavior
Visual companion
Python
Type and operator map

Python Week 1: the first filter for runtime behavior

Open galleryRelated surface
View
Revision summary

What this note is really saying

Short form

# ➡️ Forward Propagation & Loss Functions ## 1. 🎯 Learning Objectives - Compute forward propagation through a 2-layer MLP - Implement softmax function and explain its purpose - Compare MSE vs cross-entropy loss for different problems - Trace a forward pass with actual numbers ## 2.

➡️ Forward Propagation & Loss Functions

1. 🎯 Learning Objectives

  • Compute forward propagation through a 2-layer MLP
  • Implement softmax function and explain its purpose
  • Compare MSE vs cross-entropy loss for different problems
  • Trace a forward pass with actual numbers

2. 📖 Core Content

3.1 Forward Propagation

Forward propagation computes the output of a neural network for a given input. For a 2-layer MLP:
z1=W1x+b1z_1 = W_1 x + b_1 a1=ReLU(z1)a_1 = \text{ReLU}(z_1) z2=W2a1+b2z_2 = W_2 a_1 + b_2 y^=Softmax(z2)\hat{y} = \text{Softmax}(z_2)

3.2 Softmax Function

Softmax(zi)=ezi∑jezj\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}
Softmax converts logits to probabilities that sum to 1. Used for multi-class classification. Example: Logits = [2.0, 1.0, 0.1]
Softmax(2.0)=e2/(e2+e1+e0.1)=7.389/(7.389+2.718+1.105)=7.389/11.212=0.659\text{Softmax}(2.0) = e^2/(e^2+e^1+e^{0.1}) = 7.389/(7.389+2.718+1.105) = 7.389/11.212 = 0.659 Softmax(1.0)=2.718/11.212=0.242\text{Softmax}(1.0) = 2.718/11.212 = 0.242 Softmax(0.1)=1.105/11.212=0.099\text{Softmax}(0.1) = 1.105/11.212 = 0.099
Output: [0.659, 0.242, 0.099] — class 0 is most likely.

3.3 Loss Functions

Mean Squared Error (MSE): For regression:
L=1N∑i=1N(yi−y^i)2L = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2
Cross-Entropy Loss: For classification:
L=−∑i=1Nyilog⁡(y^i)L = -\sum_{i=1}^N y_i \log(\hat{y}_i)
For binary classification:
L=−[ylog⁡(y^)+(1−y)log⁡(1−y^)]L = -[y\log(\hat{y}) + (1-y)\log(1-\hat{y})]

3.4 Why Cross-Entropy for Classification?

  • MSE penalizes all errors equally (probability 0.7 vs 0.8)
  • Cross-entropy penalizes confident wrong predictions heavily
  • Cross-entropy + softmax gradient is simpler: y^−y\hat{y} - y

4. 📝 Practice Questions

Q1: Compute cross-entropy loss for y=[1,0,0], ŷ=[0.7,0.2,0.1].
Answer: L = -[1·log(0.7) + 0·log(0.2) + 0·log(0.1)] = -log(0.7) ≈ 0.357. If prediction were [0.3,0.3,0.4]: L = -log(0.3) ≈ 1.204 (higher = worse). Join Discord PreviousPerceptron to MLPNextBackpropagation
Document outline

Keep your place and jump directly to a heading.

Table of Contents
System Normal // Awaiting Context

Intelligence Hub

Navigate the knowledge graph to generate context. The Hub adapts dynamically to surface backlinks, related notes, and metadata insights.

Open library →Launch Assessment Engine