⚙️ Optimization Methods
292 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# ⚙️ Optimization Methods ## 1. 🎯 Learning Objectives - Compare batch, mini-batch, and stochastic gradient descent - Trace Momentum update with velocity - Explain Adam: adaptive learning rates + momentum - Choose appropriate optimizer for a given problem ## 2.
⚙️ Optimization Methods
1. 🎯 Learning Objectives
- Compare batch, mini-batch, and stochastic gradient descent
- Trace Momentum update with velocity
- Explain Adam: adaptive learning rates + momentum
- Choose appropriate optimizer for a given problem
2. 📖 Core Content
3.1 Gradient Descent Variants
Batch GD: — uses ALL data per step Stochastic GD: — uses ONE example per step Mini-batch GD: — uses BATCH of examples
| Variant | Per-step Cost | Convergence | Memory |
|---|---|---|---|
| Batch GD | O(N) | Stable, smooth | O(1) |
| SGD | O(1) | Noisy, can escape local minima | O(1) |
| Mini-batch | O(batch_size) | Trade-off | O(batch_size) |
3.2 Momentum
Momentum accelerates gradient descent by accumulating velocity:
- : momentum coefficient (typically 0.9)
- Helps overcome local minima and plateaus
3.3 RMSprop
Adaptive learning rate per parameter:
3.4 Adam (Adaptive Moment Estimation)
Combines Momentum + RMSprop:
Default:
Adam is the default optimizer for most deep learning tasks.
4. 📝 Practice Questions
Q1: With momentum γ=0.9 and learning rate η=0.01, if gradient g=[1, -2], compute the update step (assuming initial velocity 0).Answer: v₁ = 0.9×0 + 0.01×[1,-2] = [0.01, -0.02]. θ_new = θ - [0.01, -0.02]. Next step: v₂ = 0.9×[0.01,-0.02] + 0.01×g₂. Join Discord PreviousBackpropagationNextRegularization