Neural Sync Active
⚙️ Optimization Methods
Registry Synced
⚙️ Optimization Methods
292 words
1 min read
Reading compass
Now · 1. 🎯 Learning Objectives
⚙️ Optimization Methods
1. 🎯 Learning Objectives
- Compare batch, mini-batch, and stochastic gradient descent
- Trace Momentum update with velocity
- Explain Adam: adaptive learning rates + momentum
- Choose appropriate optimizer for a given problem
2. 📖 Core Content
3.1 Gradient Descent Variants
Batch GD: θ=θ−η⋅∇θJ(θ) — uses ALL data per step Stochastic GD: θ=θ−η⋅∇θJ(θ(i)) — uses ONE example per step Mini-batch GD: θ=θ−η⋅∇θJ(θ(i:i+n)) — uses BATCH of examples
| Variant | Per-step Cost | Convergence | Memory |
|---|---|---|---|
| Batch GD | O(N) | Stable, smooth | O(1) |
| SGD | O(1) | Noisy, can escape local minima | O(1) |
| Mini-batch | O(batch_size) | Trade-off | O(batch_size) |
3.2 Momentum
Momentum accelerates gradient descent by accumulating velocity:
- γ: momentum coefficient (typically 0.9)
- Helps overcome local minima and plateaus
3.3 RMSprop
Adaptive learning rate per parameter:
3.4 Adam (Adaptive Moment Estimation)
Combines Momentum + RMSprop:
Default: β1=0.9,β2=0.999,ϵ=10−8
Adam is the default optimizer for most deep learning tasks.
4. 📝 Practice Questions
Q1: With momentum γ=0.9 and learning rate η=0.01, if gradient g=[1, -2], compute the update step (assuming initial velocity 0).Answer: v₁ = 0.9×0 + 0.01×[1,-2] = [0.01, -0.02]. θ_new = θ - [0.01, -0.02]. Next step: v₂ = 0.9×[0.01,-0.02] + 0.01×g₂. Join Discord PreviousBackpropagationNextRegularization