⚙️ Optimizers Comparison
136 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# ⚙️ Optimizers Comparison ## 1. 🎯 Learning Objectives - Compare update rules of SGD, Momentum, RMSprop, Adam - Choose appropriate optimizer and learning rate - Diagnose optimization issues ## 2.
⚙️ Optimizers Comparison
1. 🎯 Learning Objectives
- Compare update rules of SGD, Momentum, RMSprop, Adam
- Choose appropriate optimizer and learning rate
- Diagnose optimization issues
2. 📖 Core Content
3.1 Update Rules Summary
| Optimizer | Update Rule | Key Hyperparameters |
|---|---|---|
| SGD | θ = θ - η·g | η (learning rate) |
| SGD+Momentum | v=γv+ηg; θ=θ-v | η, γ=0.9 |
| NAG | v=γv+η·∇f(θ-γv); θ=θ-v | η, γ=0.9 |
| AdaGrad | θ=θ-(η/√(G+ε))·g | η=0.01, ε=1e-8 |
| RMSprop | θ=θ-(η/√(v+ε))·g; v=βv+(1-β)g² | η=0.001, β=0.9 |
| Adam | m=β₁m+(1-β₁)g; v=β₂v+(1-β₂)g²; θ=θ-η·m̂/√(v̂+ε) | η=0.001, β₁=0.9, β₂=0.999 |
3.2 When to Use Each
| Scenario | Recommended Optimizer | Why |
|---|---|---|
| Simple convex problem | SGD | Works well, fewer hyperparameters |
| Deep neural network | Adam | Adaptive LR, works out-of-box |
| Computer vision | SGD+Momentum | Often generalizes better than Adam |
| NLP tasks | Adam | Handles sparse gradients well |
| Learning rate tuning | Adam | Less sensitive to LR choice |
| Sparse features | AdaGrad | Per-parameter adaptive LR |
3.3 Learning Rate Scheduling
Step decay: Reduce LR by factor γ every k epochs Cosine annealing: τ_t = τ_min + 0.5(τ_max-τ_min)(1+cos(tπ/T)) Warmup: Gradually increase LR from 0 to target over first few epochs
Join Discord
PreviousTransformer Deep DiveNextBatch Norm Details