Randomized Optimization
117 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Randomized Optimization ## 10.1 Stochastic Gradient Descent $$ \mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta_t \nabla \ell(\mathbf{w}^{(t)}; x_{i_t}, y_{i_t}) $$ Convergence: $E[||\nabla f(\mathbf{w}_t)||^2] \leq O(1/\sqrt{t})$ ## 10.2 SVRG (Stochastic Variance Reduced Gradient) Maintains a snapshot of the full gra...
Randomized Optimization
10.1 Stochastic Gradient Descent
Convergence:
pythonimport numpy as np def sgd(X, y, lr=0.1, epochs=100): n, p = X.shape w = np.zeros(p) for t in range(epochs * n): idx = np.random.randint(n) grad = 2 * X[idx] * (X[idx] @ w - y[idx]) w -= lr * grad / np.sqrt(t + 1) return w
10.2 SVRG (Stochastic Variance Reduced Gradient)
Maintains a snapshot of the full gradient periodically, reducing variance at each step.
10.3 Convergence Comparison
| Method | Convergence Rate | Per-Iteration Cost |
|---|---|---|
| Full GD | ||
| SGD | ||
| SVRG |