Gradient Descent Variants

🎯 Learning Objectives

• Compare batch, stochastic, and mini-batch gradient descent
• Understand learning rate schedules and momentum
• Implement SGD with sklearn's SGDRegressor/SGDClassifier
• Diagnose convergence issues and fix them

📖 Core Content

13.1 Intuition: Choosing How to Walk Downhill

13.2 Comparison

13.3 Learning Rate Schedules

• Step decay: Reduce α by factor every k steps
• Exponential decay: $\alpha_t = \alpha_0 e^{-kt}$
• 1/t decay: $\alpha_t = \alpha_0 / (1 + kt)$

python
Run Inline
# runnable
from sklearn.linear_model import SGDClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np
X, y = make_classification(n_samples=10000, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# SGD with adaptive learning rate
sgd = SGDClassifier(
    loss='log_loss',
    learning_rate='adaptive',  # Decrease when validation loss plateaus
    eta0=0.01,
    max_iter=1000,
    early_stopping=True,
    random_state=42
)
sgd.fit(X_train, y_train)
print(f"SGD test accuracy: {sgd.score(X_test, y_test):.3f}")

13.4 Momentum

📝 Practice Questions

Q1: With m=1,000,000, which GD variant is most practical?

Mini-batch GD with batch size 32-256. Batch GD would be too slow (processing all 1M points per step). SGD would be too noisy. Mini-batch balances speed and stability, and can leverage GPU parallelization. Q2: Why does SGD oscillate near the minimum?

Each step uses only one random example. The gradient is a noisy estimate of the true gradient. Near the minimum, the noise causes the path to bounce around rather than settling precisely. Solution: use learning rate decay or switch to mini-batch. Q3: What does early stopping do?

Monitors validation loss during training. If validation loss hasn't improved for n_iter_no_change iterations, stop training. This prevents overfitting and saves computation. It's a form of regularization.

python
Run Inline
sgd = SGDClassifier(early_stopping=True, n_iter_no_change=5, validation_fraction=0.1)

Q4: How does momentum help escape local minima?

Momentum accumulates velocity in the direction of consistent gradient updates. If the gradient changes sign (entering a local minimum basin), the accumulated momentum carries it past small local minima. This is like a ball rolling past small bumps. Join Discord PreviousLinear RegressionNextMultiple & Polynomial Regression