Neural Sync Active

Gradient Descent Variants

Registry Synced

Gradient Descent Variants

504 words

3 min read

Reading compass

Now · 🎯 Learning Objectives

Gradient Descent Variants

🎯 Learning Objectives

Compare batch, stochastic, and mini-batch gradient descent
Understand learning rate schedules and momentum
Implement SGD with sklearn's SGDRegressor/SGDClassifier
Diagnose convergence issues and fix them

📖 Core Content

13.1 Intuition: Choosing How to Walk Downhill

Batch gradient descent uses ALL data to compute each step — it's like carefully surveying the entire mountain before taking a single step. Stochastic gradient descent uses ONE random point per step — like running down the mountain, taking quick steps based on immediate feedback. Mini-batch uses a small random batch — like taking 10-100 quick measurements per step. (Diagram)

13.2 Comparison

Variant	Update per step	Speed per step	Convergence	Memory
Batch GD	All data	Slow	Smooth guaranteed	High
SGD	1 example	Fast	Noisy, oscillates	Low
Mini-Batch	b examples (32-256)	Moderate	Smooth, fast	Moderate

13.3 Learning Rate Schedules

The learning rate doesn't need to be constant. Common schedules:

Step decay: Reduce α by factor every k steps
Exponential decay: $\alpha_t = \alpha_0 e^{-kt}$
1/t decay: $\alpha_t = \alpha_0 / (1 + kt)$


python
# runnable
from sklearn.linear_model import SGDClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np
X, y = make_classification(n_samples=10000, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# SGD with adaptive learning rate
sgd = SGDClassifier(
    loss='log_loss',
    learning_rate='adaptive',  # Decrease when validation loss plateaus
    eta0=0.01,
    max_iter=1000,
    early_stopping=True,
    random_state=42
)
sgd.fit(X_train, y_train)
print(f"SGD test accuracy: {sgd.score(X_test, y_test):.3f}")

13.4 Momentum

Momentum accelerates gradient descent by adding a fraction of the previous update to the current one:

v_t = \gamma v_{t-1} + \alpha \nabla J(\theta_t)

\theta_{t+1} = \theta_t - v_t

Where

\gamma

(typically 0.9) controls momentum strength. This helps escape local minima and speeds up convergence in shallow gradient regions.

📝 Practice Questions

Q1: With m=1,000,000, which GD variant is most practical?

Mini-batch GD with batch size 32-256. Batch GD would be too slow (processing all 1M points per step). SGD would be too noisy. Mini-batch balances speed and stability, and can leverage GPU parallelization. Q2: Why does SGD oscillate near the minimum?

Each step uses only one random example. The gradient is a noisy estimate of the true gradient. Near the minimum, the noise causes the path to bounce around rather than settling precisely. Solution: use learning rate decay or switch to mini-batch. Q3: What does early stopping do?

Monitors validation loss during training. If validation loss hasn't improved for n_iter_no_change iterations, stop training. This prevents overfitting and saves computation. It's a form of regularization.
python
sgd = SGDClassifier(early_stopping=True, n_iter_no_change=5, validation_fraction=0.1)
Q4: How does momentum help escape local minima?

Momentum accumulates velocity in the direction of consistent gradient updates. If the gradient changes sign (entering a local minimum basin), the accumulated momentum carries it past small local minima. This is like a ball rolling past small bumps. Join Discord PreviousLinear RegressionNextMultiple & Polynomial Regression