Gradient Descent and Optimization
105 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Gradient Descent and Optimization ## 4.1 Gradient Descent $$ \boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} - \eta \nabla L(\boldsymbol{\beta}^{(t)}) $$ ## 4.2 Stochastic Gradient Descent Update using a single observation at a time — much faster for large datasets. ## 4.3 Newton-Raphson $$ \boldsymbol{\beta...
Gradient Descent and Optimization
4.1 Gradient Descent
pythonimport numpy as np def gradient_descent(X, y, lr=0.01, epochs=1000): n, p = X.shape beta = np.zeros(p) for epoch in range(epochs): grad = -2/n * X.T @ (y - X @ beta) beta -= lr * grad return beta
4.2 Stochastic Gradient Descent
Update using a single observation at a time — much faster for large datasets.
4.3 Newton-Raphson
Where is the Hessian matrix. Faster convergence but per iteration.
Join Discord
PreviousCross-ValidationNextSGD Variants