Neural Sync Active

Gradient Descent and Optimization

Registry Synced

Gradient Descent and Optimization

105 words

1 min read

Reading compass

Now · 4.1 Gradient Descent

Gradient Descent and Optimization

4.1 Gradient Descent

\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} - \eta \nabla L(\boldsymbol{\beta}^{(t)})


python
import numpy as np
def gradient_descent(X, y, lr=0.01, epochs=1000):
    n, p = X.shape
    beta = np.zeros(p)
    for epoch in range(epochs):
        grad = -2/n * X.T @ (y - X @ beta)
        beta -= lr * grad
    return beta

4.2 Stochastic Gradient Descent

Update using a single observation at a time — much faster for large datasets.

4.3 Newton-Raphson

\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} - \mathbf{H}^{-1} \nabla L(\boldsymbol{\beta}^{(t)})

Where

\mathbf{H}

is the Hessian matrix. Faster convergence but

O(p^3)

per iteration. Join Discord PreviousCross-ValidationNextSGD Variants

Gradient Descent and Optimization

Gradient Descent and Optimization

4.1 Gradient Descent

4.2 Stochastic Gradient Descent

4.3 Newton-Raphson

Intelligence Hub