$v_{t+1} = \beta v_t + \nabla L(\theta_t)$ $\theta_{t+1} = \theta_t - \eta v_{t+1}$

Combines momentum + adaptive learning rates. Default: $\beta_1 = 0.9, \beta_2 = 0.999$. Join Discord PreviousGradient DescentNextEM Algorithm