momentum

Momentum

Momentum is a popular heuristic used along with SGD in place of full second order optimization: It updates weights based on an exponentially weighted moving average of the past gradients.
This helps to smooth out the updates/approximate curvature to find a better path towards the minimum. Essentially: If past updates all point in the same direction, we’re more confident of taking large steps in that direction.
The update rule for gradient dececnt with momentum is:
TODO
$\alpha >0,\mu \in [0,1]$

(reminiscent of the effect of Batch gradient descent - Mini-batch gradient descent)

redo the equations after reading the nesterov paper… NOTE: $v$ likely stands for velocity!

$$\begin{array}{rlr}{g}_t& \leftarrow \lambda g_{t-1}+\lambda \nabla L({\theta }_t)& \text{weight decay }\lambda \text{, learning rate }\alpha \\ g_t& \leftarrow \mu g_{t-1}+(1-\tau )g_t& \text{momentum }\mu \text{, friction/dampening }\tau \\ {\theta }_t& \leftarrow {\theta }_{t-1}-g_t& \end{array}$$

got confused with (https://pytorch.org/docs/stable/generated/torch.optim.SGD.html) which imlpements differently (see note)

References

On the importance of initialization and momentum in deep learning (nesterov)

Adam