**Stochastic Weight Averaging** is an optimization procedure that averages multiple points along the trajectory of [SGD](https://paperswithcode.com/method/sgd), with a cyclical or constant learning rate. On the one hand it averages weights, but it also has the property that, with a cyclical or constant learning rate, SGD proposals are approximately sampling from the loss surface of the network, leading to stochastic weights and helping to discover broader optima.

**RMSProp** is an unpublished adaptive learning rate optimizer [proposed by Geoff Hinton](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:

$$E\left[g^{2}\right]\_{t} = \gamma E\left[g^{2}\right]\_{t-1} + \left(1 - \gamma\right) g^{2}\_{t}$$

$$\theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]\_{t} + \epsilon}}g\_{t}$$

Hinton suggests $\gamma=0.9$, with a good default for $\eta$ as $0.001$.

Image: [Alec Radford](https://twitter.com/alecrad)

RMSProp

Stochastic Weight Averaging

Averaging Weights Leads to Wider Optima and Better Generalization

**Pythia** is a suite of decoder-only autoregressive language models all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. The model architecture and hyperparameters largely follow GPT-3, with a few notable deviations based on recent advances in best practices for large scale language modeling.

Source	Averaging Weights Leads to Wider Optima and Better Generalization
Year	2000
Data Source	Papers with Code archive (CC BY-SA)