**Quasi-Hyperbolic Momentum (QHM)** is a stochastic optimization technique that alters [momentum SGD](https://paperswithcode.com/method/sgd-with-momentum) with a momentum step, averaging an [SGD](https://paperswithcode.com/method/sgd) step with a momentum step:

$$ g\_{t+1} = \beta{g\_{t}} + \left(1-\beta\right)\cdot{\nabla}\hat{L}\_{t}\left(\theta\_{t}\right) $$
$$ \theta\_{t+1} = \theta\_{t} - \alpha\left[\left(1-v\right)\cdot\nabla\hat{L}\_{t}\left(\theta\_{t}\right) + v\cdot{g\_{t+1}}\right]$$

The authors suggest a rule of thumb of $v = 0.7$ and $\beta = 0.999$.

**DIME**, or **Distance to Modelled Embedding**, is a method for detecting out-of-distribution examples during prediction time. Given a trained neural network, the training data drawn from some high-dimensional distribution in data space $X$ is transformed into the model’s intermediate feature vector space $\mathbb{R}^{p}$. The training set embedding is linearly approximated as a hyperplane. When we then receive new observations it is difficult to assess if observations are out-of-distribution directly in data space, so we transform them into the same intermediate feature space. Finally, the Distance-to-Modelled-Embedding (DIME) can be used to assess whether new observations fit into the expected embedding covariance structure.

DIME

Out-of-Distribution Example Detection in Deep Neural Networks using Distance to Modelled Embedding

Quasi-hyperbolic momentum and Adam for deep learning

The **Swin Transformer** is a type of [Vision Transformer](https://paperswithcode.com/method/vision-transformer). It builds hierarchical feature maps by merging image patches (shown in gray) in deeper layers and has linear computation complexity to input image size due to computation of self-attention only within each local window (shown in red). It can thus serve as a general-purpose backbone for both image classification and dense recognition tasks. In contrast, previous vision Transformers produce feature maps of a single low resolution and have quadratic computation complexity to input image size due to computation of self-attention globally.

Source	Quasi-hyperbolic momentum and Adam for deep learning
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: QHM?

Viet-Anh on Software