AugMix mixes augmented images through linear interpolations. Consequently it is like [Mixup](https://paperswithcode.com/method/mixup) but instead mixes augmented versions of the same image.

**Additive Attention**, also known as **Bahdanau Attention**, uses a one-hidden layer feed-forward network to calculate the attention alignment score:

$$f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) = v\_{a}^{T}\tanh\left(\textbf{W}\_{a}\left[\textbf{h}\_{i};\textbf{s}\_{j}\right]\right)$$

where $\textbf{v}\_{a}$ and $\textbf{W}\_{a}$ are learned attention parameters. Here $\textbf{h}$ refers to the hidden states for the encoder, and $\textbf{s}$ is the hidden states for the decoder. The function above is thus a type of alignment score function. We can use a matrix of alignment scores to show the correlation between source and target words, as the Figure to the right shows.

Within a neural network, once we have the alignment scores, we calculate the final scores using a [softmax](https://paperswithcode.com/method/softmax) function of these alignment scores (ensuring it sums to 1).

Additive Attention

Neural Machine Translation by Jointly Learning to Align and Translate

AugMix

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

**Activation Regularization (AR)**, or $L\_{2}$ activation regularization, is regularization performed on activations as opposed to weights. It is usually used in conjunction with [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks). It is defined as:

$$\alpha{L}\_{2}\left(m\circ{h\_{t}}\right) $$

where $m$ is a [dropout](https://paperswithcode.com/method/dropout) mask used by later parts of the model, $L\_{2}$ is the $L\_{2}$ norm, and $h_{t}$ is the output of an RNN at timestep $t$, and $\alpha$ is a scaling coefficient. 

When applied to the output of a dense layer, AR penalizes activations that are substantially away from 0, encouraging activations to remain small.

Source	AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
Year	2000
Data Source	Papers with Code archive (CC BY-SA)