TAM is designed to capture complex temporal relationships both  efficiently and  flexibly,
It adopts an adaptive kernel instead of self-attention to capture  global contextual information, with lower time complexity 
than GLTR.

TAM has two branches, a local branch and a global branch. Given the input feature map $X\in \mathbb{R}^{C\times T\times H\times W}$,  global spatial average pooling $\text{GAP}$ is first applied to the feature map to ensure TAM has a low computational cost. Then the local branch in TAM employs several 1D convolutions with ReLU nonlinearity across the temporal domain to produce location-sensitive importance maps for enhancing frame-wise features.
The local branch can be written as
\begin{align}
    s &= \sigma(\text{Conv1D}(\delta(\text{Conv1D}(\text{GAP}(X)))))
\end{align}
\begin{align}
    X^1 &= s X
\end{align}
Unlike the local branch, the global branch is location invariant and focuses on generating a channel-wise adaptive kernel based on global temporal information in each channel. For the $c$-th channel, the  kernel can be written as

\begin{align}
    \Theta_c = \text{Softmax}(\text{FC}_2(\delta(\text{FC}_1(\text{GAP}(X)_c)))) 
\end{align}

where $\Theta_c \in \mathbb{R}^{K}$ and $K$ is the adaptive kernel size. Finally, TAM  convolves the adaptive kernel $\Theta$ with $ X_\text{out}^1$:
\begin{align}
    Y = \Theta \otimes  X^1
\end{align}

With the help of the local branch and global branch,
TAM can capture the complex temporal structures in video and 
enhance per-frame features at low computational cost.
Due to its flexibility and lightweight design,
TAM can be added to any existing 2D CNNs.

TAM: Temporal Adaptive Module for Video Recognition

**Embeddings from Language Models**, or **ELMo**, is a type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus.

A biLM combines both a forward and backward LM.  ELMo jointly maximizes the log likelihood of the forward and backward directions. To add ELMo to a supervised model, we freeze the weights of the biLM and then concatenate the ELMo vector $\textbf{ELMO}^{task}_k$ with $\textbf{x}_k$ and pass the ELMO enhanced representation $[\textbf{x}_k; \textbf{ELMO}^{task}_k]$ into the task RNN. Here $\textbf{x}_k$ is a context-independent token representation for each token position. 

Image Source: [here](https://medium.com/@duyanhnguyen_38925/create-a-strong-text-classification-with-the-help-from-elmo-e90809ba29da)

ELMo

Deep contextualized word representations

IGSA

GSANet: Semantic Segmentation with Global and Selective Attention

**Hamburger** is a global context module that employs matrix decomposition to factorize the learned representation into sub-matrices so as to recover the clean low-rank signal subspace. The key idea is, if we formulate the inductive bias like the global context into an objective function, the optimization algorithm to minimize the objective function can construct a computational graph, i.e., the architecture we need in the networks.

Hamburger

Is Attention Better Than Matrix Decomposition?

**CS-GAN** is a type of generative adversarial network that uses a form of deep compressed sensing, and [latent optimisation](https://paperswithcode.com/method/latent-optimisation), to improve the quality of generated samples.

CS-GAN

Deep Compressed Sensing

TuckER model trained with a relation prediction objective on top of the 1vsAll loss

TuckER-RP

Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations

**Inception-B** is an image model block used in the [Inception-v4](https://paperswithcode.com/method/inception-v4) architecture.

Inception-B

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Inspired by the spatial propagation mechanism utilized in the depth completion task \cite{NLSPN}, we introduce a normal incorporated non-local disparity propagation module in which we hub NDP to generate non-local affinities and offsets for spatial propagation at the disparity level. The motivation lies that the sampled pixels for edges and occluded regions are supposed to be selected. The propagation process aggregates disparities via plane affinity relations, which alleviates the phenomenon of disparity blurring at object edges due to frontal parallel windows. And the disparities in occluded areas are also optimized at the same time by being propagated from non-occluded areas where the predicted disparities are with high confidence.

Spatial Propagation

Digging Into Normal Incorporated Stereo Matching

**Generative Adversarial Imitation Learning** presents a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning.

GAIL

Generative Adversarial Imitation Learning

**Visformer**, or **Vision-friendly Transformer**, is an architecture that combines [Transformer](https://paperswithcode.com/methods/category/transformers)-based architectural features with those from [convolutional neural network](https://paperswithcode.com/methods/category/convolutional-neural-networks) architectures. Visformer adopts the stage-wise design for higher base performance. But [self-attentions](https://paperswithcode.com/method/multi-head-attention) are only utilized in the last two stages, considering that self-attention in the high-resolution stage is relatively inefficient even when the FLOPs are balanced. Visformer employs [bottleneck blocks](https://paperswithcode.com/method/bottleneck-residual-block) in the first stage and utilizes [group 3 × 3 convolutions](https://paperswithcode.com/method/grouped-convolution) in bottleneck blocks inspired by [ResNeXt](https://paperswithcode.com/method/resnext). It also introduces [BatchNorm](https://paperswithcode.com/method/batch-normalization) to patch embedding modules as in CNNs.

Visformer

Visformer: The Vision-friendly Transformer

**SpineNet** is a convolutional neural network backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by [Neural Architecture Search](https://paperswithcode.com/method/neural-architecture-search).

SpineNet

SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization

**Feedback Memory** is a type of attention module used in the [Feedback Transformer](https://paperswithcode.com/method/feedback-transformer) architecture. It allows a [transformer](https://paperswithcode.com/method/transformer) to to use the most abstract representations from the past directly as inputs for the current timestep. This means that the model does not form its representation in parallel, but sequentially token by token. More precisely, we replace the context inputs to attention modules with memory vectors that are computed over the past, i.e.:

$$ \mathbf{z}^{l}\_{t} = \text{Attn}\left(\mathbf{x}^{l}\_{t},  \left[\mathbf{m}\_{t−\tau}, \dots, \mathbf{m}\_{t−1}\right]\right) $$

where a memory vector $\mathbf{m}\_{t}$ is computed by summing the representations of each layer at the $t$-th time step:

$$ \mathbf{m}\_{t} = \sum^{L}\_{l=0}\text{Softmax}\left(w^{l}\right)\mathbf{x}\_{t}^{l} $$

where $w^{l}$ are learnable scalar parameters. Here $l = 0$ corresponds to token embeddings. The weighting of different layers by a [softmax](https://paperswithcode.com/method/softmax) output gives the model more flexibility as it can average them or select one of them. This modification of the self-attention input adapts the computation of the Transformer from parallel to sequential, summarized in the Figure. Indeed, it gives the ability to formulate the representation $\mathbf{x}^{l}\_{t+1}$ based on past representations from any layer $l'$, while in a standard Transformer this is only true for $l > l'$. This change can be viewed as exposing all previous computations to all future computations, providing better representations of the input. Such capacity would allow much shallower models to capture the same level of abstraction as a deeper architecture.

Feedback Memory

Addressing Some Limitations of Transformers with Feedback Memory

**Content-based attention** is an attention mechanism based on cosine similarity:

$$f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) = \cos\left[\textbf{h}\_{i};\textbf{s}\_{j}\right] $$

It was utilised in [Neural Turing Machines](https://paperswithcode.com/method/neural-turing-machine) as part of the Addressing Mechanism.

We produce a normalized attention weighting by taking a [softmax](https://paperswithcode.com/method/softmax) over these attention alignment scores.

Content-based Attention

Neural Turing Machines

**ShakeDrop regularization** extends [Shake-Shake regularization](https://paperswithcode.com/method/shake-shake-regularization) and can be applied not only to [ResNeXt](https://paperswithcode.com/method/resnext) but also [ResNet](https://paperswithcode.com/method/resnet), [WideResNet](https://paperswithcode.com/method/wideresnet), and [PyramidNet](https://paperswithcode.com/method/pyramidnet). The proposed ShakeDrop is given as

$$G\left(x\right) = x + \left(b\_{l} + \alpha − b\_{l}\alpha\right)F\left(x\right), \text{ in train-fwd} $$
$$G\left(x\right) = x + \left(b\_{l} + \beta − b\_{l}\beta\right)F\left(x\right), \text{ in train-bwd} $$
$$G\left(x\right) = x + E\left[b\_{l} + \alpha − b\_{l}\alpha\right]F\left(x\right), \text{ in test} $$

where $b\_{l}$ is a Bernoulli random variable with probability $P\left(b\_{l} = 1\right) = E\left[b\_{l}
\right] = p\_{l}$ given by the linear decay rule in each layer, and $\alpha$ and $\beta$ are independent uniform random variables in each element. 

The most effective ranges of $\alpha$ and $\beta$ were experimentally found to be different from those of Shake-Shake, and are $\alpha$ = 0, $\beta \in \left[0, 1\right]$ and $\alpha \in \left[−1, 1\right]$, $\beta \in \left[0, 1\right]$.

ShakeDrop

ShakeDrop Regularization for Deep Residual Learning

A **Hermite Activations** is a type of activation function which uses a smooth finite Hermite polynomial base as a substitute for non-smooth [ReLUs](https://paperswithcode.com/method/relu). 

Relevant Paper: [Lokhande et al](https://arxiv.org/pdf/1909.05479.pdf)

Hermite Activation

**Linear Warmup With Cosine Annealing** is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule afterwards.

Linear Warmup With Cosine Annealing

A GNN for dags, which injects their topological order as an inductive bias via asynchronous message passing.

DAGNN

Directed Acyclic Graph Neural Networks

**SERLU**, or **Scaled Exponentially-Regularized Linear Unit**, is a type of activation function. The new function introduces a bump-shaped function in the region of negative input. The bump-shaped function has approximately zero response to large negative input while being able to push the output of SERLU towards zero mean statistically.

$$ \text{SERLU}\left(x\right)) = \lambda\_{serlu}x \text{ if } x \geq 0 $$
$$ \text{SERLU}\left(x\right)) = \lambda\_{serlu}\alpha\_{serlu}xe^{x} \text{ if } x < 0 $$

where the two parameters $\lambda\_{serlu} > 0$ and $\alpha\_{serlu} > 0$ remain to be specified.

AI Glossary

Temporal Adaptive Module

ELMo

Improved Gravitational Search algorithm

Hamburger

CS-GAN

TuckER with Relation Prediction

Inception-B

Surface Nomral-based Spatial Propagation