**Recurrent Dropout** is a regularization method for [recurrent neural networks](https://paperswithcode.com/methods/category/recurrent-neural-networks). [Dropout](https://paperswithcode.com/method/dropout) is applied to the updates to [LSTM](https://paperswithcode.com/method/lstm) memory cells (or [GRU](https://paperswithcode.com/method/gru) states), i.e. it drops out the input/update gate in LSTM/GRU.

Recurrent Dropout

Recurrent Dropout without Memory Loss

**ShuffleNet V2 Block** is an image model block used in the [ShuffleNet V2](https://paperswithcode.com/method/shufflenet-v2) architecture, where speed is the metric optimized for (instead of indirect ones like FLOPs). It utilizes a simple operator called channel split. At the beginning of each unit, the input of $c$ feature channels are split into two branches with $c - c'$ and $c'$ channels, respectively. Following **G3**, one branch remains as identity. The other branch consists of three convolutions with the same input and output channels to satisfy **G1**. The two $1\times1$ convolutions are no longer group-wise, unlike the original [ShuffleNet](https://paperswithcode.com/method/shufflenet). This is partially to follow **G2**, and partially because the split operation already produces two groups. After [convolution](https://paperswithcode.com/method/convolution), the two branches are concatenated. So, the number of channels keeps the same (G1). The same “[channel shuffle](https://paperswithcode.com/method/channel-shuffle)” operation as in ShuffleNet is then used to enable information communication between the two branches.

The motivation behind channel split is that alternative architectures, where pointwise group convolutions and bottleneck structures are used, lead to increased memory access cost. Additionally more network fragmentation with group convolutions reduces parallelism (less friendly for GPU), and the element-wise addition operation, while they have low FLOPs, have high memory access cost. Channel split is an alternative where we can maintain a large number of equally wide channels (equally wide minimizes memory access cost) without having dense convolutions or too many groups.

ShuffleNet V2 Block

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

**TSRUs**, or **Transformation-based Spatial Recurrent Unit p**, is a modification of a [ConvGRU](https://paperswithcode.com/method/cgru) used in the [TriVD-GAN](https://paperswithcode.com/method/trivd-gan) architecture for video generation.

It largely follows [TSRUc](https://paperswithcode.com/method/tsruc), but computes each intermediate output in a fully sequential manner: like in TSRUc, $c$ is given access to $\hat{h}\_{t-1}$, but additionally, $u$ is given access to both outputs $\hat{h}\_{t-1}$ and $c$, so as to make an informed decision prior to mixing. This yields the following replacement for $u$:

$$ u = \sigma\left(W\_{u} \star\_{n}\left[\hat{h}\_{t-1};c\right] + b\_{u} \right) $$

In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and [ReLU](https://paperswithcode.com/method/relu) functions respectively and the $\star\_{n}$ represents a [convolution](https://paperswithcode.com/method/convolution) with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUs

Transformation-based Adversarial Video Prediction on Large-Scale Data

**Dual Softmax Loss** is a loss function based on symmetric cross-entropy loss used in the [CAMoE](https://paperswithcode.com/method/camoe) video-text retrieval model. Every text and video are calculated the
similarity with other videos or texts, which should be maximum in terms of the ground truth pair. For DSL, a prior is introduced to revise the similarity score. Multiplying the prior with the original similarity matrix imposes an efficient constraint and can help to filter those single side match pairs. As a result, DSL highlights the one with both great Text-to-Video and Video-to-Text probability, conducting a more convincing result.

Dual Softmax Loss

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

**CutMix** is an image data augmentation strategy. Instead of simply removing pixels as in [Cutout](https://paperswithcode.com/method/cutout), we replace the removed regions with a patch from another image. The ground truth labels are also mixed proportionally to the number of pixels of combined images. The added patches further enhance localization ability by requiring the model to identify the object from a partial view.

CutMix

CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features

**Field Embedded Factorization Machine**, or **FEFM**, is a factorization machine variant. For each field pair, FEFM introduces symmetric matrix embeddings along with the usual feature vector embeddings that are present in FM. Like FM, $v\_{i}$ is the vector embedding of the $i^{t h}$ feature. However, unlike Field-Aware Factorization Machines (FFMs), FEFM doesn't explicitly learn field-specific feature embeddings. The learnable symmetric matrix $W\_{F(i), F(j)}$ is the embedding for the field pair $F(i)$ and $F(j) .$ The interaction between the $i^{t h}$ feature and the $j^{t h}$ feature is mediated through $W_{F(i), F(j)} .$

$$
\phi(\theta, x)=\phi\_{F E F M}((w, v, W), x)=w\_{0}+\sum\_{i=1}^{m} w_{i} x_{i}+\sum\_{i=1}^{m} \sum\_{j=i+1}^{m} v\_{i}^{T} W\_{F(i), F(j)} v\_{j} x\_{i} x\_{j}
$$

where $W\_{F(i), F(j)}$ is a $k \times k$ symmetric matrix ( $k$ is the dimension of the feature vector embedding space containing feature vectors $v\_{i}$ and $v\_{j}$ ).

The symmetric property of the learnable matrix $W\_{F(i), F(j)}$ is ensured by reparameterizing $W\_{F(i), F(j)}$ as $U\_{F(i), F(j)}+$ $U\_{F(i), F(j)}^{T}$, where $U\_{F(i), F(j)}^{T}$ is the transpose of the learnable matrix $U\_{F(i), F(j)} .$ Note that $W_{F(i), F(j)}$ can also be interpreted as a vector transformation matrix which transforms a feature embedding when interacting with a specific field.

FEFM

Field-Embedded Factorization Machines for Click-through rate prediction

McKernel introduces a framework to use kernel approximates in the mini-batch setting with Stochastic Gradient Descent ([SGD](https://paperswithcode.com/method/sgd)) as an alternative to Deep Learning.

The core library was developed in 2014 as integral part of a thesis of Master of Science [1,2] at Carnegie Mellon and City University of Hong Kong. The original intend was to implement a speedup of Random Kitchen Sinks (Rahimi and Recht 2007) by writing a very efficient HADAMARD tranform, which was the main bottleneck of the construction. The code though was later expanded at ETH Zürich (in McKernel by Curtó et al. 2017) to propose a framework that could explain both Kernel Methods and Neural Networks. This manuscript and the corresponding theses, constitute one of the first usages (if not the first) in the literature of FOURIER features and Deep Learning; which later got a lot of research traction and interest in the community.

More information can be found in this presentation that the first author gave at ICLR 2020 [iclr2020_DeCurto](https://www.decurto.tw/c/iclr2020_DeCurto.pdf).

[1] [https://www.curto.hk/c/decurto.pdf](https://www.curto.hk/c/decurto.pdf)

[2] [https://www.zarza.hk/z/dezarza.pdf](https://www.zarza.hk/z/dezarza.pdf)

MCKERNEL

McKernel: A Library for Approximate Kernel Expansions in Log-linear Time

**ASGD Weight-Dropped LSTM**, or **AWD-LSTM**, is a type of recurrent neural network that employs [DropConnect](https://paperswithcode.com/method/dropconnect) for regularization, as well as [NT-ASGD](https://paperswithcode.com/method/nt-asgd) for optimization - non-monotonically triggered averaged [SGD](https://paperswithcode.com/method/sgd) - which returns an average of last iterations of weights. Additional regularization techniques employed include variable length backpropagation sequences, [variational dropout](https://paperswithcode.com/method/variational-dropout), [embedding dropout](https://paperswithcode.com/method/embedding-dropout), [weight tying](https://paperswithcode.com/method/weight-tying), independent embedding/hidden size, [activation regularization](https://paperswithcode.com/method/activation-regularization) and [temporal activation regularization](https://paperswithcode.com/method/temporal-activation-regularization).

AWD-LSTM

Regularizing and Optimizing LSTM Language Models

A **Siamese Network** consists of twin networks which accept distinct inputs but are joined by an energy function at the top. This function computes a metric between the highest level feature representation on each side. The parameters between the twin networks are tied. [Weight tying](https://paperswithcode.com/method/weight-tying) guarantees that two extremely similar images are not mapped by each network to very different locations in feature space because each network computes the same function. The network is symmetric, so that whenever we present two distinct images to the twin networks, the top conjoining layer will compute the same metric as if we were to we present the same two images but to the opposite twins.

Intuitively instead of trying to classify inputs, a siamese network learns to differentiate between inputs, learning their similarity. The loss function used is usually a form of contrastive loss.

Source: [Koch et al](https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf)

Siamese Network

As they carry great potential for modeling complex interactions, graph neural network (GNN)-based methods have been widely used to predict quantum mechanical properties of molecules. Most of the existing methods treat molecules as molecular graphs in which atoms are modeled as nodes. They characterize each atom's chemical environment by modeling its pairwise interactions with other atoms in the molecule. Although these methods achieve a great success, limited amount of works explicitly take many-body interactions, i.e., interactions between three and more atoms, into consideration. In this paper, we introduce a novel graph representation of molecules, heterogeneous molecular graph (HMG) in which nodes and edges are of various types, to model many-body interactions. HMGs have the potential to carry complex geometric information. To leverage the rich information stored in HMGs for chemical prediction problems, we build heterogeneous molecular graph neural networks (HMGNN) on the basis of a neural message passing scheme. HMGNN incorporates global molecule representations and an attention mechanism into the prediction process. The predictions of HMGNN are invariant to translation and rotation of atom coordinates, and permutation of atom indices. Our model achieves state-of-the-art performance in 9 out of 12 tasks on the QM9 dataset.

HMGNN

Heterogeneous Molecular Graph Neural Networks for Predicting Molecule Properties

**SNGAN**, or **Spectrally Normalised GAN**, is a type of generative adversarial network that uses [spectral normalization](https://paperswithcode.com/method/spectral-normalization), a type of [weight normalization](https://paperswithcode.com/method/weight-normalization), to stabilise the training of the discriminator.

SNGAN

Spectral Normalization for Generative Adversarial Networks

The Robust Loss is a generalization of the Cauchy/Lorentzian, Geman-McClure, Welsch/Leclerc, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and L2 loss functions. By introducing robustness as a continuous parameter, the loss function allows algorithms built around robust loss minimization to be generalized, which improves performance on basic vision tasks such as registration and clustering. Interpreting the loss as the negative log of a univariate density yields a general probability distribution that includes normal and Cauchy distributions as special cases. This probabilistic interpretation enables the training of neural networks in which the robustness of the loss automatically adapts itself during training, which improves performance on learning-based tasks such as generative image synthesis and unsupervised monocular depth estimation, without requiring any manual parameter tuning.

Adaptive Loss

A General and Adaptive Robust Loss Function

A **ResNest** is a variant on a [ResNet](https://paperswithcode.com/method/resnet), which instead stacks Split-Attention blocks. The cardinal group representations are then concatenated along the channel dimension: $V = \text{Concat}${$V^{1},V^{2},\cdots{V}^{K}$}. As in standard residual blocks, the final output $Y$ of otheur Split-Attention block is produced using a shortcut connection: $Y=V+X$, if the input and output feature-map share the same shape.  For blocks with a stride, an appropriate transformation $\mathcal{T}$ is applied to the shortcut connection to align the output shapes:  $Y=V+\mathcal{T}(X)$. For example, $\mathcal{T}$ can be strided [convolution](https://paperswithcode.com/method/convolution) or combined convolution-with-pooling.

ResNeSt

ResNeSt: Split-Attention Networks

**DVD-GAN DBlock** is a residual block for the discriminator used in the [DVD-GAN](https://paperswithcode.com/method/dvd-gan) architecture for video generation. Unlike regular [residual blocks](https://paperswithcode.com/method/residual-block), [3D convolutions](https://paperswithcode.com/method/3d-convolution) are employed due to the application to multiple frames in a video.

DVD-GAN DBlock

Adversarial Video Generation on Complex Datasets

The **Cosine Linear Unit**, or **CosLU**, is a type of activation function that has trainable parameters and uses the cosine function.

$$CosLU(x) = (x + \alpha \cos(\beta x))\sigma(x)$$

CosLU

Trainable Activations for Image Classification

FLAVA aims at building a single holistic universal model that targets all modalities at once. FLAVA is a language vision alignment model that learns strong representations from multimodal data (image-text pairs) and unimodal data (unpaired images and text). The model consists of an image encode transformer to capture unimodal image representations, a text encoder transformer to process unimodal text information, and a multimodal encode transformer that takes as input the encoded unimodal image and text and integrates their representations for multimodal reasoning. During pretraining, masked image modeling (MIM) and mask language modeling (MLM) losses are applied onto the image and text encoders over a single image or a text piece, respectively, while contrastive, masked multimodal modeling (MMM), and image-text matching (ITM) loss are used over paired image-text data. For downstream tasks, classification heads are applied on the outputs from the image, text, and multimodal encoders respectively for visual recognition, language understanding, and multimodal reasoning tasks It can be applied to broad scope of tasks from three domains (visual recognition, language understanding, and multimodal reasoning) under a common transformer model architecture.

FLAVA

FLAVA: A Foundational Language And Vision Alignment Model

An ECA block has similar formulation to an SE block including a squeeze module for aggregating global spatial information and an efficient excitation module for modeling cross-channel interaction. Instead of indirect correspondence, an ECA block only considers direct interaction between each channel and its k-nearest neighbors to control model complexity. Overall, the formulation of an ECA block is:
\begin{align}
    s = F_\text{eca}(X, \theta) & = \sigma (\text{Conv1D}(\text{GAP}(X))) 
\end{align}
\begin{align}
    Y & = s  X
\end{align}
where $\text{Conv1D}(\cdot)$ denotes 1D convolution with a kernel of shape $k$ across the channel domain, to model local cross-channel interaction. The parameter $k$ decides the coverage of interaction, and in ECA the kernel size $k$ is adaptively determined from the channel dimensionality $C$ instead of by manual tuning, using cross-validation:
\begin{equation}
    k = \psi(C) = \left | \frac{\log_2(C)}{\gamma}+\frac{b}{\gamma}\right |_\text{odd}
\end{equation}

where $\gamma$ and $b$ are hyperparameters. $|x|_\text{odd}$ indicates the nearest odd function of $x$. 

Compared to SENet, ECANet has an 
improved excitation module, and provides an efficient and effective block which can readily be 
 incorporated into various
CNNs.

ECANet

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

**DeepViT** is a type of [vision transformer](https://paperswithcode.com/method/vision-transformer) that replaces the self-attention layer within the [transformer](https://paperswithcode.com/method/transformer) block with a [Re-attention module](https://paperswithcode.com/method/re-attention-module) to address the issue of attention collapse and enables training deeper ViTs.

Viet-Anh on Software

AI Glossary

Tech Terminology

Recurrent Dropout

ShuffleNet V2 Block

TSRUs

Dual Softmax Loss

CutMix

Field Embedded Factorization Machine

MCKERNEL

ASGD Weight-Dropped LSTM

Siamese Network

Heterogeneous Molecular Graph Neural Network

Spectrally Normalised GAN

Adaptive Robust Loss

ResNeSt

DVD-GAN DBlock

Cosine Linear Unit

FLAVA

efficient channel attention

DeepViT

Viet-Anh on Software