The **Feature-wise linear modulation** (**FiLM**) module combines information from both noisy waveform and input mel-spectrogram. It is used in the [WaveGrad](https://paperswithcode.com/method/wavegrad) model. The authors also added iteration index $n$ which indicates the noise level of the input waveform by using the [Transformer](https://paperswithcode.com/method/transformer) sinusoidal positional embedding. To condition on the noise level directly, $n$ is replaced by $\sqrt{\bar{\alpha}}$ and a linear scale $C = 5000$ is applied. The FiLM module produces both scale and bias vectors given inputs, which are used in a UBlock for feature-wise affine transformation as:

$$ \gamma\left(D, \sqrt{\bar{\alpha}}\right) \odot U + \zeta\left(D, \sqrt{\bar{\alpha}}\right) $$

where $\gamma$ and $\zeta$ correspond to the scaling and shift vectors from the FiLM module, $D$ is the output from corresponding [DBlock](https://paperswithcode.com/method/dblock), $U$ is an intermediate output in the UBlock.

FiLM Module

WaveGrad: Estimating Gradients for Waveform Generation

TabNN is a universal neural network solution to derive effective NN architectures for tabular data in all kinds of tasks automatically. Specifically, the design of TabNN follows two principles: to explicitly leverage expressive feature combinations and to reduce model complexity. Since GBDT has empirically proven its strength in modeling tabular data, GBDT is used to power the implementation of TabNN.

TabNN

TabNN: A Universal Neural Network Solution for Tabular Data

**MobileDet** is an object detection model developed for mobile accelerators. MobileDets uses regular convolutions extensively on EdgeTPUs and DSPs, especially in the early stage of the network where depthwise convolutions tend to be less efficient.  This helps boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via [neural architecture search](https://paperswithcode.com/method/neural-architecture-search). By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, an efficient family of object detection models is obtained.

MobileDet

MobileDets: Searching for Object Detection Architectures for Mobile Accelerators

**AutoInt** is a deep tabular learning method that models high-order feature interactions of input features. AutoInt can be applied to both numerical and categorical input features. Specifically, both the numerical and categorical features are mapped into the same low-dimensional space. Afterwards, a multi-head self-attentive neural network with residual connections is proposed to explicitly model the feature interactions in the low-dimensional space. With different layers of the multi-head self-attentive neural networks, different orders of feature combinations of input features can be modeled.

AutoInt

AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks

**AdaGPR** is an adaptive, layer-wise graph [convolution](https://paperswithcode.com/method/convolution) model. AdaGPR applies adaptive generalized Pageranks at each layer of a [GCNII](https://paperswithcode.com/method/gcnii) model by learning to predict the coefficients of generalized Pageranks using sparse solvers.

AdaGPR

Layer-wise Adaptive Graph Convolution Networks Using Generalized Pagerank

A **trainable 3D interaction space** aims to captures the associations between the triplet components and helps model the recognition of multiple triplets in the same frame.

Source: [Nwoye et al.](https://arxiv.org/pdf/2007.05405v1.pdf)

Image source: [Nwoye et al.](https://arxiv.org/pdf/2007.05405v1.pdf)

3DIS

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

**MATE** is a [Transformer](https://paperswithcode.com/method/transformer) architecture designed to model the structure of web tables. It uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. Each attention head reorders the tokens by either column or row index and then applies a windowed attention mechanism. Unlike traditional self-attention, Mate scales linearly in the sequence length.

MATE

MATE: Multi-view Attention for Table Transformer Efficiency

**Primer** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based architecture that improves upon the [Transformer](https://paperswithcode.com/method/transformer) architecture with two improvements found through [neural architecture search](https://paperswithcode.com/methods/category/neural-architecture-search): [squared RELU activations](https://paperswithcode.com/method/squared-relu) in the feedforward block, and [depthwise convolutions]() added to the attention multi-head projections: resulting in a new module called [Multi-DConv-Head-Attention](https://paperswithcode.com/method/multi-dconv-head-attention).

Primer

Primer: Searching for Efficient Transformers for Language Modeling

**ComiRec** is a multi-interest framework for sequential recommendation. The multi-interest module captures multiple interests from user behavior sequences, which can be exploited for retrieving candidate items from the large-scale item pool. These items are then fed into an aggregation module to obtain the overall recommendation. The aggregation module leverages a controllable factor to balance the recommendation accuracy and diversity.

ComiRec

Controllable Multi-Interest Framework for Recommendation

**Weight Decay**, or **$L_{2}$ Regularization**, is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a penalty on the $L\_{2}$ Norm of the weights:

$$L\_{new}\left(w\right) = L\_{original}\left(w\right) + \lambda{w^{T}w}$$

where $\lambda$ is a value determining the strength of the penalty (encouraging smaller weights). 

Weight decay can be incorporated directly into the weight update rule, rather than just implicitly by defining it through to objective function. Often weight decay refers to the implementation where we specify it directly in the weight update rule (whereas L2 regularization is usually the implementation which is specified in the objective function).

Image Source: Deep Learning, Goodfellow et al

Weight Decay

**Hardtanh** is an activation function used for neural networks:

$$ f\left(x\right) = -1 \text{ if } x < - 1 $$
$$ f\left(x\right) = x \text{ if } -1 \leq x \leq 1 $$
$$ f\left(x\right) = 1 \text{ if } x > 1 $$

It is a cheaper and more computationally efficient version of the [tanh activation](https://paperswithcode.com/method/tanh-activation).

Image Source: [Zhuan Lan](https://zhuanlan.zhihu.com/p/30385380)

Hardtanh Activation

**Spatial Gating Unit**, or **SGU**, is a gating unit used in the [gMLP](https://paperswithcode.com/method/gmlp) architecture to captures spatial interactions. To enable cross-token interactions, it is necessary for the layer $s(\cdot)$ to contain a contraction operation over the spatial dimension. The layer $s(\cdot)$ is formulated as the output of linear gating:

$$
s(Z)=Z \odot f\_{W, b}(Z)
$$

where $\odot$ denotes element-wise multiplication. For training stability, the authors find it critical to initialize $W$ as near-zero values and $b$ as ones, meaning that $f\_{W, b}(Z) \approx 1$ and therefore $s(Z) \approx Z$ at the beginning of training. This initialization ensures each [gMLP](https://paperswithcode.com/method/gmlp) block behaves like a regular [FFN](https://paperswithcode.com/method/gmlp) at the early stage of training, where each token is processed independently, and only gradually injects spatial information across tokens during the course of learning.

The authors find it further effective to split $Z$ into two independent parts $\left(Z\_{1}, Z\_{2}\right)$ along the channel dimension for the gating function and for the multiplicative bypass:

$$
s(Z)=Z\_{1} \odot f\_{W, b}\left(Z\_{2}\right)
$$

They also normalize the input to $f\_{W, b}$ which empirically improved the stability of large NLP models.

Spatial Gating Unit

Pay Attention to MLPs

**Distributional Generalization** is a type of generalization that roughly states that outputs of a classifier at train and test time are close as distributions, as opposed to close in just their average error. This behavior is not captured by classical generalization, which would only consider the average error and not the distribution of errors over the input domain.

Distributional Generalization

Distributional Generalization: A New Kind of Generalization

**Softsign** is an activation function for neural networks:

$$ f\left(x\right) = \left(\frac{x}{|x|+1}\right)$$

Image Source: [Sefik Ilkin Serengil](https://sefiks.com/2017/11/10/softsign-as-a-neural-networks-activation-function/)

Softsign Activation

A **Non-Local Operation** is a component for capturing long-range dependencies with deep neural networks. It is a generalization of the classical non-local mean operation in computer vision. Intuitively a non-local operation computes the response at a position as a weighted sum of the features at all positions in the input feature maps. The set of positions can be in space, time, or spacetime, implying that these operations are applicable for image, sequence, and video problems.

Following the non-local mean operation, a generic non-local operation for deep neural networks is defined as:

$$ \mathbb{y}\_{i} = \frac{1}{\mathcal{C}\left(\mathbb{x}\right)}\sum\_{\forall{j}}f\left(\mathbb{x}\_{i}, \mathbb{x}\_{j}\right)g\left(\mathbb{x}\_{j}\right) $$

Here $i$ is the index of an output position (in space, time, or spacetime) whose response is to be computed and $j$ is the index that enumerates all possible positions. x is the input signal (image, sequence, video; often their features) and $y$ is the output signal of the same size as $x$. A pairwise function $f$ computes a scalar (representing relationship such as affinity) between $i$ and all $j$. The unary function $g$ computes a representation of the input signal at the position $j$. The
response is normalized by a factor $C\left(x\right)$.

The non-local behavior is due to the fact that all positions ($\forall{j}$) are considered in the operation. As a comparison, a convolutional operation sums up the weighted input in a local neighborhood (e.g., $i − 1 \leq j \leq i + 1$ in a 1D case with kernel size 3), and a recurrent operation at time $i$ is often based only on the current and the latest time steps (e.g., $j = i$ or $i − 1$).

The non-local operation is also different from a fully-connected (fc) layer. The equation above computes responses based on relationships between different locations, whereas fc uses learned weights. In other words, the relationship between $x\_{j}$ and $x\_{i}$ is not a function of the input data in fc, unlike in nonlocal layers. Furthermore, the formulation in the equation above supports inputs of variable sizes, and maintains the corresponding size in the output. On the contrary, an fc layer requires a fixed-size input/output and loses positional correspondence (e.g., that from $x\_{i}$ to $y\_{i}$ at the position $i$).

A non-local operation is a flexible building block and can be easily used together with convolutional/recurrent layers. It can be added into the earlier part of deep neural networks, unlike fc layers that are often used in the end. This allows us to build a richer hierarchy that combines both non-local and local information.

In terms of parameterisation, we usually parameterise $g$ as a linear embedding of the form $g\left(x\_{j}\right) = W\_{g}\mathbb{x}\_{j}$ , where $W\_{g}$ is a weight matrix to be learned. This is implemented as, e.g., 1×1 [convolution](https://paperswithcode.com/method/convolution) in space or 1×1×1 convolution in spacetime. For $f$ we use an affinity function, a list of which can be found [here](https://paperswithcode.com/methods/category/affinity-functions).

Non-Local Operation

Non-local Neural Networks

**SRGAN** is a generative adversarial network for single image super-resolution. It uses a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes the solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, the authors use a content loss motivated by perceptual similarity instead of similarity in pixel space. The actual networks - depicted in the Figure to the right - consist mainly of residual blocks for feature extraction.

Formally we write the perceptual loss function as a weighted sum of a ([VGG](https://paperswithcode.com/method/vgg)) content loss $l^{SR}\_{X}$ and an adversarial loss component $l^{SR}\_{Gen}$:

$$ l^{SR} = l^{SR}\_{X} + 10^{-3}l^{SR}\_{Gen} $$

SRGAN

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Many communication-efficient variants of [SGD](https://paperswithcode.com/method/sgd) use gradient quantization schemes. These schemes are often heuristic and fixed over the course of training. We empirically observe that the statistics of gradients of deep models change during the training. Motivated by this observation, we introduce two adaptive quantization schemes, ALQ and AMQ. In both schemes, processors update their compression schemes in parallel by efficiently computing sufficient statistics of a parametric distribution. We improve the validation accuracy by almost 2% on CIFAR-10 and 1% on ImageNet in challenging low-cost communication setups. Our adaptive methods are also significantly more robust to the choice of hyperparameters.

ALQ and AMQ

Adaptive Gradient Quantization for Data-Parallel SGD

**CPM-2** is a 11 billion parameters pre-trained language model based on a standard Transformer architecture consisting of a bidirectional encoder and a unidirectional decoder. The model is pre-trained on WuDaoCorpus which contains 2.3TB cleaned Chinese data as well as 300GB cleaned English data. The pre-training process of CPM-2 can be divided into three stages: Chinese pre-training, bilingual pre-training, and MoE pre-training. Multi-stage training with knowledge inheritance can significantly reduce the computation cost.

Viet-Anh on Software

AI Glossary

Tech Terminology