**Label Quality Model** is an intermediate supervised task aimed at predicting the clean labels from noisy labels by leveraging rater features and a paired subset for supervision. The LQM technique assumes the existence of rater features and a subset of training data with both noisy and clean labels, which we call paired-subset. In real world scenarios, some level of label noise may be unavoidable. The LQM approach still works as long as the clean(er) label is less noisy than a label from a rater that is randomly selected from the pool, e.g., clean labels can be from either expert raters or aggregation of multiple raters. LQM is trained on the paired-subset using rater features and noisy label as input, and inferred on the entire training corpus. The output of LQM is used during model training as a more accurate alternative to the noisy labels.

Label Quality Model

An Instance-Dependent Simulation Framework for Learning with Label Noise

**DAFNe** is a dense one-stage anchor-free deep model for oriented object detection. It is a deep neural network that performs predictions on a dense grid over the input image, being architecturally simpler in design, as well as easier to optimize than its two-stage counterparts. Furthermore, it reduces the prediction complexity by refraining from employing bounding box anchors. This enables a tighter fit to oriented objects, leading to a better separation of bounding boxes especially in case of dense object distributions. Moreover, it introduces an orientation-aware generalization of the center-ness function to arbitrary quadrilaterals that takes into account the object's orientation and that, accordingly, accurately down-weights low-quality predictions

DAFNe

DAFNe: A One-Stage Anchor-Free Approach for Oriented Object Detection

**Gradient Checkpointing** is a method used for reducing the memory footprint when training deep neural networks, at the cost of having a small increase in computation time.

Gradient Checkpointing

Training Deep Nets with Sublinear Memory Cost

**SRGAN Residual Block** is a residual block used in the [SRGAN](https://paperswithcode.com/method/srgan) generator for image super-resolution. It is similar to standard [residual blocks](https://paperswithcode.com/method/residual-block), although it uses a [PReLU](https://paperswithcode.com/method/prelu) activation function to help training (preventing sparse gradients during [GAN](https://paperswithcode.com/method/gan) training).

SRGAN Residual Block

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

**REINFORCE** is a Monte Carlo variant of a policy gradient algorithm in reinforcement learning. The agent collects samples of an episode using its current policy, and uses it to update the policy parameter $\theta$. Since one full trajectory must be completed to construct a sample space, it is updated as an off-policy algorithm.

$$ \nabla\_{\theta}J\left(\theta\right) = \mathbb{E}\_{\pi}\left[G\_{t}\nabla\_{\theta}\ln\pi\_{\theta}\left(A\_{t}\mid{S\_{t}}\right)\right]$$

Image Credit: [Tingwu Wang](http://www.cs.toronto.edu/~tingwuwang/REINFORCE.pdf)

REINFORCE

**GrowNet** is a novel approach to combine the power of gradient boosting to incrementally build complex deep neural networks out of shallow components. It introduces a versatile framework that can readily be adapted for a diverse range of machine learning tasks in a wide variety of domains.

GrowNet

Gradient Boosting Neural Networks: GrowNet

The **Rectified Linear Unit N**, or **ReLUN**, is a modification of **[ReLU6](https://paperswithcode.com/method/relu6)** activation function that has trainable parameter **n**.

$$ReLUN(x) = min(max(0, x), n)$$

ReLUN

Trainable Activations for Image Classification

ComplEx model trained with a nuclear norm regularizer

ComplEx-N3

Canonical Tensor Decomposition for Knowledge Base Completion

**HiFi-GAN** is a generative adversarial network for speech synthesis. HiFi-GAN consists of one generator and two discriminators: multi-scale and multi-period discriminators. The generator and discriminators are trained adversarially, along with two additional losses for improving training stability and model performance.

The generator is a fully convolutional neural network. It uses a mel-spectrogram as input and upsamples it through transposed convolutions until the length of the output sequence matches the temporal resolution of raw waveforms. Every [transposed convolution](https://paperswithcode.com/method/transposed-convolution) is followed by a multi-receptive field fusion (MRF) module.

For the discriminator, a multi-period discriminator (MPD) is used consisting of several sub-discriminators each handling a portion of periodic signals of input audio. Additionally, to capture consecutive patterns and long-term dependencies, the multi-scale discriminator (MSD) proposed in [MelGAN](https://paperswithcode.com/method/melgan) is used, which consecutively evaluates audio samples at different levels.

HiFi-GAN

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

**ShuffleNet V2 Downsampling Block** is a block for spatial downsampling used in the [ShuffleNet V2](https://paperswithcode.com/method/shufflenet-v2) architecture. Unlike the regular [ShuffleNet](https://paperswithcode.com/method/shufflenet) V2 block, the channel split operator is removed so the number of output channels is doubled.

ShuffleNet V2 Downsampling Block

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

An implementation of model & data parallel [GPT3-like](https://paperswithcode.com/method/gpt-3) models using the [mesh-tensorflow](https://github.com/tensorflow/mesh) library.

Source: [EleutherAI/GPT-Neo](https://github.com/EleutherAI/gpt-neo)

GPT-Neo

VL-BERT is pre-trained on a large-scale image-captions dataset together with text-only corpus. The input to the model are either words from the input sentences or regions-of-interest (RoI) from input images. It can be fine-tuned to fit most visual-linguistic downstream tasks. Its backbone is a multi-layer bidirectional Transformer encoder, modified to accommodate visual contents, and new type of visual feature embedding to the input feature embeddings. VL-BERT takes both visual and linguistic elements as input, represented as RoIs in images and subwords in input sentences. Four different types of embeddings are used to represent each input: token embedding, visual feature embedding, segment embedding, and sequence position embedding. VL-BERT is pre-trained using Conceptual Captions and text-only datasets. Two pre-training tasks are used: masked language modeling with visual clues, and masked RoI classification with linguistic clues.

VL-BERT

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

To avoid the problem caused by low-frequent entity-relation pairs, our MBS uses the estimated probabilities from a trained model $\mathbf{\theta}'$ to calculate frequencies for each triplet and query. By using $\mathbf{\theta}'$, the NS loss in KGE with MBS is represented as follows:
\begin{align}
    &\ell_{mbs}(\mathbf{\theta};\mathbf{\theta}') \nonumber \\
=&-\frac{1}{|D|}\sum_{(x,y) \in D} \Bigl[A_{mbs}(\mathbf{\theta}')\log(\sigma(s_{\mathbf{\theta}}(x,y)+\gamma))\nonumber\\
    &+\frac{1}{\nu}sum_{y_{i}\sim p_n(y_{i}|x)}^{\nu}B_{mbs}(\mathbf{\theta}')\log(\sigma(-s_{\mathbf{\theta}}(x,y_i)-\gamma))\Bigr],
\end{align}

Model-based Subsampling for Knowledge Graph Completion

**Hard Swish** is a type of activation function based on [Swish](https://paperswithcode.com/method/swish), but replaces the computationally expensive sigmoid with a piecewise linear analogue:

$$\text{h-swish}\left(x\right) = x\frac{\text{ReLU6}\left(x+3\right)}{6} $$

Hard Swish

Searching for MobileNetV3

A **Feedforward Network**, or a **Multilayer Perceptron (MLP)**, is a neural network with solely densely connected layers. This is the classic neural network architecture of the literature. It consists of inputs $x$ passed through units $h$ (of which there can be many layers) to predict a target $y$. Activation functions are generally chosen to be non-linear to allow for flexible functional approximation.

Image Source: Deep Learning, Goodfellow et al

Feedforward Network

The multi-head of mixed attention combines both self- and cross-attentions, encouraging high-level learning of interactions between entities captured in the various attention features. It is build with several attention heads, each of the head can implement either self or cross attention. A self attention is when the key and query features are the same or come from the same domain features. A cross attention is when the key and query features are generated from different features. Modeling MHMA allows a model to identity the relationship between features of different domains. This is very useful in tasks involving relationship modeling such as human-object interaction, tool-tissue interaction, man-machine interaction, human-computer interface, etc.

MHMA

Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos

A **3D Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) where the kernel slides in 3 dimensions as opposed to 2 dimensions with 2D convolutions. One example use case is medical imaging where a model is constructed using 3D image slices. Additionally video based data has an additional temporal dimension over images making it suitable for this module. 

Image: Lung nodule detection based on 3D convolutional neural networks, Fan et al

3D Convolution

**LAMB** is a a layerwise adaptive large batch optimization technique. It provides a strategy for adapting the learning rate in large batch settings. LAMB uses [Adam](https://paperswithcode.com/method/adam) as the base algorithm and then forms an update as:

$$r\_{t} = \frac{m\_{t}}{\sqrt{v\_{t}} + \epsilon}$$
$$x\_{t+1}^{\left(i\right)} = x\_{t}^{\left(i\right)}  - \eta\_{t}\frac{\phi\left(|| x\_{t}^{\left(i\right)} ||\right)}{|| m\_{t}^{\left(i\right)} || }\left(r\_{t}^{\left(i\right)}+\lambda{x\_{t}^{\left(i\right)}}\right) $$

Unlike [LARS](https://paperswithcode.com/method/lars), the adaptivity of LAMB is two-fold: (i) per dimension normalization with respect to the square root of the second moment used in Adam and (ii) layerwise normalization obtained due to layerwise adaptivity.

Viet-Anh on Software

AI Glossary

Tech Terminology

Label Quality Model

DAFNe

Gradient Checkpointing

SRGAN Residual Block

REINFORCE

GrowNet

Rectified Linear Unit N

ComplEx with N3 Regularizer

HiFi-GAN

ShuffleNet V2 Downsampling Block

GPT-Neo

Visual-Linguistic BERT

Model-based Subsampling

Hard Swish

Feedforward Network

Multi-Heads of Mixed Attention

3D Convolution

LAMB

Viet-Anh on Software