**Virtual Data Augmentation**, or **VDA**, is a framework for robustly fine-tuning pre-trained language model. Based on the original token embeddings, a multinomial mixture for augmenting virtual data is constructed, where a masked language model guarantees the semantic relevance and the Gaussian noise provides the augmentation diversity. Furthermore, a regularized training strategy is proposed to balance the two aspects.

Virtual Data Augmentation

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models

Simulations of multi-modal distributions can be very costly and often lead to unreliable predictions. To accelerate the computations, we propose to sample from a flattened distribution to accelerate the computations and estimate the importance weights between the original distribution and the flattened distribution to ensure the correctness of the distribution.

CSGLD

A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

**Multiplicative Attention** is an attention mechanism where the alignment score function is calculated as:

$$f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right) = \mathbf{h}\_{i}^{T}\textbf{W}\_{a}\mathbf{s}\_{j}$$

Here $\mathbf{h}$ refers to the hidden states for the encoder/source, and $\mathbf{s}$ is the hidden states for the decoder/target. The function above is thus a type of alignment score function. We can use a matrix of alignment scores to show the correlation between source and target words, as the Figure to the right shows. Within a neural network, once we have the alignment scores, we calculate the final scores using a [softmax](https://paperswithcode.com/method/softmax) function of these alignment scores (ensuring it sums to 1).

Additive and multiplicative attention are similar in complexity, although multiplicative attention is faster and more space-efficient in practice as it can be implemented more efficiently using matrix multiplication. Both variants perform similar for small dimensionality $d_{h}$ of the decoder states, but [additive attention](https://paperswithcode.com/method/additive-attention) performs better for larger dimensions. One way to mitigate this is to scale $f_{att}\left(\textbf{h}_{i}, \textbf{s}\_{j}\right)$ by $1/\sqrt{d\_{h}}$ as with [scaled dot-product attention](https://paperswithcode.com/method/scaled).

Multiplicative Attention

Effective Approaches to Attention-based Neural Machine Translation

Deep Learning for NLP Best Practices by Sebastian Ruder

**Neural Oblivious Decision Ensembles (NODE)** is a tabular data architecture that consists of differentiable
oblivious decision trees (ODT) that are trained end-to-end by backpropagation. 

The core building block is a Neural Oblivious Decision Ensemble (NODE) layer. The layer is composed of $m$ differentiable oblivious decision trees (ODTs) of equal depth $d$. As an input, all $m$ trees get a common vector $x \in \mathbb{R}^{n}$, containing $n$ numeric features. Below we describe a design of a single differentiable ODT.

In its essence, an ODT is a decision table that splits the data along $d$ splitting features and compares each feature to a learned threshold. Then, the tree returns one of the $2^{d}$ possible responses, corresponding to the comparisons result. Therefore, each ODT is completely determined by its splitting features $f \in \mathbb{R}^{d}$, splitting thresholds $b \in \mathbb{R}^{d}$ and a $d$-dimensional tensor of responses $R \in \mathbb{R} \underbrace{2 \times 2 \times 2}_{d}$. In this notation, the tree output is defined as:

$$
h(x)=R\left[\mathbb{1}\left(f\_{1}(x)-b_{1}\right), \ldots, \mathbb{1}\left(f\_{d}(x)-b\_{d}\right)\right]
$$
where $\mathbb{1}(\cdot)$ denotes the Heaviside function.

NODE

Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data

**Deformable RoI Pooling** adds an offset to each bin position in the regular bin partition of the RoI Pooling. Similarly, the offsets are learned from the preceding feature maps and the RoIs, enabling adaptive part localization for objects with different shapes.

Deformable RoI Pooling

Deformable Convolutional Networks

Proposes a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. A cross-modal pre-training model is defined based on the image-text retrieval task. The main goal is thus to learn two encoders that can embed image and text samples into the same space for effective image-text retrieval. To enforce such cross-modal embedding learning, we introduce contrastive learning with the InfoNCE loss into the BriVL model. Given text embedding, the learning objective aims to find the best image embedding from a batch of image embeddings. Similarly, for a given image embedding, the learning objective is to find the best text embedding from a batch of text embeddings. The pre-training model learns a cross-modal embedding space by jointly training the image and text encoders to maximize the cosine similarity of the image and text embeddings of the true pair for each sample in the batch while minimizing the cosine similarity of the embeddings of the other incorrect pairs.

WenLan

WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

**Matrix NMS**, or **Matrix Non-Maximum Suppression**,  performs [non-maximum suppression](https://paperswithcode.com/method/non-maximum-suppression) with parallel matrix operations in one shot. It is motivated by [Soft-NMS](https://paperswithcode.com/method/soft-nms). Soft-NMS decays the other detection scores as a monotonic decreasing function $f(iou)$ of their overlaps. By decaying the scores according to IoUs recursively, higher IoU detections will be eliminated with a minimum score threshold. However, such process is sequential like traditional Greedy NMS and can not be implemented in parallel.

Matrix NMS views this process from another perspective by considering how a predicted mask $m\_{j}$ being suppressed. For $m\_{j}$, its decay factor is affected by: (a) The penalty of each prediction $m\_{i}$ on $m\_{j}$ $\left(s\_{i}>s\_{j}\right)$, where $s\_{i}$ and $s\_{j}$ are the confidence scores; and (b) the probability of $m\_{i}$ being suppressed. For (a), the penalty of each prediction $m\_{i}$ on $m\_{j}$ could be easily computed by $f\left(\right.$ iou $\left.\_{i, j}\right)$. For (b), the probability of $m\_{i}$ being suppressed is not so elegant to be computed. However, the probability usually has positive correlation with the IoUs. So here we directly approximate the probability by the most overlapped prediction on $m\_{i}$ as

$$
f\left(\text { iou. }\_{, i}\right)=\min\_{\forall s\_{k}>s\_{i}} f\left(\text { iou }\_{k, i}\right)
$$

To this end, the final decay factor becomes

$$
\operatorname{decay}\_{j}=\min\_{\forall s\_{i}>s\_{j}} \frac{f\left(\text { iou }\_{i, j}\right)}{f\left(\text { iou }\_{\cdot, i}\right)}
$$

and the updated score is computed by $s\_{j}=s\_{j} \cdot$ decay $\_{j} .$ The authors consider the two most simple decremented functions, denoted as linear $f\left(\right.$ iou $\left.\_{i, j}\right)=1-$ iou $\_{i, j}$, and Gaussian $f\left(\right.$ iou $\left.\_{i, j}\right)=\exp \left(-\frac{i o u\_{i, j}^{2}}{\sigma}\right)$.

Matrix NMS

SOLOv2: Dynamic and Fast Instance Segmentation

**CornerNet** is an object detection model that detects an object bounding box as a pair of keypoints, the top-left corner and the bottom-right corner, using a single [convolution](https://paperswithcode.com/method/convolution) neural network. By detecting objects as paired keypoints, we eliminate the need for designing a set of anchor boxes commonly used in prior single-stage detectors. It also utilises [corner pooling](https://paperswithcode.com/method/corner-pooling), a new type of pooling layer than helps the network better localize corners.

CornerNet

CornerNet: Detecting Objects as Paired Keypoints

**Fourier Contour Embedding** is a text instance representation that allows networks to learn diverse text geometry variances. Most of existing methods model text instances in image spatial domain via masks or contour point sequences in the Cartesian or the polar coordinate system. However, the mask representation might lead to expensive post-processing, while the point sequence one may have limited capability to model texts with highly-curved shapes. This motivates modeling text instances in the Fourier domain.

Fourier Contour Embedding

Fourier Contour Embedding for Arbitrary-Shaped Text Detection

**IFBlock** is a video model block used in the [IFNet](https://paperswithcode.com/method/ifnet) architecture for video frame interpolation. IFBlocks do not contain expensive operators like cost volume or forward warping and use 3 × 3 convolution and deconvolution as building blocks. Each IFBlock has a feed-forward structure consisting of several convolutional layers and an upsampling operator. Except for the layer that outputs the optical flow residuals and the fusion map, [PReLU](https://paperswithcode.com/method/prelu) activations are used.

IFBlock

RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

**AutoSync** is a pipeline for automatically optimizing synchronization strategies, given model structures and resource specifications, in data-parallel distributed machine learning. By factorizing the synchronization strategy with respect to each trainable building block of a DL model, we can construct a valid and large strategy space spanned by multiple factors. AutoSync efficiently navigates the space and locates the optimal strategy. AutoSync leverages domain knowledge about synchronization systems to reduce the search space, and is equipped with a domain adaptive simulator, which combines principled communication modeling and data-driven ML models, to estimate the runtime of strategy proposals without launching real distributed execution.

AutoSync

AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning

**DropAttack** is an adversarial training method that adds intentionally worst-case adversarial perturbations to both the input and hidden layers in different dimensions and minimizes the adversarial risks generated by each layer.

DropAttack

DropAttack: A Masked Weight Adversarial Training Method to Improve Generalization of Neural Networks

Please enter a description about the method here

RTMDet

Parametrized learnable activation function, based on the Padé approximant.

Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

**SC-GPT** is a multi-layer [Transformer](http://paperswithcode.com/method/transformer) neural language model, trained in three steps: (i) Pre-trained on plain text, similar to [GPT-2](http://paperswithcode.com/method/gpt-2); (ii) Continuously pretrained on large amounts of dialog-act labeled utterances corpora to acquire the ability of controllable generation; (iii) Fine-tuned for a target domain using very limited amounts of domain labels. Unlike [GPT-2](http://paperswithcode.com/method/gpt-2), SC-GPT generates semantically controlled responses that are conditioned on the given semantic form, similar to SC-[LSTM](https://paperswithcode.com/method/lstm) but requiring much less domain labels to generalize to new domains. It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains.

SC-GPT

Few-shot Natural Language Generation for Task-Oriented Dialog

**Instance-Level Meta Normalization** is a normalization method that addresses a learning-to-normalize problem. ILM-Norm learns to predict the normalization parameters via both the feature feed-forward and the gradient back-propagation paths. It uses an auto-encoder to predict the weights $\omega$ and bias $\beta$ as the rescaling parameters for recovering the distribution of the tensor $x$ of feature maps. Instead of using the entire feature tensor $x$ as the input for the auto-encoder, it uses the mean $\mu$ and variance $\gamma$ of $x$ for characterizing its statistics.

Instance-Level Meta Normalization

**Barlow Twins** is a self-supervised learning method that applies redundancy-reduction — a principle first proposed in neuroscience — to self supervised learning. The objective function measures the cross-correlation matrix between the embeddings of two identical networks fed with distorted versions of a batch of samples, and tries to make this matrix close to the identity. This causes the embedding vectors of distorted version of a sample to be similar, while minimizing the redundancy between the components of these vectors. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors.

Barlow Twins

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

**Thinned U-shape Module**, or **TUM**, is a feature extraction block used for object detection models. It was introduced as part of the [M2Det](https://paperswithcode.com/method/m2det) architecture. Different from [FPN](https://paperswithcode.com/method/fpn) and [RetinaNet](https://paperswithcode.com/method/retinanet), TUM adopts a thinner U-shape structure as illustrated in the Figure to the right. The encoder is a series of 3x3 [convolution](https://paperswithcode.com/method/convolution) layers with stride 2. And the decoder takes the outputs of these layers as its reference set of feature maps, while the original FPN chooses the output of the last layer of each stage in [ResNet](https://paperswithcode.com/method/resnet) backbone. 

In addition, with TUM, we add [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) layers after the upsample and element-wise sum operation at the decoder branch to enhance learning ability and keep smoothness for the features. In the context of M2Det, all of the outputs in the decoder of each TUM form the multi-scale features of the current level. As a whole, the outputs of stacked TUMs form the multi-level multi-scale features, while the front TUM mainly provides shallow-level features, the middle TUM provides medium-level features, and the back TUM provides deep-level features.

Viet-Anh on Software

AI Glossary

Tech Terminology

Virtual Data Augmentation

Contour Stochastic Gradient Langevin Dynamics