The **Adaptively Sparse Transformer** is a type of [Transformer](https://paperswithcode.com/method/transformer).

Adaptively Sparse Transformer

Adaptively Sparse Transformers

We are introducing a universal handwritten image augmentation method that is language-agnostic. This groundbreaking technique can be applied to handwritten images in any language worldwide, marking it as the first of its kind. There are four methods for handwritten images which are ThickOCR, ThinOCR, Elongate OCR, Line Erase OCR.

Handwritten OCR

Handwritten image augmentation

**Adaptive Feature Pooling** pools features from all levels for each proposal in object detection and fuses them for the following prediction. For each proposal, we map them to different feature levels. Following the idea of [Mask R-CNN](https://paperswithcode.com/method/adaptive-feature-pooling), [RoIAlign](https://paperswithcode.com/method/roi-align) is used to pool feature grids from each level. Then a fusion operation (element-wise max or sum) is utilized to fuse feature grids from different levels.

The motivation for this technique is that in an [FPN](https://paperswithcode.com/method/fpn) we assign proposals to different feature levels based on the size of proposals, which could be suboptimal if images with small differences are assigned to different levels, or if the importance of features is not strongly correlated to their level which they belong.

Adaptive Feature Pooling

Path Aggregation Network for Instance Segmentation

**Attention-augmented Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) with a two-dimensional relative self-attention mechanism that can replace convolutions as a stand-alone computational primitive for image classification. It employs [scaled-dot product attention](https://paperswithcode.com/method/scaled) and [multi-head attention](https://paperswithcode.com/method/multi-head-attention) as with [Transformers](https://paperswithcode.com/method/transformer).

It works by concatenating convolutional and attentional feature map. To see this, consider an original convolution operator with kernel size $k$, $F\_{in}$ input filters and $F\_{out}$ output filters. The corresponding attention augmented convolution can be written as"

$$\text{AAConv}\left(X\right) = \text{Concat}\left[\text{Conv}(X), \text{MHA}(X)\right] $$

$X$ originates from an input tensor of shape $\left(H, W, F\_{in}\right)$. This is flattened to become $X \in \mathbb{R}^{HW \times F\_{in}}$ which is passed into a multi-head attention module, as well as a convolution (see above).

Similarly to the convolution, the attention augmented convolution 1) is equivariant to translation and 2) can readily operate on inputs of different spatial dimensions.

Attention-augmented Convolution

Attention Augmented Convolutional Networks

**Multi-band MelGAN**, or **MB-MelGAN**, is a waveform generation model focusing on high-quality text-to-speech. It improves the original [MelGAN](https://paperswithcode.com/method/melgan) in several ways. First, it increases the receptive field of the generator, which is proven to be beneficial to speech generation. Second, it substitutes the feature matching loss with the multi-resolution STFT loss to better measure the difference between fake and real speech. Lastly, [MelGAN](https://paperswithcode.com/method/melgan) is extended with multi-band processing: the generator takes mel-spectrograms as input and produces sub-band signals which are subsequently summed back to full-band signals as discriminator input.

Multi-band MelGAN

Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech

**Mixture Normalization** is normalization technique that relies on an approximation of the probability density function of the internal representations. Any continuous distribution can be approximated with arbitrary precision using a Gaussian Mixture Model (GMM). Hence, instead of computing one set of statistical measures from the entire population (of instances in the mini-batch) as [Batch Normalization](https://paperswithcode.com/method/batch-normalization) does, Mixture Normalization works on sub-populations which can be identified by disentangling modes of the distribution, estimated via GMM. 

While BN can only scale and/or shift the whole underlying probability density function, mixture normalization operates like a soft piecewise normalizing transform, capable of completely re-structuring the data distribution by independently scaling and/or shifting individual modes of distribution.

Mixture Normalization

Training Faster by Separating Modes of Variation in Batch-normalized Models

**VC R-CNN** is an unsupervised feature representation learning method, which uses Region-based Convolutional Neural Network ([R-CNN](https://paperswithcode.com/method/r-cnn)) as the visual backbone, and the causal intervention as the training objective. Given a set of detected object regions in an image (e.g., using [Faster R-CNN](https://paperswithcode.com/method/faster-r-cnn)), like any other unsupervised feature learning methods (e.g., word2vec), the proxy training objective of VC R-CNN is to predict the contextual objects of a region. However, they are fundamentally different: the prediction of VC R-CNN is by using causal intervention: P(Y|do(X)), while others are by using the conventional likelihood: P(Y|X). This is also the core reason why VC R-CNN can learn "sense-making" knowledge like chair can be sat -- while not just "common" co-occurrences such as the chair is likely to exist if table is observed.

VC R-CNN

Visual Commonsense R-CNN

**PGNet** is a point-gathering network for reading arbitrarily-shaped text in real-time. It is a single-shot text spotter, where the pixel-level character classification map is learned with proposed PG-CTC loss avoiding the usage of character-level annotations. With a PG-CTC decoder, we gather high-level character classification vectors from two-dimensional space and decode them into text symbols without NMS and RoI operations involved, which guarantees high efficiency. Additionally, reasoning the relations between each character and its neighbors, a graph refinement module (GRM) is proposed to optimize the coarse recognition and improve the end-to-end performance.

PGNet

PGNet: Real-time Arbitrarily-Shaped Text Spotting with Point Gathering Network

**Elastic Dense Block** is a skip connection block that modifies the [Dense Block](https://paperswithcode.com/method/dense-block) with downsamplings and upsamplings in parallel branches at each layer to let the network learn from a data scaling policy in which inputs are processed at different resolutions in each layer. It is called "elastic" because each layer in the network is flexible in terms of choosing the best scale by a soft policy.

Elastic Dense Block

ELASTIC: Improving CNNs with Dynamic Scaling Policies

**RotNet** is a self-supervision approach that relies on predicting image rotations as the pretext task
in order to learn image representations.

RotNet

RotNet: Fast and Scalable Estimation of Stellar Rotation Periods Using Convolutional Neural Networks

**Class-MLP** is an alternative to [average pooling](https://paperswithcode.com/method/average-pooling), which is an adaptation of the class-attention token introduced in [CaiT](https://paperswithcode.com/method/cait). In CaiT, this consists of two layers that have the same structure as the [transformer](https://paperswithcode.com/method/transformer), but in which only the class token is updated based on the frozen patch embeddings. In Class-MLP, the same approach is used, but after aggregating the patches with a [linear layer](https://paperswithcode.com/method/linear-layer), we replace the [attention-based interaction](https://paperswithcode.com/method/scaled) between the class and patch embeddings by simple linear layers, still keeping the patch embeddings frozen. This increases the performance, at the expense of adding some parameters and computational cost. This pooling variant is referred to as “class-MLP”, since the purpose of these few layers is to replace average pooling.

Class-MLP

ResMLP: Feedforward networks for image classification with data-efficient training

With the increasing demand of autonomous systems, pixelwise semantic segmentation for visual scene understanding needs to be not only accurate but also efficient for potential real-time applications. In this paper, we propose Context Aggregation Network, a dual branch convolutional neural network, with significantly lower computational costs as compared to the state-of-the-art, while maintaining a competitive prediction accuracy. Building upon the existing dual branch architectures for high-speed semantic segmentation, we design a high resolution branch for effective spatial detailing and a context branch with light-weight versions of global aggregation and local distribution blocks, potent to capture both long-range and local contextual dependencies required for accurate semantic segmentation, with low computational overheads. We evaluate our method on two semantic segmentation datasets, namely Cityscapes dataset and UAVid dataset. For Cityscapes test set, our model achieves state-of-the-art results with mIOU of 75.9%, at 76 FPS on an NVIDIA RTX 2080Ti and 8 FPS on a Jetson Xavier NX. With regards to UAVid dataset, our proposed network achieves mIOU score of 63.5% with high execution speed (15 FPS).

CABiNet

**Memory-Associated Differential** (**MAD**) Learning was developed to inference from the memorized facts that we already know to predict what we want to know.

Image source: [Luo et al.](https://arxiv.org/pdf/2102.05246v1.pdf)

MAD Learning

Memory-Associated Differential Learning

**TD-VAE**, or **Temporal Difference VAE**, is a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of [temporal difference learning](https://paperswithcode.com/method/td-lambda) used in reinforcement learning.

TD-VAE

Temporal Difference Variational Auto-Encoder

**SortCut Sinkhorn Attention** is a variant of [Sparse Sinkhorn Attention](https://paperswithcode.com/method/sparse-sinkhorn-attention) where a post-sorting truncation of the input sequence is performed, essentially performing a hard top-k operation on the input sequence blocks within the computational graph. While most attention models mainly re-weight or assign near-zero weights during training, this allows for explicitly and dynamically truncate the input sequence. Specifically:

$$ Y = \text{Softmax}\left(Q{\psi\_{S}}\left(K\right)^{T}\_{\left[:n\right]}\right)\psi\_{S}\left(V\right)\_{\left[:n\right]} $$

where $n$ is the Sortfut budget hyperparameter.

SortCut Sinkhorn Attention

Sparse Sinkhorn Attention

The **Exponential Linear Squashing Activation Function**, or **ELiSH**, is an activation function used for neural networks. It shares common properties with [Swish](https://paperswithcode.com/method/swish), being made up of an [ELU](https://paperswithcode.com/method/elu) and a [Sigmoid](https://paperswithcode.com/method/sigmoid-activation):

$$f\left(x\right) = \frac{x}{1+e^{-x}} \text{ if } x \geq 0 $$
$$f\left(x\right) = \frac{e^{x} - 1}{1+e^{-x}} \text{ if } x < 0 $$

The Sigmoid part of **ELiSH** improves information flow, while the linear parts solve issues of vanishing gradients.

ELiSH

The Quest for the Golden Activation Function

**SAINT** is a hybrid deep learning approach to solving tabular data problems. SAINT performs attention over both rows and columns, and it includes an enhanced embedding method. The architecture, pre-training and training pipeline are as follows: 

- $L$ layers with 2 attention blocks each, one self-attention block, and a novel intersample attention blocks that computes attention across samples are used.
- For pre-training, this involves minimizing the contrastive and denoising losses between a given data point and its views generated by [CutMix](https://paperswithcode.com/method/cutmix) and [mixup](https://paperswithcode.com/method/mixup). During finetuning/regular training, data passes through an embedding layer and then the SAINT model. Lastly, the contextual embeddings from SAINT are used to pass only the embedding corresponding to the CLS token through an [MLP](https://paperswithcode.com/method/feedforward-network) to obtain the final prediction.

SAINT

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

**IB-BERT**, or **Inverted Bottleneck BERT**, is a [BERT](https://paperswithcode.com/method/bert) variant that uses an [inverted bottleneck](https://paperswithcode.com/method/inverted-residual-block) structure. It is used as a teacher network to train the [MobileBERT](https://paperswithcode.com/method/mobilebert) models.

Viet-Anh on Software

AI Glossary

Tech Terminology

Adaptively Sparse Transformer

Handwritten OCR augmentation