**ParamCrop** is a parametric cubic cropping for video contrastive learning, where cubic cropping refers to cropping a 3D cube
from the input video. The central component of ParamCrop is a differentiable spatio-temporal cropping operation, which enables ParamCrop to be trained simultaneously with the video backbone and adjust the cropping strategy on the fly. The objective of ParamCrop is set to be adversarial to the video backbone, which is to increase the contrastive loss. Hence, initialized with the simplest setting where two cropped views largely overlaps, ParamCrop gradually increases the disparity between two views.

ParamCrop

ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning

**RIFE**, or **Real-time Intermediate Flow Estimation** is an intermediate flow estimation algorithm for Video Frame Interpolation (VFI). Many recent flow-based VFI methods first estimate the bi-directional optical flows, then scale and reverse them to approximate intermediate flows, leading to artifacts on motion boundaries. RIFE uses a neural network named [IFNet](https://paperswithcode.com/method/ifnet) that can directly estimate the intermediate flows from coarse-to-fine with much better speed. It introduces a privileged distillation scheme for training intermediate flow model, which leads to a large performance improvement.

In RIFE training, given two input frames $I_{0}, I_{1}$, we directly feed them into the IFNet to approximate intermediate flows $F_{t \rightarrow 0}, F_{t \rightarrow 1}$ and the fusion map $M$. During training phase, a privileged teacher refines student's results to get $F_{t \rightarrow 0}^{T e a}, F_{t \rightarrow 1}^{T e a}$ and $M^{\text {Tea }}$ based on ground truth $I_{t}$. The student model and the teacher model are jointly trained from scratch using the reconstruction loss. The teacher's approximations are more accurate so that they can guide the student to learn.

RIFE

RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

**Semi-Supervised Knowledge Distillation** is a type of knowledge distillation for person re-identification that exploits weakly annotated data by assigning soft pseudo labels to YouTube-Human to improve models' generalization ability. SSKD first trains a student model (e.g. [ResNet](https://paperswithcode.com/method/resnet)-50) and a teacher model (e.g. ResNet-101) using labeled data from multi-source domain datasets. Then, SSKD develops an [auxiliary classifier](https://paperswithcode.com/method/auxiliary-classifier) to imitate the soft predictions of unlabeled data generated by the teacher model. Meanwhile, the student model is also supervised by hard labels and predicted soft labels by the teacher model for labeled data.

SSKD

Semi-Supervised Domain Generalizable Person Re-Identification

**Disentangled Attention Mechanism** is an attention mechanism used in the [DeBERTa](https://paperswithcode.com/method/deberta) architecture. Unlike [BERT](https://paperswithcode.com/method/bert) where each word in the input layer is represented using a vector which is the sum of its word (content) embedding and position embedding, each word in DeBERTa is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices based on their contents and relative positions, respectively. This is motivated by the observation that the attention weight of a word pair depends on not only their contents but their relative positions. For example, the dependency between the words “deep” and “learning” is much stronger when they occur next to each other than when they occur in different sentences.

Disentangled Attention Mechanism

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

**Res2Net** is an image model that employs a variation on bottleneck residual blocks. The motivation is to be able to represent features at multiple scales. This is achieved through a novel building block for CNNs that constructs hierarchical residual-like connections within one single [residual block](https://paperswithcode.com/method/residual-block).
This represents multi-scale features at a granular level and increases the range of receptive fields for each network layer.

Res2Net

Res2Net: A New Multi-scale Backbone Architecture

**HetPipe** is a hybrid parallel method that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance.

HetPipe

**TernaryBERT** is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based model which ternarizes the weights of a pretrained [BERT](https://paperswithcode.com/method/bert) model to $\{-1,0,+1\}$, with different granularities for word embedding and weights in the Transformer layer. Instead of directly using knowledge distillation to compress a model, it is used to improve the performance of ternarized student model with the same size as the teacher model. In this way, we transfer the knowledge from the highly-accurate teacher model to the ternarized student model with smaller capacity.

TernaryBERT

TernaryBERT: Distillation-aware Ultra-low Bit BERT

**NormFormer** is a type of [Pre-LN](https://paperswithcode.com/method/layer-normalization) transformer that adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first [fully connected layer](https://paperswithcode.com/method/position-wise-feed-forward-layer). The modifications introduce a small number of additional learnable parameters, which provide a cost-effective way for each layer to change the magnitude of its features, and therefore the magnitude of the gradients to subsequent components.

NormFormer

NormFormer: Improved Transformer Pretraining with Extra Normalization

A **ParaNet Convolution Block** is a convolutional block that appears in the encoder and decoder of the [ParaNet](https://paperswithcode.com/method/paranet) text-to-speech architecture. It consists of a 1-D [convolution](https://paperswithcode.com/method/convolution) with a gated linear unit ([GLU](https://paperswithcode.com/method/glu)) and a [residual connection](https://paperswithcode.com/method/residual-connection). It is similar to the [DV3 Convolution Block](https://paperswithcode.com/method/dv3-convolution-block).

ParaNet Convolution Block

Non-Autoregressive Neural Text-to-Speech

**MODNet** is a light-weight matting objective decomposition network that can process portrait matting from a single input image in real time. The design of MODNet benefits from optimizing a series of correlated sub-objectives simultaneously via explicit constraints. To overcome the domain shift problem, MODNet introduces a self-supervised strategy based on subobjective consistency (SOC) and  a one-frame delay trick to smooth the results when applying MODNet to portrait video sequence.

Given an input image $I$, MODNet predicts human semantics $s\_{p}$, boundary details $d\_{p}$, and final alpha matte $\alpha\_{p}$ through three interdependent branches, $S, D$, and $F$, which are constrained by specific supervisions generated from the ground truth matte $\alpha\_{g}$. Since the decomposed sub-objectives are correlated and help strengthen each other, we can optimize MODNet end-to-end.

MODNet

MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition

**Conditional Positional Encoding**, or **CPE**, is a type of positional encoding for [vision transformers](https://paperswithcode.com/methods/category/vision-transformer). Unlike previous fixed or learnable positional encodings, which are predefined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE aims to generalize to the input sequences that are longer than what the model has ever seen during training. CPE can also keep the desired translation-invariance in the image classification task. CPE can be implemented with a [Position
Encoding Generator](https://paperswithcode.com/method/positional-encoding-generator) (PEG) and incorporated into the current [Transformer framework](https://paperswithcode.com/methods/category/transformers).

Conditional Positional Encoding

Conditional Positional Encodings for Vision Transformers

**DNN2LR** is an automatic feature crossing method to find feature interactions in a deep neural network, and use them as cross features in logistic regression. In general, DNN2LR consists of two steps: (1) generating a compact and accurate candidate set of cross feature fields; (2) searching in the candidate set for the final cross feature fields.

DNN2LR

DNN2LR: Interpretation-inspired Feature Crossing for Real-world Tabular Data

**ColorJitter** is a type of image data augmentation where we randomly change the brightness, contrast and saturation of an image.

Image Credit: [Apache MXNet](https://mxnet.apache.org/versions/1.5.0/tutorials/gluon/data_augmentation.html)

ColorJitter

Multi-head attention consists of multiple attention layers (heads) in parallel with different linear
transformations on the queries, keys, values and outputs. **Multi-query attention** is identical except that the
different heads share a single set of keys and values.

Multi-Query Attention

Fast Transformer Decoding: One Write-Head is All You Need

$s \cdot (\mathrm{ReLU}(x))^2 + b$

where $s \in \mathbb{R}$ and $b \in \mathbb{R}$ are shared for all channels and can be set as constants (s=0.8944, b=-0.4472) or learnable parameters.

StarReLU

MetaFormer Baselines for Vision

**Cutout** is an image augmentation and regularization technique that randomly masks out square regions of input during training. and can be used to improve the robustness and overall performance of convolutional neural networks. The main motivation for cutout comes from the problem of object occlusion, which is commonly encountered in many computer vision tasks, such as object recognition,
tracking, or human pose estimation. By generating new images which simulate occluded examples, we not only better prepare the model for encounters with occlusions in the real world, but the model also learns to take more of the image context into consideration when making decisions

Cutout

Improved Regularization of Convolutional Neural Networks with Cutout

The **Exponential Linear Unit** (ELU) is an activation function for neural networks. In contrast to [ReLUs](https://paperswithcode.com/method/relu), ELUs have negative values which allows them to push mean unit activations closer to zero like [batch normalization](https://paperswithcode.com/method/batch-normalization) but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While [LReLUs](https://paperswithcode.com/method/leaky-relu) and [PReLUs](https://paperswithcode.com/method/prelu) have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information.

The exponential linear unit (ELU) with $0 < \alpha$ is:

$$f\left(x\right) = x \text{ if } x > 0$$
$$\alpha\left(\exp\left(x\right) − 1\right) \text{ if } x \leq 0$$

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

An **Hourglass Module** is an image block module used mainly for pose estimation tasks. The design of the hourglass is motivated by the need to capture information at every scale. While local evidence is essential for identifying features like faces and hands, a final pose estimate requires a coherent understanding of the full body. The person’s orientation, the arrangement of their limbs, and the relationships of adjacent joints are among the many cues that are best recognized at different scales in the image. The hourglass is a simple, minimal design that has the capacity to capture all of these features and bring them together to output pixel-wise predictions.

The network must have some mechanism to effectively process and consolidate features across scales. The Hourglass uses a single pipeline with skip layers to preserve spatial information at each resolution. The network reaches its lowest resolution at 4x4 pixels allowing smaller spatial filters to be applied that compare features across the entire space of the image.

The hourglass is set up as follows: Convolutional and [max pooling](https://paperswithcode.com/method/max-pooling) layers are used to process features down to a very low resolution. At each max pooling step, the network branches off and applies more convolutions at the original pre-pooled resolution. After reaching the lowest resolution, the network begins the top-down sequence of upsampling and combination of features across scales. To bring together information across two adjacent resolutions, we do nearest neighbor upsampling of the lower resolution followed by an elementwise addition of the two sets of features. The topology of the hourglass is symmetric, so for every layer present on the way down there is a corresponding layer going up.

After reaching the output resolution of the network, two consecutive rounds of 1x1 convolutions are applied to produce the final network predictions. The output of the network is a set of heatmaps where for a given [heatmap](https://paperswithcode.com/method/heatmap) the network predicts the probability of a joint’s presence at each and every pixel.

Viet-Anh on Software

AI Glossary

Tech Terminology