**Absolute Position Encodings** are a type of position embeddings for [[Transformer](https://paperswithcode.com/method/transformer)-based models] where positional encodings are added to the input embeddings at the bottoms of the encoder and decoder stacks. The positional encodings have the same dimension $d\_{model}$ as the embeddings, so that the two can be summed. In the original implementation, sine and cosine functions of different frequencies are used:

$$ \text{PE}\left(pos, 2i\right) = \sin\left(pos/10000^{2i/d\_{model}}\right) $$

$$ \text{PE}\left(pos, 2i+1\right) = \cos\left(pos/10000^{2i/d\_{model}}\right) $$

where $pos$ is the position and $i$ is the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from $2\pi$ to $10000 \dot 2\pi$. This function was chosen because the authors hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$,  $\text{PE}\_{pos+k}$ can be represented as a linear function of $\text{PE}\_{pos}$.

Image Source: [D2L.ai](https://d2l.ai/chapter_attention-mechanisms/self-attention-and-positional-encoding.html)

Absolute Position Encodings

Attention Is All You Need

A **Time-aware Large Kernel (TaLK) convolution** is a type of temporal [convolution](https://paperswithcode.com/method/convolution) that learns the kernel size of a summation kernel for each time-step instead of learning the kernel weights as in a typical convolution operation. For each time-step, a function is responsible for predicting the appropriate size of neighbor representations to use in the form of left and right offsets relative to the time-step.

TaLK Convolution

Time-aware Large Kernel Convolutions

**ACGPN**, or **Adaptive Content Generating and Preserving Network**, is a [generative adversarial network](https://www.paperswithcode.com/method/category/generative-adversarial-network) for virtual try-on clothing applications. 

In Step I, the Semantic Generation Module (SGM) takes the target clothing image $\mathcal{T}\_{c}$, the pose map $\mathcal{M}\_{p}$, and the fused body part mask $\mathcal{M}^{F}$ as the input to predict the semantic layout and to output the synthesized body part mask $\mathcal{M}^{S}\_{\omega}$ and the target clothing mask $\mathcal{M}^{S\_{c}$.

In Step II, the Clothes Warping Module (CWM) warps the target clothing image to $\mathcal{T}^{R}\_{c}$ according to the predicted semantic layout, where a second-order difference constraint is introduced to stabilize the warping process. 

In Steps III and IV, the Content Fusion Module (CFM) first produces the composited body part mask $\mathcal{M}^{C}\_{\omega}$ using the original clothing mask $\mathcal{M}\_{c}$, the synthesized clothing mask $\mathcal{M}^{S}\_{c}$, the body part mask $\mathcal{M}\_{\omega}$, and the synthesized body part mask $\mathcal{M}\_{\omega}^{S}$, and then exploits a fusion network to generate the try-on images $\mathcal{I}^{S}$ by utilizing the information $\mathcal{T}^{R}\_{c}$, $\mathcal{M}^{S}\_{c}$, and the body part image $I\_{\omega}$ from previous steps.

ACGPN

Towards Photo-Realistic Virtual Try-On by Adaptively Generating-Preserving Image Content

**Dynamic Keypoint Head** is an output head for pose estimation that are conditioned on each instance (person), and can encode the instance concept in the dynamically-generated weights of their filters. They are used in the [FCPose](https://paperswithcode.com/method/fcpose) architecture.

The Figure shows the core idea. $F$ denotes a level of feature maps. "Rel. Coord." means the relative coordinates, denoting the relative offsets from the locations of $F$ to the location where the filters are generated. Refer to the text for details. $f\_{\theta\_{i}}$ is the dynamically-generated keypoint head for the $i$-th person instance. Note that each person instance has its own keypoint head.

Dynamic Keypoint Head

FCPose: Fully Convolutional Multi-Person Pose Estimation with Dynamic Instance-Aware Convolutions

**Bayesian Reward Extrapolation** is a Bayesian reward learning algorithm that scales to high-dimensional imitation learning problems by pre-training a low-dimensional feature encoding via self-supervised tasks and then leveraging preferences over demonstrations to perform fast Bayesian inference.

Bayesian REX

Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

**ERNIE-GEN** is a multi-flow sequence to sequence pre-training and fine-tuning framework which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder.

ERNIE-GEN

ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

**Probabilistic Continuously Indexed Domain Adaptation** (**PCIDA**) enjoys better theoretical guarantees to match both the mean and variance of the distribution $p(u|z)$. PCIDA can be extended to match higher-order moments.

PCIDA

Continuously Indexed Domain Adaptation

Based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. Adversarial Model Perturbation (AMP) improves generalization via minimizing the **AMP loss**, which is obtained from the empirical risk by applying the **worst** norm-bounded perturbation on each point in the parameter space.

Regularizing Neural Networks via Adversarial Model Perturbation

A **Res2Net Block** is an image model block that constructs hierarchical residual-like connections
within one single [residual block](https://paperswithcode.com/method/residual-block). It was proposed as part of the [Res2Net](https://paperswithcode.com/method/res2net) CNN architecture.

The block represents multi-scale features at a granular level and increases the range of receptive fields for each network layer. The $3 \times 3$ filters of $n$ channels is replaced with a set of smaller filter groups, each with $w$ channels. These smaller filter groups are connected in a hierarchical residual-like style to increase the number of scales that the output features can represent. Specifically, we divide input feature maps into several groups. A group of filters first extracts features from a group of input feature maps. Output features of the previous group are then sent to the next group of filters along with another group of input feature maps. 

This process repeats several times until all input feature maps are processed. Finally, feature maps from all groups are concatenated and sent to another group of $1 \times 1$ filters to fuse information altogether. Along with any possible path in which input features are transformed to output features, the equivalent receptive field increases whenever it passes a $3 \times 3$ filter, resulting in many equivalent feature scales due to combination effects.

One way of thinking of these blocks is that they expose a new dimension, **scale**,  alongside the existing dimensions of depth, width, and cardinality.

Res2Net Block

Res2Net: A New Multi-scale Backbone Architecture

The **Affine Operator** is an affine transformation layer introduced in the [ResMLP](https://paperswithcode.com/method/resmlp) architecture. This replaces [layer normalization](https://paperswithcode.com/method/layer-normalization), as in [Transformer based networks](https://paperswithcode.com/methods/category/transformers), which is possible since in the ResMLP, there are no [self-attention layers](https://paperswithcode.com/method/scaled) which makes training more stable - hence allowing a more simple affine transformation.

The affine operator is defined as:

$$ \operatorname{Aff}_{\mathbf{\alpha}, \mathbf{\beta}}(\mathbf{x})=\operatorname{Diag}(\mathbf{\alpha}) \mathbf{x}+\mathbf{\beta} $$

where $\alpha$ and $\beta$ are learnable weight vectors. This operation only rescales and shifts the input element-wise. This operation has several advantages over other normalization operations: first, as opposed to Layer Normalization, it has no cost at inference time, since it can absorbed in the adjacent linear layer. Second, as opposed to [BatchNorm](https://paperswithcode.com/method/batch-normalization) and Layer Normalization, the Aff operator does not depend on batch statistics.

Affine Operator

ResMLP: Feedforward networks for image classification with data-efficient training

**Shape Adaptor** is a novel resizing module for neural networks. It is a drop-in enhancement built on top of traditional resizing layers, such as pooling, bilinear sampling, and strided [convolution](https://paperswithcode.com/method/convolution). This module allows for a learnable shaping factor which differs from the traditional resizing layers that are fixed and deterministic.

Image Source: [Liu et al.](https://arxiv.org/pdf/2008.00892v2.pdf)

Shape Adaptor

Shape Adaptor: A Learnable Resizing Module

**GPipe** is a distributed model parallel method for neural networks. With GPipe, each model can be specified as a sequence of layers, and consecutive groups of layers can be partitioned into cells. Each cell is then placed on a separate accelerator. Based on this partitioned setup, batch splitting is applied. A mini-batch of training examples is split into smaller micro-batches, then the execution of each set of micro-batches is pipelined over cells. Synchronous mini-batch gradient descent is applied for training, where gradients are accumulated across all micro-batches in a mini-batch and applied at the end of a mini-batch.

GPipe

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

**AutoAugment** is an automated approach to find data augmentation policies from data. It formulates the problem of finding the best augmentation policy as a discrete search problem. It consists of two components: a search algorithm and a search space. 

At a high level, the search algorithm (implemented as a controller RNN) samples a data augmentation policy $S$, which has information about what image processing operation to use, the probability of using the operation in each batch, and the magnitude of the operation. The policy $S$ is used to train a neural network with a fixed architecture, whose validation accuracy $R$ is sent back to update the controller. Since $R$ is not differentiable, the controller will be updated by policy gradient methods. 

The operations used are from PIL, a popular Python image library: all functions in PIL that accept an image as input and output an image. It additionally uses two other augmentation techniques: [Cutout](https://paperswithcode.com/method/cutout) and SamplePairing. The operations searched over are ShearX/Y, TranslateX/Y, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout and Sample Pairing.

AutoAugment

AutoAugment: Learning Augmentation Policies from Data

Chain-of-thought prompts contain a series of intermediate reasoning steps, and they are shown to significantly improve the ability of large language models to perform certain tasks that involve complex reasoning (e.g., arithmetic, commonsense reasoning, symbolic reasoning, etc.)

CoT Prompting

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

**AdaGrad** is a stochastic optimization method that adapts the learning rate to the parameters. It performs smaller updates for parameters associated with frequently occurring features, and larger updates for parameters associated with infrequently occurring features. In its update rule, Adagrad modifies the general learning rate $\eta$ at each time step $t$ for every parameter $\theta\_{i}$ based on the past gradients for $\theta\_{i}$: 

$$ \theta\_{t+1, i} = \theta\_{t, i} - \frac{\eta}{\sqrt{G\_{t, ii} + \epsilon}}g\_{t, i} $$

The benefit of AdaGrad is that it eliminates the need to manually tune the learning rate; most leave it at a default value of $0.01$. Its main weakness is the accumulation of the squared gradients in the denominator. Since every added term is positive, the accumulated sum keeps growing during training, causing the learning rate to shrink and becoming infinitesimally small.

Image: [Alec Radford](https://twitter.com/alecrad)

AdaGrad

A **Split Attention** block enables attention across feature-map groups. As in [ResNeXt blocks](https://paperswithcode.com/method/resnext-block), the feature can be divided into several groups, and the number of feature-map groups is given by a cardinality hyperparameter $K$. The resulting feature-map groups are called cardinal groups. Split Attention blocks introduce a new radix hyperparameter $R$ that indicates the number of splits within a cardinal group, so the total number of feature groups is $G = KR$. We may apply a series of transformations {$\mathcal{F}\_1, \mathcal{F}\_2, \cdots\mathcal{F}\_G$} to each individual group, then the intermediate representation of each group is $U\_i = \mathcal{F}\_i\left(X\right)$, for $i \in$ {$1, 2, \cdots{G}$}.

A combined representation for each cardinal group can be obtained by fusing via an element-wise summation across multiple splits. The representation for $k$-th cardinal group is 
$\hat{U}^k = \sum_{j=R(k-1)+1}^{R k} U_j $, where $\hat{U}^k \in \mathbb{R}^{H\times W\times C/K}$ for $k\in{1,2,...K}$, and $H$, $W$ and $C$ are the block output feature-map sizes. 
Global contextual information with embedded channel-wise statistics can be gathered with [global average pooling](https://paperswithcode.com/method/global-average-pooling) across spatial dimensions  $s^k\in\mathbb{R}^{C/K}$. Here the $c$-th component is calculated as:

$$
    s^k\_c = \frac{1}{H\times W} \sum\_{i=1}^H\sum\_{j=1}^W \hat{U}^k\_c(i, j).
$$

A weighted fusion of the cardinal group representation $V^k\in\mathbb{R}^{H\times W\times C/K}$ is aggregated using [channel-wise soft attention](https://paperswithcode.com/method/channel-wise-soft-attention), where each feature-map channel is produced using a weighted combination over splits. The $c$-th channel is calculated as:

$$
    V^k_c=\sum_{i=1}^R a^k_i(c) U_{R(k-1)+i} ,
$$

where $a_i^k(c)$ denotes a (soft) assignment weight given by:

$$
a_i^k(c) =
\begin{cases}
  \frac{exp(\mathcal{G}^c_i(s^k))}{\sum_{j=0}^R exp(\mathcal{G}^c_j(s^k))} & \quad\textrm{if } R>1, \\
   \frac{1}{1+exp(-\mathcal{G}^c_i(s^k))} & \quad\textrm{if } R=1,\\
\end{cases}
$$

and mapping $\mathcal{G}_i^c$ determines the weight of each split for the $c$-th channel based on the global context representation $s^k$.

Split Attention

ResNeSt: Split-Attention Networks

**IoU-guided NMS** is a type of non-maximum suppression that help to eliminate the suppression failure caused by the misleading classification confidences. This is achieved through using the predicted IoU instead of the classification confidence as the ranking keyword for bounding boxes. 

IoU-guided NMS

Acquisition of Localization Confidence for Accurate Object Detection

**Synergistic Image and Feature Alignment** is an unsupervised domain adaptation framework that conducts synergistic alignment of domains from both image and feature perspectives. In SIFA, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features by leveraging adversarial learning in multiple aspects and with a deeply supervised mechanism. The feature encoder is shared between both adaptive perspectives to leverage their mutual benefits via end-to-end learning.

Viet-Anh on Software

AI Glossary

Tech Terminology

Absolute Position Encodings

Time-aware Large Kernel Convolution

Adaptive Content Generating and Preserving Network

Dynamic Keypoint Head

Bayesian Reward Extrapolation