**Contrastive Predictive Coding (CPC)** learns self-supervised representations by predicting the future in latent space by using powerful autoregressive models. The model uses a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful
to predict future samples.

First, a non-linear encoder $g\_{enc}$ maps the input sequence of observations $x\_{t}$ to a sequence of latent representations $z\_{t} = g\_{enc}\left(x\_{t}\right)$, potentially with a lower temporal resolution. Next, an autoregressive model $g\_{ar}$ summarizes all $z\leq{t}$ in the latent space and produces a context latent representation $c\_{t} = g\_{ar}\left(z\leq{t}\right)$.

A density ratio is modelled which preserves the mutual information between $x\_{t+k}$ and $c\_{t}$ as follows:

$$ f\_{k}\left(x\_{t+k}, c\_{t}\right) \propto \frac{p\left(x\_{t+k}|c\_{t}\right)}{p\left(x\_{t+k}\right)} $$

where $\propto$ stands for ’proportional to’ (i.e. up to a multiplicative constant). Note that the density ratio $f$ can be unnormalized (does not have to integrate to 1). The authors use a simple log-bilinear model:

$$ f\_{k}\left(x\_{t+k}, c\_{t}\right) = \exp\left(z^{T}\_{t+k}W\_{k}c\_{t}\right) $$

Any type of autoencoder and autoregressive can be used. An example the authors opt for is strided convolutional layers with residual blocks and GRUs.

The autoencoder and autoregressive models are trained to minimize an [InfoNCE](https://paperswithcode.com/method/infonce) loss (see components).

Contrastive Predictive Coding

Representation Learning with Contrastive Predictive Coding

A **Unitary RNN** is a recurrent neural network architecture that uses a unitary hidden to hidden matrix. Specifically they concern dynamics of the form:

$$ h\_{t} = f\left(Wh\_{t−1} + Vx\_{t}\right) $$

where $W$ is a unitary matrix $\left(W^{†}W = I\right)$. The product of unitary matrices is a unitary matrix, so $W$ can be parameterised as a product of simpler unitary matrices:

$$ h\_{t} = f\left(D\_{3}R\_{2}F^{−1}D\_{2}PR\_{1}FD\_{1}h\_{t−1} + Vxt\right) $$

where $D\_{3}$, $D\_{2}$, $D\_{1}$ are learned diagonal complex matrices, and $R\_{2}$, $R\_{1}$ are learned reflection matrices. Matrices $F$ and $F^{−1}$ are the discrete Fourier transformation and its inverse. P is any constant random permutation. The activation function $f\left(h\right)$ applies a rectified linear unit with a learned bias to the modulus of each complex number. Only
the diagonal and reflection matrices, $D$ and $R$, are learned, so Unitary RNNs have fewer parameters than [LSTMs](https://paperswithcode.com/method/lstm) with comparable numbers of hidden units.

Source: [Associative LSTMs](https://arxiv.org/pdf/1602.03032.pdf)

Unitary RNN

Unitary Evolution Recurrent Neural Networks

\begin{equation}
DiceLoss\left( y, \overline{p} \right) = 1 - \dfrac{\left(  2y\overline{p} + 1 \right)} {\left( y+\overline{p } + 1 \right)}
\end{equation}

Dice Loss

Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations

**mT0** is a Multitask prompted finetuning (MTF) variant of mT5.

Crosslingual Generalization through Multitask Finetuning

**Blender** is a proposal-based instance mask generation module which incorporates rich instance-level information with accurate dense pixel features. A single [convolution](https://paperswithcode.com/method/convolution) layer is added on top of the detection towers to produce attention masks along with each bounding box prediction. For each predicted instance, the blender crops predicted bases with its bounding box and linearly combines them according the learned attention maps.

The inputs of the blender module are bottom-level bases $\mathbf{B}$, the selected top-level attentions $A$ and bounding box proposals $P$. First [RoIPool](https://paperswithcode.com/method/roi-pooling) of Mask R-CNN to crop bases with each proposal $\mathbf{p}\_{d}$ and then resize the region to a fixed size $R \times R$ feature map $\mathbf{r}\_{d}$

$$
\mathbf{r}\_{d}=\operatorname{RoIPool}_{R \times R}\left(\mathbf{B}, \mathbf{p}\_{d}\right), \quad \forall d \in\{1 \ldots D\}
$$

More specifically,  asampling ratio 1 is used for [RoIAlign](https://paperswithcode.com/method/roi-align), i.e. one bin for each sampling point. During training, ground truth boxes are used as the proposals. During inference, [FCOS](https://paperswithcode.com/method/fcos) prediction results are used.

The attention size $M$ is smaller than $R$. We interpolate $\mathbf{a}\_{d}$ from $M \times M$ to $R \times R$, into the shapes of $R=\left\(\mathbf{r}\_{d} \mid d=1 \ldots D\right)$

$$
\mathbf{a}\_{d}^{\prime}=\text { interpolate }\_{M \times M \rightarrow R \times R}\left(\mathbf{a}\_{d}\right), \quad \forall d \in\{1 \ldots D\}
$$

Then $\mathbf{a}\_{d}^{\prime}$ is normalized with a softmax function along the $K$ dimension to make it a set of score maps $\mathbf{s}\_{d}$.

$$
\mathbf{s}\_{d}=\operatorname{softmax}\left(\mathbf{a}\_{d}^{\prime}\right), \quad \forall d \in\{1 \ldots D\}
$$

Then we apply element-wise product between each entity $\mathbf{r}\_{d}, \mathbf{s}\_{d}$ of the regions $R$ and scores $S$, and sum along the $K$ dimension to get our mask logit $\mathbf{m}\_{d}:$

$$
\mathbf{m}\_{d}=\sum\_{k=1}^{K} \mathbf{s}\_{d}^{k} \circ \mathbf{r}\_{d}^{k}, \quad \forall d \in\{1 \ldots D\}
$$

where $k$ is the index of the basis. The mask blending process with $K=4$ is visualized in the Figure.

Blender

BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation

**Factorized Dense Synthesized Attention** is a synthesized attention mechanism, similar to [dense synthesized attention](https://paperswithcode.com/method/dense-synthesized-attention), but we factorize the outputs to reduce parameters and prevent overfitting. It was proposed as part of the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture. The factorized variant of the dense synthesizer can be expressed as follows:

$$A, B = F\_{A}\left(X\_{i}\right), F\_{B}\left(X\_{i}\right)$$

where $F\_{A}\left(.\right)$ projects input $X\_{i}$ into $a$ dimensions, $F\_B\left(.\right)$ projects $X\_{i}$ to $b$ dimensions, and $a \text{ x } b = l$. The output of the factorized module is now written as:

$$ Y = \text{Softmax}\left(C\right)G\left(X\right) $$

where $C = H\_{A}\left(A\right) * H\_{B}\left(B\right)$, where $H\_{A}$, $H\_{B}$ are tiling functions and $C \in \mathbb{R}^{l \text{ x } l}$. The tiling function simply duplicates the vector $k$ times, i.e., $\mathbb{R}^{l} \rightarrow \mathbb{R}^{lk}$. In this case, $H\_{A}\left(\right)$ is a projection of $\mathbb{R}^{a} \rightarrow \mathbb{R}^{ab}$ and $H\_{B}\left(\right)$ is a projection of $\mathbb{R}^{b} \rightarrow \mathbb{R}^{ba}$. To avoid having similar values within the same block, we compose the outputs of $H\_{A}$ and $H\_{B}$.

Factorized Dense Synthesized Attention

Synthesizer: Rethinking Self-Attention in Transformer Models

**Conditional Instance Normalization** is a normalization technique where all convolutional weights of a style transfer network are shared across many styles.  The goal of the procedure is transform
a layer’s activations $x$ into a normalized activation $z$ specific to painting style $s$. Building off
[instance normalization](https://paperswithcode.com/method/instance-normalization), we augment the $\gamma$ and $\beta$ parameters so that they’re $N \times C$ matrices, where $N$ is the number of styles being modeled and $C$ is the number of output feature maps. Conditioning on a style is achieved as follows:

$$ z = \gamma\_{s}\left(\frac{x - \mu}{\sigma}\right) + \beta\_{s}$$

where $\mu$ and $\sigma$ are $x$’s mean and standard deviation taken across spatial axes and $\gamma\_{s}$ and $\beta\_{s}$ are obtained by selecting the row corresponding to $s$ in the $\gamma$ and $\beta$ matrices. One added benefit of this approach is that one can stylize a single image into $N$ painting styles with a single feed forward pass of the network with a batch size of $N$.

Conditional Instance Normalization

A Learned Representation For Artistic Style

A **Deactivable Skip Connection** is a type of skip connection which, instead of concatenating the encoder features
(red) and decoder features (blue), as with [standard skip connections](https://paperswithcode.com/methods/category/skip-connections), it instead fuses the encoder features with part of the decoder features (light blue), to be able to deactivate this operation when needed.

Deactivable Skip Connection

Cross-modal Deep Face Normals with Deactivable Skip Connections

**Randomized Leaky Rectified Linear Units**, or **RReLU**, are an activation function that randomly samples the negative slope for activation values. It was first proposed and used in the Kaggle NDSB Competition. During training, $a\_{ji}$ is a random number sampled from a uniform distribution $U\left(l, u\right)$. Formally:

$$ y\_{ji} = x\_{ji} \text{   if } x\_{ji} \geq{0} $$
$$ y\_{ji} = a\_{ji}x\_{ji} \text{   if } x\_{ji} < 0 $$

where

$$\alpha\_{ji} \sim U\left(l, u\right), l < u \text{ and } l, u \in \left[0,1\right)$$

In the test phase, we take average of all the $a\_{ji}$ in training similar to [dropout](https://paperswithcode.com/method/dropout), and thus set $a\_{ji}$ to $\frac{l+u}{2}$ to get a deterministic result. As suggested by the NDSB competition winner, $a\_{ji}$ is sampled from $U\left(3, 8\right)$. 

At test time, we use:

$$ y\_{ji} = \frac{x\_{ji}}{\frac{l+u}{2}} $$

RReLU

Empirical Evaluation of Rectified Activations in Convolutional Network

A **Non-Local Block** is an image block module used in neural networks that wraps a [non-local operation](https://paperswithcode.com/method/non-local-operation). We can define a non-local block as:

$$ \mathbb{z}\_{i} = W\_{z}\mathbb{y\_{i}} + \mathbb{x}\_{i} $$

where $y\_{i}$ is the output from the non-local operation and $+ \mathbb{x}\_{i}$ is a [residual connection](https://paperswithcode.com/method/residual-connection).

Non-Local Block

Non-local Neural Networks

**CenterMask** is an anchor-free instance segmentation method that adds a novel [spatial attention-guided mask](https://paperswithcode.com/method/spatial-attention-guided-mask) (SAG-Mask) branch to anchor-free one stage object detector ([FCOS](https://paperswithcode.com/method/fcos)) in the same vein with [Mask R-CNN](https://paperswithcode.com/method/mask-r-cnn). Plugged into the FCOS object detector, the SAG-Mask branch predicts a segmentation mask on each detected box with the spatial attention map that helps to focus on informative pixels and suppress noise.

CenterMask

CenterMask : Real-Time Anchor-Free Instance Segmentation

The **Self-Attention Generative Adversarial Network**, or **SAGAN**, allows for attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other.

SAGAN

Self-Attention Generative Adversarial Networks

**Region of Interest Pooling**, or **RoIPool**, is an operation for extracting a small feature map (e.g., $7×7$) from each RoI in detection and segmentation based tasks. Features are extracted from each candidate box, and thereafter in models like [Fast R-CNN](https://paperswithcode.com/method/fast-r-cnn), are then classified and bounding box regression performed.

The actual scaling to, e.g., $7×7$, occurs by dividing the region proposal into equally sized sections, finding the largest value in each section, and then copying these max values to the output buffer. In essence, **RoIPool** is [max pooling](https://paperswithcode.com/method/max-pooling) on a discrete grid based on a box.

Image Source: [Joyce Xu](https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9)

RoIPool

Rich feature hierarchies for accurate object detection and semantic segmentation

**Graph Path Feature Learning** is a probabilistic rule learner optimized to mine instantiated first-order logic rules from knowledge graphs. Instantiated rules contain constants extracted from KGs. Compared to abstract rules that contain no constants, instantiated rules are capable of explaining and expressing concepts in more detail. GPFL utilizes a novel two-stage rule generation mechanism that first generalizes extracted paths into templates that are acyclic abstract rules until a certain degree of template saturation is achieved, then specializes the generated templates into instantiated rules.

GPFL

Towards Learning Instantiated Logical Rules from Knowledge Graphs

**DynaBERT** is a [BERT](https://paperswithcode.com/method/bert)-variant which can flexibly adjust the size and latency by selecting adaptive width and depth. The training process of DynaBERT includes first training a width-adaptive BERT and then allowing both adaptive width and depth, by distilling knowledge from the full-sized model to small sub-networks. Network rewiring is also used to keep the more important attention heads and neurons shared by more sub-networks. 

A two-stage procedure is used to train DynaBERT. First, using knowledge distillation (dashed lines) to transfer the knowledge from a fixed teacher model to student sub-networks with adaptive width in DynaBERTW. Then, using knowledge distillation (dashed lines) to transfer the knowledge from a trained DynaBERTW to student sub-networks with adaptive width and depth in DynaBERT.

DynaBERT

DynaBERT: Dynamic BERT with Adaptive Width and Depth

**Precise RoI Pooling**, or **PrRoI Pooling**, is a region of interest feature extractor that avoids any quantization of coordinates and has a continuous gradient on bounding box coordinates. Given the feature map $\mathcal{F}$ before RoI/PrRoI Pooling (eg from Conv4 in [ResNet](https://paperswithcode.com/method/resnet)-50), let $w_{i,j}$ be the feature at one discrete location $(i,j)$ on the feature map. Using bilinear interpolation, the discrete feature map can be considered continuous at any continuous coordinates $(x,y)$:

$$
f(x,y) = \sum_{i,j}IC(x,y,i,j) \times w_{i,j},
$$

where $IC(x,y,i,j) = max(0,1-|x-i|)\times max(0,1-|y-j|)$ is the interpolation coefficient. Then denote a bin of a RoI as $bin=\{(x_1,y_1),(x_2,y_2)\}$, where $(x_1,y_1)$ and $(x_2,y_2)$ are the continuous coordinates of the top-left and bottom-right points, respectively. We perform pooling (e.g. [average pooling](https://paperswithcode.com/method/average-pooling)) given $bin$ and feature map $\mathcal{F}$ by computing a two-order integral:

Precise RoI Pooling

Acquisition of Localization Confidence for Accurate Object Detection

**Shrink and Fine-Tune**, or **SFT**, is a type of distillation that avoids explicit distillation by copying parameters to a student student model and then fine-tuning. Specifically it extracts a student model from the maximally spaced layers of a fine-tuned teacher. Each layer $l \in L'$ is copied fully from $L$. For example, when creating a [BART](https://paperswithcode.com/method/bart) student with 3 decoder layers from the 12 encoder layer 12 decoder layer teacher, we copy the teacher’s full $Enc^{L}$ and decoder layers 0, 6, and 11 to the student. When deciding which layers to copy, we break ties arbitrarily; copying layers 0, 5, and 11 might work just as well. When copy only 1 decoder layer, we copy layer 0. This was found this to work better than copying layer 11. The impact of initialization on performance is measured experimentally in Section 6.1. After initialization, the student model continues to fine-tune on the summarization dataset, with the objective of minimizing $\mathcal{L}\_{Data}$.

Pre-trained Summarization Distillation

**Layer-Sequential Unit-Variance Initialization** (**LSUV**) is a simple method for weight initialization for deep net learning. The initialization strategy involves the following two step:

1) First, pre-initialize weights of each [convolution](https://paperswithcode.com/method/convolution) or inner-product layer with
orthonormal matrices. 

2) Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one.

AI Glossary