**Demon CM**, or **SGD with Momentum and Demon**,  is the [Demon](https://paperswithcode.com/method/demon) momentum rule applied to [SGD with momentum](https://paperswithcode.com/method/sgd-with-momentum).

$$ \beta\_{t} = \beta\_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta\_{init}\right) + \beta\_{init}\left(1-\frac{t}{T}\right)} $$

$$ \theta\_{t+1} = \theta\_{t} - \eta{g}\_{t} + \beta\_{t}v\_{t} $$

$$ v\_{t+1} = \beta\_{t}{v\_{t}} - \eta{g\_{t}} $$

Demon CM

Demon: Improved Neural Network Training with Momentum Decay

Unified VLP is unified encoder-decoder model for general vision-language pre-training. The models uses a shared multi-layer transformers network for both encoding and decoding. The model is pre-trained on large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. Model architecture for pre-training. For pre-training , the input comprises of image input, sentence input, and three special tokens ([CLS], [SEP], [STOP]). The image is processed as $N$ Region of Interests (RoIs) and region features are extracted. The sentence is tokenized and masked with [MASK] tokens for the later masked language modeling task. The model consists of 12 layers of Transformer blocks, each having a masked self-attention layer and feed-forward module, where the self-attention mask controls what input context the prediction conditions on. Two self-attention masks are implemented depending on whether the objective is bidirectional or seq2seq. The model is fine-tuned for image captioning and visual question answering.

Unified VLP

Unified Vision-Language Pre-Training for Image Captioning and VQA

**MoCo v2** is an improved version of the [Momentum Contrast](https://paperswithcode.com/method/moco) self-supervised learning algorithm. Motivated by the findings presented in the [SimCLR](https://paperswithcode.com/method/simclr) paper, authors:

- Replace the 1-layer fully connected layer with a 2-layer MLP head with [ReLU](https://paperswithcode.com/method/relu) for the unsupervised training stage.
- Include blur augmentation.
- Use cosine learning rate schedule.

These modifications enable MoCo to outperform the state-of-the-art SimCLR with a smaller batch size and fewer epochs.

MoCo v2

Improved Baselines with Momentum Contrastive Learning

Conditional Flow Matching (CFM) is a fast way to train continuous normalizing flow (CNF) models. CFM is a simulation-free training objective for continuous normalizing flows that allows conditional generative modelling and speeds up training and inference.

Flow Matching

Flow Matching for Generative Modeling

Per the authors, Graph Isomorphism Network (GIN) generalizes the WL test and hence achieves maximum discriminative power among GNNs.

How Powerful are Graph Neural Networks?

A **PyramidNet** is a type of convolutional network where the key idea is to concentrate on the feature map dimension by increasing it gradually instead of by increasing it sharply at each residual unit with downsampling. In addition, the network architecture works as a mixture of both plain and residual networks by using zero-padded identity-mapping shortcut connections when increasing the feature map dimension.

PyramidNet

Deep Pyramidal Residual Networks

**DenseNAS-C** is a mobile convolutional neural network discovered through the [DenseNAS](https://paperswithcode.com/method/densenas) [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) method. The basic building block is MBConvs, or inverted bottleneck residuals, from the [MobileNet](https://paperswithcode.com/method/mobilenetv2) architectures.

DenseNAS-C

Densely Connected Search Space for More Flexible Neural Architecture Search

Given a pattern $P,$ that is more complicated than the patterns, we fragment $P$ into simpler patterns such that their exact count is known. In the subgraph GNN proposed earlier, look into the subgraph of the host graph. We have seen that this technique is scalable on large graphs. Also, we have seen that subgraph GNN is more expressive and efficient than traditional GNN. So, we tried to explore the expressibility when the pattern is fragmented into smaller subpatterns.

Fragmentation

Improving Expressivity of Graph Neural Networks using Localization

**Selective Search** is a region proposal algorithm for object detection tasks. It starts by over-segmenting the image based on intensity of the pixels using a graph-based segmentation method by Felzenszwalb and Huttenlocher. Selective Search then takes these oversegments as initial input and performs the following steps

1. Add all bounding boxes corresponding to segmented parts to the list of regional proposals
2. Group adjacent segments based on similarity
3. Go to step 1

At each iteration, larger segments are formed and added to the list of region proposals. Hence we create region proposals from smaller segments to larger segments in a bottom-up approach. This is what we mean by computing “hierarchical” segmentations using Felzenszwalb and Huttenlocher’s oversegments.

Selective Search

**Point-wise Spatial Attention (PSA)** is a [semantic segmentation](https://paperswithcode.com/task/semantic-segmentation) module. The goal is capture contextual information, especially in the long range, by aggregating information. Through the PSA module, information aggregation is performed as a kind of information flow where we adaptively learn a pixel-wise global attention map for each position from two perspectives to aggregate contextual information over the entire feature map.

The PSA module takes a spatial feature map $\mathbf{X}$ as input. We denote the spatial size of $\mathbf{X}$ as $H \times W$. Through the two branches as illustrated, we generate pixel-wise global attention maps for each position in feature map $\mathbf{X}$ through several convolutional layers.

We aggregate input feature maps based on attention maps to generate new feature representations with the long-range contextual information incorporated, i.e., $\mathbf{Z}\_{c}$ from the ‘collect’ branch and $\mathbf{Z}\_{d}$ from the ‘distribute’ branch.

We concatenate the new representations $\mathbf{Z}\_{c}$ and $\mathbf{Z}\_{d}$ and apply a convolutional layer with [batch normalization](https://paperswithcode.com/method/batch-normalization) and activation layers for dimension reduction and feature fusion. Then we concatenate the new global contextual feature with the local representation feature $\mathbf{X}$. It is followed by one or several convolutional layers with batch normalization and activation layers to generate the final feature map for following subnetworks.

Point-wise Spatial Attention

PSANet: Point-wise Spatial Attention Network for Scene Parsing

**Swish** is an activation function, $f(x) = x \cdot \text{sigmoid}(\beta x)$, where $\beta$ a learnable parameter. Nearly all implementations do not use the learnable parameter $\beta$, in which case the activation function is $x\sigma(x)$ ("Swish-1").

The function $x\sigma(x)$ is exactly the [SiLU](https://paperswithcode.com/method/silu), which was introduced by other authors before the swish.
See [Gaussian Error Linear Units](https://arxiv.org/abs/1606.08415) ([GELUs](https://paperswithcode.com/method/gelu)) where the SiLU (Sigmoid Linear Unit) was originally coined, and see [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://arxiv.org/abs/1702.03118) and [Swish: a Self-Gated Activation Function](https://arxiv.org/abs/1710.05941v1) where the same activation function was experimented with later.

Swish

Searching for Activation Functions

**SchNet** is an end-to-end deep neural network architecture based on continuous-filter convolutions. It follows the deep tensor neural network framework, i.e. atom-wise representations are constructed by starting from embedding vectors that characterize the atom type before introducing the configuration of the system by a series of interaction blocks.

SchNet

SchNet: A continuous-filter convolutional neural network for modeling quantum interactions

**Discriminative Fine-Tuning** is a fine-tuning strategy that is used for [ULMFiT](https://paperswithcode.com/method/ulmfit) type models. Instead of using the same learning rate for all layers of the model, discriminative fine-tuning allows us to tune each layer with different learning rates. For context, the regular stochastic gradient descent ([SGD](https://paperswithcode.com/method/sgd)) update of a model’s parameters $\theta$ at time step $t$ looks like the following (Ruder, 2016):

$$ \theta\_{t} = \theta\_{t-1} − \eta\cdot\nabla\_{\theta}J\left(\theta\right)$$

where $\eta$ is the learning rate and $\nabla\_{\theta}J\left(\theta\right)$ is the gradient with regard to the model’s objective function. For discriminative fine-tuning, we split the parameters $\theta$ into {$\theta\_{1}, \ldots, \theta\_{L}$} where $\theta\_{l}$ contains the parameters of the model at the $l$-th layer and $L$ is the number of layers of the model. Similarly, we obtain {$\eta\_{1}, \ldots, \eta\_{L}$} where $\theta\_{l}$ where $\eta\_{l}$ is the learning rate of the $l$-th layer. The SGD update with discriminative finetuning is then:

$$ \theta\_{t}^{l} = \theta\_{t-1}^{l} - \eta^{l}\cdot\nabla\_{\theta^{l}}J\left(\theta\right) $$

The authors find that empirically it worked well to first choose the learning rate $\eta^{L}$ of the last layer by fine-tuning only the last layer and using $\eta^{l-1}=\eta^{l}/2.6$ as the learning rate for lower layers.

Discriminative Fine-Tuning

Universal Language Model Fine-tuning for Text Classification

A **Relativistic GAN** is a type of generative adversarial network. It has a relativistic discriminator which estimates the probability that the given real data is more realistic than a randomly sampled fake data. The idea is to endow GANs with the property that the probability of real data being real ($D\left(x\_{r}\right)$) should decrease as the probability of fake data being real ($D\left(x\_{f}\right)$) increases.

With a standard [GAN](https://paperswithcode.com/method/gan), we can achieve this as follows. The standard GAN discriminator can be defined, in term of the non-transformed layer $C\left(x\right)$, as $D\left(x\right) = \text{sigmoid}\left(C\left(x\right)\right)$. A simple way to make discriminator relativistic - having the output of $D$ depend on both real and fake data - is to sample from real/fake data pairs $\tilde{x} = \left(x\_{r}, x\_{f}\right)$ and define it as $D\left(\tilde{x}\right) = \text{sigmoid}\left(C\left(x\_{r}\right) − C\left(x\_{f}\right)\right)$. The modification can be interpreted as: the discriminator estimates the probability
that the given real data is more realistic than a randomly sampled fake data.

More generally a Relativistic GAN can be interpreted as having a discriminator of the form $a\left(C\left(x\_{r}\right)−C\left(x\_{f}\right)\right)$, where $a$ is the activation function, to be relativistic.

Relativistic GAN

The relativistic discriminator: a key element missing from standard GAN

A **Dynamic Memory Network** is a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations. These results are then reasoned over in a hierarchical recurrent sequence model to generate answers. 

The DMN consists of a number of modules:

- Input Module: The input module encodes raw text inputs from the task into distributed vector representations. The input takes forms like a sentence, a long story, a movie review and so on.
- Question Module: The question module encodes the question of the task into a distributed
vector representation. For question answering, the question may be a sentence such as "Where did the author first fly?". The representation is fed into the episodic memory module, and forms the basis, or initial state, upon which the episodic memory module iterates.
- Episodic Memory Module: Given a collection of input representations, the episodic memory module chooses which parts of the inputs to focus on through the attention mechanism. It then produces a ”memory” vector representation taking into account the question as well as the previous memory. Each iteration provides the module with newly relevant information about the input. In other words,
the module has the ability to retrieve new information, in the form of input representations, which were thought to be irrelevant in previous iterations.
- Answer Module: The answer module generates an answer from the final memory vector of the memory module.

Dynamic Memory Network

Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

A **Style-based Recalibration Module (SRM)** is a module for convolutional neural networks that adaptively recalibrates intermediate feature maps by exploiting their styles. SRM first extracts the style information from each channel of the feature maps by style pooling, then estimates per-channel recalibration weight via channel-independent style integration. By incorporating the relative importance of individual styles into feature maps, SRM is aimed at enhancing the representational ability of a CNN.

The overall structure of SRM is illustrated in the Figure to the right. It consists of two main components: style pooling and style integration. The style pooling operator extracts style features
from each channel by summarizing feature responses across spatial dimensions. It is followed by the style integration operator, which produces example-specific style weights by utilizing the style features via channel-wise operation. The style weights finally recalibrate the feature maps to either
emphasize or suppress their information.

Style-based Recalibration Module

SRM : A Style-based Recalibration Module for Convolutional Neural Networks

An effective approach to the quantification of patchiness in biomedical images according to their local edge densities.

PALED

Segmentation of patchy areas in biomedical images based on local edge density estimation

A new kind of implicit models, where the output of the network is defined as the solution to an "infinite-level" fixed point equation. Thanks to this we can compute the gradient of the output without activations and therefore with a significantly reduced memory footprint.

AI Glossary