**Parameterized Exponential Linear Units**, or **PELU**, is an activation function for neural networks. It involves learning a parameterization of [ELU](https://paperswithcode.com/method/elu) in order to learn the proper activation shape at each layer in a CNN. 

The PELU has two additional parameters over the ELU:

$$ f\left(x\right) = cx \text{ if } x > 0 $$
$$ f\left(x\right) = \alpha\exp^{\frac{x}{b}} - 1 \text{ if } x \leq 0 $$

Where $a$, $b$, and $c > 0$. Here $c$ causes a change in the slope in the positive quadrant, $b$ controls the scale of the [exponential decay](https://paperswithcode.com/method/exponential-decay), and $\alpha$ controls the saturation in the negative quadrant.

Source: [Activation Functions](https://arxiv.org/pdf/1811.03378.pdf)

PELU

Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

**Capsule Network** is a machine learning system that is a type of artificial neural network that can be used to better model hierarchical relationships. The approach is an attempt to more closely mimic biological neural organization.

CapsNet

Dynamic Routing Between Capsules

Trainable activation function as a sigmoid-based generalization of ReLU, Swish and SiLU.

AHAF

Adaptive hybrid activation function for deep neural networks

**ZeRO-Infinity** is a sharded data parallel system that extends [ZeRO](https://paperswithcode.com/method/zero) with new innovations in heterogeneous memory access called the infinity offload engine. This allows ZeRO-Infinity to support massive model sizes on limited GPU resources by exploiting CPU and NVMe memory simultaneously. In addition, ZeRO-Infinity also introduces a novel GPU memory optimization technique called memory-centric tiling to support extremely large individual layers that would otherwise not fit in GPU memory even one layer at a time.

ZeRO-Infinity

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

**Partition Filter Network** is a framework designed specifically for joint entity and relation extraction. The framework consists of three components: partition filter encoder, NER unit and RE unit. In task units, we use table-filling for word pair prediction. Orange, yellow and green represents NER-related, shared and RE-related component or features. (b) Detailed depiction of partition filter encoder in one single time step. We decompose feature encoding into two steps: partition and filter (shown in the gray area). In partition, we first segment neurons into two task partitions and one shared partition. Then in filter, partitions are selected and combined to form task-specific features and shared features, filtering out information irrelevant to each task.

Partition Filter Network

A Partition Filter Network for Joint Entity and Relation Extraction

**Assemble-ResNet** is a modification to the [ResNet](https://paperswithcode.com/method/resnet) architecture with several tweaks including using [ResNet-D](https://paperswithcode.com/method/resnet-d), channel attention, [anti-alias downsampling](https://paperswithcode.com/method/anti-alias-downsampling), and Big Little Networks.

Assemble-ResNet

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

The **Recurrent Entity Network** is equipped with a dynamic long-term memory which allows it to maintain and update a representation of the state of the world as it receives new data. For language understanding tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a [Memory Network](https://paperswithcode.com/method/memory-network). Like a [Neural Turing Machine](https://paperswithcode.com/method/neural-turing-machine) or Differentiable Neural Computer, it maintains a fixed size memory and can learn to perform location and content-based read and write operations.  However, unlike those models it has a simple parallel  architecture in which several memory locations can be updated simultaneously. 

The model consists of a fixed number of dynamic memory cells, each containing a vector key $w_j$ and a vector value (or content) $h_j$. Each cell is associated with its own processor, a simple gated recurrent network that may update the cell value given an input. If each cell learns to represent a concept or entity in the world, one can imagine a gating mechanism that, based on the key and content of the memory cells, will only modify the cells that concern the entities mentioned in the input. There is no direct interaction between the memory cells, hence the system can be seen as multiple identical processors functioning in parallel, with distributed local memory. 

The sharing of these parameters reflects an invariance of these laws across object instances, similarly to how the [weight tying](https://paperswithcode.com/method/weight-tying) scheme in a CNN reflects an invariance of image statistics across locations. Their hidden state is updated only when new information relevant to their concept is received, and remains otherwise unchanged. The keys used in the addressing/gating mechanism also correspond to concepts or entities, but are modified only during learning, not during inference.

Recurrent Entity Network

Tracking the World State with Recurrent Entity Networks

The **Maxout Unit** is a generalization of the [ReLU](https://paperswithcode.com/method/relu) and the [leaky ReLU](https://paperswithcode.com/method/leaky-relu) functions. It is a piecewise linear function that returns the maximum of the inputs, designed to be used in conjunction with [dropout](https://paperswithcode.com/method/dropout). Both ReLU and leaky ReLU are special cases of Maxout. 

$$f\left(x\right) = \max\left(w^{T}\_{1}x + b\_{1}, w^{T}\_{2}x + b\_{2}\right)$$

The main drawback of Maxout is that it is computationally expensive as it doubles the number of parameters for each neuron.

Maxout

Maxout Networks

**ProGAN**, or **Progressively Growing GAN**, is a generative adversarial network that utilises a progressively growing training approach. The idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses.

ProGAN

Progressive Growing of GANs for Improved Quality, Stability, and Variation

**ALiBi**, or **Attention with Linear Biases**, is a [positioning method](https://paperswithcode.com/methods/category/position-embeddings) that allows [Transformer](https://paperswithcode.com/methods/category/transformers) language models to consume, at inference time, sequences which are longer than the ones they were trained on. 

ALiBi does this without using actual position embeddings. Instead, computing the attention between a certain key and query, ALiBi penalizes the attention value that that query can assign to the key depending on how far away the key and query are. So when a key and query are close by, the penalty is very low, and when they are far away, the penalty is very high. 

This method was motivated by the simple reasoning that words that are close-by matter much more than ones that are  far away.

This method is as fast as the sinusoidal or absolute embedding methods (the fastest positioning methods there are). It outperforms those methods and Rotary embeddings when evaluating sequences that are longer than the ones the model was trained on (this is known as extrapolation).

ALiBi

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

**Decorrelated Batch Normalization (DBN)** 
is a normalization technique which not just centers and scales activations but whitens them. ZCA whitening instead of [PCA](https://paperswithcode.com/method/pca) whitening is employed since PCA whitening causes a problem called *stochastic axis swapping*, which is detrimental to learning.

Decorrelated Batch Normalization

**Feature Fusion Module v1** is a feature fusion module from the [M2Det](https://paperswithcode.com/method/m2det) object detection model, and feature fusion modules are crucial for constructing the final multi-level feature pyramid. They use [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) layers to compress the channels of the input features and use concatenation operation to aggregate these feature map. FFMv1 takes two feature maps with different scales in backbone as input, it adopts one upsample operation to rescale the deep features to the same scale before the concatenation operation.

FFMv1

M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network

**Squared ReLU** is an activation function used in the [Primer](https://paperswithcode.com/method/primer) architecture in the feedforward block of the [Transformer](https://paperswithcode.com/methods/category/transformers) layer. It is simply squared [ReLU](https://paperswithcode.com/method/relu) activations.

The effectiveness of higher order polynomials can also be observed in other effective [Transformer](https://paperswithcode.com/method/transformer) nonlinearities, such as [GLU](https://paperswithcode.com/method/glu) variants like [ReGLU](https://paperswithcode.com/method/reglu) and point-wise activations like [approximate GELU](https://paperswithcode.com/method/gelu). However, squared ReLU has drastically different asymptotics as $x \rightarrow \inf$ compared to the most commonly used activation functions: [ReLU](https://paperswithcode.com/method/relu), [GELU](https://paperswithcode.com/method/gelu) and [Swish](https://paperswithcode.com/method/swish). Squared ReLU does have significant overlap with ReGLU and in fact is equivalent when ReGLU’s $U$ and $V$ weight matrices are the same and squared ReLU is immediately preceded by a linear transformation with weight matrix $U$. This leads the authors to believe that squared ReLUs capture the benefits of these GLU variants, while being simpler, without additional parameters, and delivering better quality.

Squared ReLU

Primer: Searching for Efficient Transformers for Language Modeling

A **Gated Recurrent Unit**, or **GRU**, is a type of recurrent neural network. It is similar to an [LSTM](https://paperswithcode.com/method/lstm), but only has two gates - a reset gate and an update gate - and notably lacks an output gate. Fewer parameters means GRUs are generally easier/faster to train than their LSTM counterparts.

Image Source: [here](https://www.google.com/url?sa=i&url=https%3A%2F%2Fcommons.wikimedia.org%2Fwiki%2FFile%3AGated_Recurrent_Unit%2C_type_1.svg&psig=AOvVaw3EmNX8QXC5hvyxeenmJIUn&ust=1590332062671000&source=images&cd=vfe&ved=0CA0QjhxqFwoTCMiev9-eyukCFQAAAAAdAAAAABAR)

Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

**Transformer-XL** (meaning extra long) is a [Transformer](https://paperswithcode.com/method/transformer) architecture that introduces the notion of recurrence to the deep self-attention network. Instead of computing the hidden states from scratch for each new segment, Transformer-XL reuses the hidden states obtained in previous segments. The reused hidden states serve as memory for the current segment, which builds up a recurrent connection between the segments. As a result, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections. As an additional contribution, the Transformer-XL uses a new relative positional encoding formulation that generalizes to attention lengths longer than the one observed during training.

Transformer-XL

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

**MultiGrain** is a type of image model that learns a single embedding for classes, instances and copies.  In other words, it is a convolutional neural network that is suitable for both image classification and instance retrieval. We learn MultiGrain by jointly training an image embedding for multiple tasks. The resulting representation is compact and can outperform narrowly-trained embeddings. The learned embedding output incorporates different levels of granularity.

MultiGrain

MultiGrain: a unified image embedding for classes and instances

**ReLU6** is a modification of the [rectified linear unit](https://paperswithcode.com/method/relu) where we limit the activation to a maximum size of $6$. This is due to increased robustness when used with low-precision computation.

Image Credit: [PyTorch](https://pytorch.org/docs/master/generated/torch.nn.ReLU6.html)

ReLU6

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

**UFLoss**, or **Unsupervised Feature Loss**, is a patch-based unsupervised learned feature loss for deep learning (DL) based reconstructions. The UFLoss provides instance-level discrimination by mapping similar instances to similar low-dimensional feature vectors using a pre-trained mapping network (UFLoss Network). The rationale of using features from large-patches (typically 40×40 pixels for a 300×300 pixels image) is that we want the UFLoss to capture mid-level structural and semantic features instead of using small patches (typically around 10×10 pixels), which only contain local edge information. On the other hand, the authors avoid using global features due to the fact that the training set (typically around 5000 slices) is usually not large enough to capture common and general features at a large-image scale.

Viet-Anh on Software

AI Glossary

Tech Terminology

Parametric Exponential Linear Unit

Capsule Network

Adaptive Hybrid Activation Function

ZeRO-Infinity

Partition Filter Network

Assemble-ResNet

Recurrent Entity Network

Maxout

Progressively Growing GAN

Attention with Linear Biases

Decorrelated Batch Normalization

Feature Fusion Module v1

Squared ReLU

Gated Recurrent Unit

Transformer-XL

MultiGrain

ReLU6

Unsupervised Feature Loss

Viet-Anh on Software