**Convolutional Block Attention Module (CBAM)** is an attention module for convolutional neural networks. Given an intermediate feature map, the module sequentially infers attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature refinement.

Given an intermediate feature map $\mathbf{F} \in \mathbb{R}^{C×H×W}$ as input, CBAM sequentially infers a 1D channel attention map $\mathbf{M}\_{c} \in \mathbb{R}^{C×1×1}$ and a 2D spatial attention map $\mathbf{M}\_{s} \in \mathbb{R}^{1×H×W}$. The overall attention process can be summarized as:

$$ \mathbf{F}' = \mathbf{M}\_{c}\left(\mathbf{F}\right) \otimes \mathbf{F} $$

$$ \mathbf{F}'' = \mathbf{M}\_{s}\left(\mathbf{F'}\right) \otimes \mathbf{F'} $$

During multiplication, the attention values are broadcasted (copied) accordingly: channel attention values are broadcasted along the spatial dimension, and vice versa. $\mathbf{F}''$ is the final refined
output.

CBAM

CBAM: Convolutional Block Attention Module

Multi-task learning (MTL) introduces an inductive bias, based on a-priori relations between tasks: the trainable model is compelled to model more general dependencies by using the abovementioned relation as an important data feature. Hierarchical MTL, in which different tasks use different levels of the deep neural network, provides more effective inductive bias compared to “flat” MTL. Also, hierarchical MTL helps to solve the vanishing gradient problem in deep learning.

Hierarchical MTL

Deep multi-task learning with low level tasks supervised at lower layers

**Attentional Liquid Warping Block**, or **AttLWB**, is a module for human image synthesis GANs that propagates the source information - such as texture, style, color and face identity - in both image and feature spaces to the synthesized reference. It firstly learns similarities of the global features among all multiple sources features, and then it fuses the multiple sources features by a linear combination of the learned similarities and the multiple sources in the feature spaces. Finally, to better propagate the source identity (style, color, and texture) into the global stream, the fused source features are warped to the global stream by [Spatially-Adaptive Normalization](https://paperswithcode.com/method/spade) (SPADE).

AttLWB

Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis

**Early Stopping** is a regularization technique for deep neural networks that stops training when parameter updates no longer begin to yield improves on a validation set. In essence, we store and update the current best parameters during training, and when parameter updates no longer yield an improvement (after a set number of iterations) we stop training and use the last best parameters. It works as a regularizer by restricting the optimization procedure to a smaller volume of parameter space.

Image Source: [Ramazan Gençay](https://www.researchgate.net/figure/Early-stopping-based-on-cross-validation_fig1_3302948)

Early Stopping

FIERCE is an entropic regularization on the **feature** space

FIERCE

Preserving Fine-Grain Feature Information in Classification via Entropic Regularization

**Local Patch Interaction**, or **LPI**, is a module used for the [XCiT layer](https://paperswithcode.com/method/xcit-layer) to enable explicit communication across patches. LPI consists of two [depth-wise 3×3 convolutional layers](https://paperswithcode.com/method/depthwise-convolution) with [Batch Normalization](https://paperswithcode.com/method/batch-normalization) and [GELU](https://paperswithcode.com/method/gelu) non-linearity in between. Due to its depth-wise structure, the LPI block has a negligible overhead in terms of parameters, as well as a limited overhead in terms of throughput and memory usage during inference.

Local Patch Interaction

XCiT: Cross-Covariance Image Transformers

A **Pyramidal Residual Unit** is a type of residual unit where the number of channels gradually increases as a function of the depth at which the layer occurs, which is similar to a pyramid structure of which the shape gradually widens from the top downwards. It was introduced as part of the [PyramidNet](https://paperswithcode.com/method/pyramidnet) architecture.

Pyramidal Residual Unit

Deep Pyramidal Residual Networks

**DExTra**, or **Deep and Light-weight Expand-reduce Transformation**, is a light-weight expand-reduce transformation that enables learning wider representations efficiently.

DExTra maps a $d\_{m}$ dimensional input vector into a high dimensional space (expansion) and then
reduces it down to a $d\_{o}$ dimensional output vector (reduction) using $N$ layers of group transformations. During these expansion and reduction phases, DExTra uses group linear transformations because they learn local representations by deriving the output from a specific part of the input and are more efficient than linear transformations. To learn global representations, DExTra shares information between different groups in the group linear transformation using feature shuffling 

Formally, the DExTra transformation is controlled by five configuration parameters: (1) depth $N$, (2)
width multiplier $m\_{w}$, (3) input dimension $d\_{m}$, (4) output dimension $d\_{o}$, and (5) maximum groups $g\_{max}$ in a group linear transformation. In the expansion phase, DExTra projects the $d\_{m}$-dimensional input to a high-dimensional space, $d\_{max} = m\_{w}d\_{m}$, linearly using $\text{ceil}\left(\frac{N}{2}\right)$ layers. In the reduction phase, DExTra projects the $d\_{max}$-dimensional vector to a $d\_{o}$-dimensional space using the remaining $N -\text{ceil}\left(\frac{N}{2}\right)$ layers. Mathematically, we define the output $Y$ at each layer $l$ as:

$$ \mathbf{Y}\_{l} = \mathcal{F}\left(\mathbf{X}, \mathbf{W}^{l}, \mathbf{b}^{l}, g^{l}\right) \text{ if } l=1 $$
$$ \mathbf{Y}\_{l} = \mathcal{F}\left(\mathcal{H}\left(\mathbf{X}, \mathbf{Y}^{l-1}\right), \mathbf{W}^{l}, \mathbf{b}^{l}, g^{l}\right) \text{ Otherwise } $$

where the number of groups at each layer $l$ are computed as:

$$ g^{l} = \text{min}\left(2^{l-1}, g\_{max}\right), 1 \leq l \leq \text{ceil}\left(N/2\right) $$
$$ g^{N-l}, \text{Otherwise}$$

In the above equations, $\mathcal{F}$ is a group linear transformation function. The function $\mathcal{F}$ takes the input $\left(\mathbf{X} \text{ or } \mathcal{H}\left(\mathbf{X}, \mathbf{Y}^{l-1}\right) \right)$, splits it into $g^{l}$ groups, and then applies a linear transformation with learnable parameters $\mathbf{W}^{l}$ and bias $\mathbf{b}^{l}$ to each group independently. The outputs of each group are then concatenated to produce the final output $\mathbf{Y}^{l}$. The function $\mathcal{H}$ first shuffles the output of each group in $\mathbf{Y}^{l−1}$ and then combines it with the input $\mathbf{X}$ using an input mixer connection.

In the authors' experiments, they use $g\_{max} = \text{ceil}\left(\frac{d\_{m}}{32}\right)$ so that each group has at least 32 input elements. Note that (i) group linear transformations reduce to linear transformations when $g^{l} = 1$, and (ii) DExTra is equivalent to a multi-layer perceptron when $g\_{max} = 1$.

DExTra

DeLighT: Deep and Light-weight Transformer

**Cascade Corner Pooling** is a pooling layer for object detection that builds upon the [corner pooling](https://paperswithcode.com/method/corner-pooling) operation. Corners are often outside the objects, which lacks local appearance features. [CornerNet](https://paperswithcode.com/method/cornernet) uses corner pooling to address this issue, where we find the maximum values on the boundary directions so as to determine corners. However, it makes corners sensitive to the edges. To address this problem, we need to let corners see the visual patterns of objects. Cascade corner pooling first looks along a boundary to find a boundary maximum value, then looks inside along the location of the boundary maximum value to find an internal maximum value, and finally, add the two maximum values together. By doing this, the corners obtain both the the boundary information and the visual patterns of objects.

Cascade Corner Pooling

CenterNet: Keypoint Triplets for Object Detection

**LightAutoML** is an AutoML solution targeted for financial services companies. A typical LightAutoML pipeline scheme is presented in the Figure, each pipeline containing:

- Reader: object that receives raw data and task as input, calculates some useful metadata, performs initial data cleaning and decides about data manipulations that should be done before fitting different model types.

- LightAutoML inner datasets that contains metadata and CV iterators that implements validation scheme for the datasets.

- Multiple ML Pipelines that are stacked and/or blended to get a single prediction.

An ML pipeline in LightAutoML is one or multiple ML models that share a single data preprocessing and validation scheme. The preprocessing step may have up to two feature selection steps, a feature engineering step or even just be empty if no preprocessing is needed. The ML pipelines can be computed independently on the same datasets and then blended together using averaging (or weighted averaging). Alternatively, a stacking ensemble scheme can be used to build multi level ensemble architectures.

LightAutoML

LightAutoML: AutoML Solution for a Large Financial Services Ecosystem

**YOLOv4** is a one-stage object detection model that improves on [YOLOv3](https://paperswithcode.com/method/yolov3) with several bags of tricks and modules introduced in the literature. The components section below details the tricks and modules used.

YOLOv4

YOLOv4: Optimal Speed and Accuracy of Object Detection

The **Adaptive Locally Connected Neuron (ALCN)** is a topology aware, and locally adaptive -focusing neuron:

$$a =  f\:\Bigg( \sum_{i=1}^{m} w_{i}\phi\left( \tau\left(i\right),\Theta\right) x_{i} + b \Bigg) %f\:\Bigg(\mathbf{X(W \circ \Phi) + b} \Bigg) $$

ALCN

An Adaptive Locally Connected Neuron Model: Focusing Neuron

A **Depthwise Dilated Separable Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) that combines [depthwise separability](https://paperswithcode.com/method/depthwise-separable-convolution) with the use of [dilated convolutions](https://paperswithcode.com/method/dilated-convolution).

Depthwise Dilated Separable Convolution

ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

**Demon Adam** is a stochastic optimizer where the [Demon](https://paperswithcode.com/method/demon) momentum rule is applied to the [Adam](https://paperswithcode.com/method/adam) optimizer.

$$ \beta\_{t} = \beta\_{init}\cdot\frac{\left(1-\frac{t}{T}\right)}{\left(1-\beta\_{init}\right) + \beta\_{init}\left(1-\frac{t}{T}\right)} $$

$$ m\_{t, i} = g\_{t, i} + \beta\_{t}m\_{t-1, i} $$

$$ v\_{t+1} = \beta\_{2}v\_{t}  + \left(1-\beta\_{2}\right)g^{2}\_{t} $$

$$ \theta_{t} = \theta_{t-1} - \eta\frac{\hat{m}\_{t}}{\sqrt{\hat{v}\_{t}} + \epsilon}  $$

Demon ADAM

Demon: Improved Neural Network Training with Momentum Decay

**Margin Rectified Linear Unit**, or **Margin ReLU**, is a type of activation function based on a [ReLU](https://paperswithcode.com/method/relu), but it has a negative threshold for negative values instead of a zero threshhold.

Margin ReLU

A Comprehensive Overhaul of Feature Distillation

**Inception-C** is an image model block used in the [Inception-v4](https://paperswithcode.com/method/inception-v4) architecture.

Inception-C

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

The **Squeeze-and-Excitation Block** is an architectural unit designed to improve the representational power of a network by enabling it to perform dynamic channel-wise feature recalibration. The process is:

- The block has a convolutional block as an input.
- Each channel is "squeezed" into a single numeric value using [average pooling](https://paperswithcode.com/method/average-pooling).
- A dense layer followed by a [ReLU](https://paperswithcode.com/method/relu) adds non-linearity and output channel complexity is reduced by a ratio.
- Another dense layer followed by a sigmoid gives each channel a smooth gating function.
- Finally, we weight each feature map of the convolutional block based on the side network; the "excitation".

Squeeze-and-Excitation Block

Squeeze-and-Excitation Networks

**PolarMask** is an anchor-box free and single-shot instance segmentation method. Specifically, PolarMask takes an image as input and predicts the distance from a sampled positive location (ie a candidate object's center) with respect to the object's contour at each angle, and then assembles the predicted points to produce the final mask. There are several benefits to the system: (1) The polar representation unifies instance segmentation (masks) and object detection (bounding boxes) into a single framework (2) Two modules are designed (i.e. soft polar centerness and polar IoU loss) to sample high-quality center examples and optimize polar contour regression, making the performance of PolarMask does not depend on the bounding box prediction results and more efficient in training. (3) PolarMask is fully convolutional and can be embedded into most off-the-shelf detection methods.

Viet-Anh on Software

AI Glossary

Tech Terminology

Convolutional Block Attention Module

Hierarchical Multi-Task Learning

Attentional Liquid Warping Block