**Exact Fusion Model (EFM)** is a method for aggregating a feature pyramid. The EFM is based on [YOLOv3](https://paperswithcode.com/method/yolov3), which assigns exactly one bounding-box prior to each ground truth object. Each ground truth bounding box corresponds to one anchor box that surpasses the threshold IoU. If the size of an anchor box is equivalent to the field-of-view of the grid cell, then for the grid cells of the $s$-th scale, the corresponding bounding box will be lower bounded by the $(s − 1)$th scale and upper bounded by the (s + 1)th scale. Therefore, the EFM assembles features from the three scales.

Exact Fusion Model

CSPNet: A New Backbone that can Enhance Learning Capability of CNN

**LapStyle**, or **Laplacian Pyramid Network**, is a feed-forward style transfer method. It uses a [Drafting Network](https://paperswithcode.com/method/drafting-network) to transfer global style patterns in low-resolution, and adopts higher resolution [Revision Networks](https://paperswithcode.com/method/revision-network) to revise local styles in a pyramid manner according to outputs of multi-level Laplacian filtering of the content image. Higher resolution details can be generated by stacking Revision Networks with multiple Laplacian pyramid levels. The final stylized image is obtained by aggregating outputs of all pyramid levels.

Specifically, we first generate image pyramid $\left$\bar{x}\_{c}, r\_{c}\right$$ from content image $x\_{c}$ with the help of Laplacian filter. Rough low-resolution stylized image are then generated by the Drafting Network. Then the Revision Network generates stylized detail image in high resolution. Then the final stylized image is generated by aggregating the outputs pyramid. $L, C$ and $A$ in an image represent Laplacian, concatenate and aggregation operation separately.

LapStyle

Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer

**CornerNet-Squeeze** is an object detector that extends [CornerNet](https://paperswithcode.com/method/cornernet) with a new compact hourglass architecture that makes use of fire modules with depthwise separable convolutions.

CornerNet-Squeeze

CornerNet-Lite: Efficient Keypoint Based Object Detection

**Neo-fuzzy-neuron** is a type of artificial neural network that combines the characteristics of both fuzzy logic and neural networks. It uses a fuzzy inference system to model non-linear relationships between inputs and outputs, and a feedforward neural network to learn the parameters of the fuzzy system. The combination of these two approaches provides a flexible and powerful tool for solving a wide range of problems in areas such as pattern recognition, control, and prediction.

Neo-fuzzy-neuron based new approach to system modeling, with application to actual system

**Flan-T5** is the instruction fine-tuned version of **T5** or **Text-to-Text Transfer Transformer** Language Model.

Flan-T5

Scaling Instruction-Finetuned Language Models

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.

BLIP

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

**Restricted Boltzmann Machines**, or **RBMs**, are two-layer generative neural networks that learn a probability distribution over the inputs. They are a special class of Boltzmann Machine in that they have a restricted number of connections between visible and hidden units. Every node in the visible layer is connected to every node in the hidden layer, but no nodes in the same group are connected. RBMs are usually trained using the contrastive divergence learning procedure.

Image Source: [here](https://medium.com/datatype/restricted-boltzmann-machine-a-complete-analysis-part-1-introduction-model-formulation-1a4404873b3)

Restricted Boltzmann Machine

**DistanceNet** is a learning algorithm for multi-source domain adaptation that uses various distance measures, or a mixture of these distance measures, as an additional loss function to be minimized jointly with the task's loss function, so as to achieve better unsupervised domain adaptation.

DistanceNet

Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits

Recently, [dense connections](https://paperswithcode.com/method/dense-connections) have attracted substantial attention in computer vision because they facilitate gradient flow and implicit deep supervision during training. Particularly, [DenseNet](https://paperswithcode.com/method/densenet) that connects each layer to every other layer in a feed-forward fashion and has shown impressive performances in natural image classification tasks. We propose HyperDenseNet, a 3-D fully convolutional neural network that extends the definition of dense connectivity to multi-modal segmentation problems. Each imaging modality has a path, and dense connections occur not only between the pairs of layers within the same path but also between those across different paths. This contrasts with the existing multi-modal CNN approaches, in which modeling several modalities relies entirely on a single joint layer (or level of abstraction) for fusion, typically either at the input or at the output of the network. Therefore, the proposed network has total freedom to learn more complex combinations between the modalities, within and in-between all the levels of abstraction, which increases significantly the learning representation. We report extensive evaluations over two different and highly competitive multi-modal brain tissue segmentation challenges, iSEG 2017 and MRBrainS 2013, with the former focusing on six month infant data and the latter on adult images. HyperDenseNet yielded significant improvements over many state-of-the-art segmentation networks, ranking at the top on both benchmarks. We further provide a comprehensive experimental analysis of features re-use, which confirms the importance of hyper-dense connections in multi-modal representation learning.

HyperDenseNet

HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation

**SimAug**, or **Simulation as Augmentation**, is a data augmentation method for trajectory prediction. It augments the representation such that it is robust to the variances in semantic scenes and camera views.  First, to deal with the gap between real and synthetic semantic scene, it represents each training trajectory by high-level scene semantic segmentation features, and defends the model from adversarial examples generated by whitebox attack methods. Second, to overcome the changes in camera views, it generates multiple views for the same trajectory, and encourages the model to focus on the “hardest” view to which the model has learned. The classification loss is adopted and the view with the highest loss is favored during training. Finally, the augmented trajectory is computed as a convex combination of the trajectories generated in previous steps. The trajectory prediction model is built on a multi-scale representation and the final model is trained to minimize the empirical vicinal risk over the distribution of augmented trajectories.

SimAug

SimAug: Learning Robust Representations from 3D Simulation for Pedestrian Trajectory Prediction in Unseen Cameras

**1-bit LAMB** is a communication-efficient stochastic optimization technique which introduces a novel way to support adaptive layerwise learning rates even when communication is compressed. Learning from the insights behind [1-bit Adam](https://paperswithcode.com/method/1-bit-adam), it is a a 2-stage algorithm which uses [LAMB](https://paperswithcode.com/method/lamb) (warmup stage) to “pre-condition” a communication compressed momentum SGD algorithm (compression stage). At compression stage where original LAMB algorithm cannot be used to update the layerwise learning rates, 1-bit LAMB employs a novel way to adaptively scale layerwise learning rates based on information from both warmup and compression stages. As a result, 1-bit LAMB is able to achieve large batch optimization (LAMB)’s convergence speed under compressed communication.

There are two major differences between 1-bit LAMB and the original LAMB:

- During compression stage, 1-bit LAMB updates the layerwise learning rate based on a novel “reconstructed gradient” based on the compressed momentum. This makes 1-bit LAMB compatible with error compensation and be able to keep track of the training dynamic under compression.
- 1-bit LAMB also introduces extra stabilized soft thresholds when updating layerwise learning rate at compression stage, which makes training more stable under compression.

1-bit LAMB

1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed

A VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and a deep neural network with 3D generative representations of objects for robust visual scene parsing.

PO3D-VQA

3D-Aware Visual Question Answering about Parts, Poses and Occlusions

### Why SGD with  Momentum?
In deep learning, we have used stochastic gradient descent as one of the optimizers because at the end we will find the minimum weight and bias at which the model loss is lowest. In the SGD we have some issues in which the SGD does not work perfectly because in deep learning we got a non-convex cost function graph and if use the simple SGD then it leads to low performance. There are 3 main reasons why it does not work:

<img src="https://www.cs.umd.edu/~tomg/img/landscapes/shortHighRes.png" alt="Non-convex graph" style="width:400px;height :300px;" />

1) We end up in local minima and not able to reach global minima
At the start, we randomly start at some point and we are going to end up at the local minimum and not able to reach the global minimum.

2) Saddle Point will be the stop for reaching global minima
A saddle point is a point where in one direction the surface goes in the upward direction and in another direction it goes downwards. So that the slope is changing very gradually so the speed of changing is going to slow and as result, the training also going to slow.

3) High curvature can be a reason
The larger radius leads to low curvature and vice-versa. It will be difficult to traverse in the large curvature which was generally high in non-convex optimization.
By using the SGD with Momentum optimizer we can overcome the problems like high curvature, consistent gradient, and noisy gradient.

### What is SGD with Momentum?
SGD with  Momentum is one of the optimizers which is used to improve the performance of the neural network.

Let's take an example and understand the intuition behind the optimizer suppose we have a ball which is sliding from the start of the slope as it goes the speed of the bowl is increased over time. If we have one point A and we want to reach point B and we don't know in which direction to move but we ask for the 4 points which have already reached point B. If all 4 points are pointing you in the same direction then the confidence of the A is more and it goes in the direction pointed very fast. This is the main concept behind the SGD with Momentum.

<img src="https://cdn-images-1.medium.com/max/1000/1*zNbZqU_uDIV13c9ZCJOEXA.jpeg" alt="Non-convex graph" style="width:400px;height :250px;" />
### How does SGD with Momentum work?
So first to understand the concept of exponentially weighted moving average (EWMA). It was a technique through which try to find the trend in time series data. The formula of the EWMA is :

<img src="https://cdn-images-1.medium.com/max/1000/1*O9Wcq-mbRgNOdRNTivSefw.png" alt="Non-convex graph" style="width:400px;height :100px;" />

 In the formula, β represents the weightage that is going to assign to the past values of the gradient. The values of β is from 0 < β < 1. If the value of the beta is 0.5 then it means that the 1/1–0.5 = 2 so it represents that the calculated average was from the previous 2 readings. 

The value of Vt depends on β. The higher the value of β the more we try to get an average of more past data and vice-versa. For example, let's take the value of β 0.98 and 0.5 for two different scenarios so if we do 1/1-β then we get 50 and 10 respectively so it was clear that to calculate the average we take past 50 and 10 outcomes respectively for both cases.
Now in SGD with Momentum, we use the same concept of EWMA. Here we introduce the term velocity v which is used to denote the change in the gradient to get to the global minima. The change in the weights is denoted by the formula:

<img src="https://cdn-images-1.medium.com/max/1000/0*i_r3u7LACa6dQyXd" alt="Non-convex graph" style="width:400px;height :100px;" />

the β part of the V formula denotes and is useful to compute the confidence or we can say the past velocity for calculating Vt we have to calculate Vt-1 and for calculating Vt-1 we have to calculate Vt-2 and likewise. So we are using the history of velocity to calculate the momentum and this is the part that provides acceleration to the formula.

<img src="https://cdn-images-1.medium.com/max/1000/1*L5lNKxAHLPYNc6-Zs4Vscw.png" alt="Non-convex graph" style="width:300px;height :100px;" />

Here we have to consider two cases:
1. β=0 then, as per the formula weight updating is going to just work as a Stochastic gradient descent. Here we called β a decaying factor because it is defining the speed of past velocity.

2. β=1 then, there will be no decay. It involves the dynamic equilibrium which is not desired so we generally use the value of β like 0.9,0.99or 0.5 only.

### Advantages of SGD with Momentum :
1. Momentum is faster than stochastic gradient descent the training will be faster than SGD.
2. Local minima can be an escape and reach global minima due to the momentum involved.

<img src="https://cdn-images-1.medium.com/max/1000/1*Nb39bHHUWGXqgisr2WcLGQ.gif" alt="Non-convex graph" style="width:400px;height :300px;" />

Here in the video, we can see that purple is SGD with Momentum and light blue is for SGD the SGD with Momentum can reach global minima whereas SGD is stuck in local minima.
But there is a catch, the momentum itself can be a problem sometimes because of the high momentum after reaching global minima it is still fluctuating and take some time to get stable at global minima. And that kind of behavior leads to time consumption which makes SGD with Momentum slower than other optimization out there but still faster than SGD.

SGD with Momentum

**TimeSformer** is a [convolution](https://paperswithcode.com/method/convolution)-free approach to video classification built exclusively on self-attention over space and time. It adapts the standard [Transformer](https://paperswithcode.com/method/transformer) architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Specifically, the method adapts the image model [[Vision Transformer](https://paperswithcode.com/method/vision-transformer)](https//www.paperswithcode.com/method/vision-transformer) (ViT) to video by extending the self-attention mechanism from the image space to the space-time 3D volume. As in ViT, each patch is linearly mapped into an embedding and augmented with positional information. This makes it possible to interpret the resulting sequence of vector

TimeSformer

Is Space-Time Attention All You Need for Video Understanding?

**SKNet** is a type of convolutional neural network that employs [selective kernel](https://paperswithcode.com/method/selective-kernel) units, with selective kernel convolutions, in its architecture. This allows for a type of attention where the network can learn to attend to different receptive fields.

SKNet

The **Self-Organizing Map (SOM)**, commonly also known as Kohonen network (Kohonen 1982, Kohonen 2001) is a computational method for the visualization and analysis of high-dimensional data, especially experimentally acquired information.

Extracted from [scholarpedia](http://www.scholarpedia.org/article/Self-organizing_map)

**Sources**:

Image: [scholarpedia](http://www.scholarpedia.org/article/File:Somnbc.png)

Paper: [Kohonen, T. Self-organized formation of topologically correct feature maps. Biol. Cybern. 43, 59–69 (1982)](https://doi.org/10.1007/BF00337288)

Book: [Self-Organizing Maps](https://doi.org/10.1007/978-3-642-56927-2)

**ProxylessNAS** directly learns neural network architectures on the target task and target hardware without any proxy task. Additional contributions include:

- Using a new path-level pruning perspective for [neural architecture search](https://paperswithcode.com/method/neural-architecture-search), showing a close connection between NAS and model compression. Memory consumption is saved by one order of magnitude by using path-level binarization.
- Using a novel gradient-based approach (latency regularization loss) for handling hardware objectives (e.g. latency). Given different hardware platforms: CPU/GPU/Mobile, ProxylessNAS enables hardware-aware neural network specialization that’s exactly optimized for the target hardware.

ProxylessNAS

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

**Graphic Mutual Information**, or **GMI**, measures the correlation between input graphs and high-level hidden representations. GMI generalizes the idea of conventional mutual information computations from vector space to the graph domain where measuring mutual information from two aspects of node features and topological structure is indispensable. GMI exhibits several benefits: First, it is invariant to the isomorphic transformation of input graphs---an inevitable constraint in many existing graph representation learning algorithms; Besides, it can be efficiently estimated and maximized by current mutual information estimation methods such as MINE.

Viet-Anh on Software

AI Glossary

Tech Terminology

Exact Fusion Model

Laplacian Pyramid Network

CornerNet-Squeeze

Neo-fuzzy-neuron

Flan-T5

BLIP: Bootstrapping Language-Image Pre-training

Restricted Boltzmann Machine

DistanceNet

HyperDenseNet

Simulation as Augmentation

1-bit LAMB

Parts, Poses, and Occlusions in 3D Visual Question Answering

SGD with Momentum

TimeSformer

SKNet

Self-Organizing Map

ProxylessNAS

Graphic Mutual Information

Viet-Anh on Software