**ESPNetv2** is a convolutional neural network that utilises group point-wise and depth-wise dilated separable convolutions to learn representations from a large effective receptive field with fewer FLOPs and parameters.

ESPNetv2

ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network

A **G-GLN Neuron** is a type of neuron used in the [G-GLN](https://paperswithcode.com/method/g-gln) architecture. G-GLN. The key idea is that further representational power can be added to a weighted product of Gaussians via a contextual gating procedure. This is achieved by extending a weighted product of Gaussians model with an additional type of input called side information. The side information will be used by a neuron to select a weight vector to apply for a given example from a table of weight vectors. In typical applications to regression, the side information is defined as the (normalized) input features for an input example: i.e. $z=(x-\bar{x}) / \sigma\_{x}$.

More formally, associated with each neuron is a context function $c: \mathcal{Z} \rightarrow \mathcal{C}$, where $\mathcal{Z}$ is the set of possible side information and $\mathcal{C}=\{0, \ldots, k-1\}$ for some $k \in \mathbb{N}$ is the context space. Each neuron $i$ is now parameterized by a weight matrix $W\_{i}=\left[w\_{i, 0} \ldots w\_{i, k-1}\right]^{\top}$ with each row vector $w\_{i j} \in \mathcal{W}$ for $0 \leq j<k$. The context function $c$ is responsible for mapping side information $z \in \mathcal{Z}$ to a particular row $w\_{i, c(z)}$ of $W_{i}$, which we then use to weight the Product of Gaussians. In other words, a G-GLN neuron can be defined by:

$$
\operatorname{PoG}\_{W}^{c}\left(y ; f_{1}(\cdot), \ldots, f\_{m}(\cdot), z\right):=\operatorname{PoG}\_{w^{c(z)}}\left(y ; f\_{1}(\cdot), \ldots, f\_{m}(\cdot)\right)
$$

with the associated loss function $-\log \left(\operatorname{PoG}\_{W}^{c}\left(y ; f\_{1}(y), \ldots, f\_{m}(y), z\right)\right)$ inheriting all the properties needed to apply Online Convex Programming.

G-GLN Neuron

Gaussian Gated Linear Networks

**Big-Little Modules** are blocks for image models that have two branches: each of which represents a separate block from a deep model and a less deep counterpart. They were proposed as part of the [BigLittle-Net](https://paperswithcode.com/method/big-little-net) architecture. The two branches are fused with a linear combination and unit weights. These two branches are known as Big-Branch (more layers and channels at low resolutions) and Little-Branch (fewer layers and channels at high resolution).

Big-Little Module

Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition

StoGCN is a control variate based algorithm which allow sampling an arbitrarily small neighbor size. Presents new theoretical guarantee for the algorithms to converge to a local optimum of GCN.

StoGCN

Stochastic Training of Graph Convolutional Networks with Variance Reduction

GCT first collects global information by computing the l2-norm of each channel. Next, a learnable vector $ \alpha $ is applied to scale the feature. Then a competition mechanism is adopted by channel normalization to interact between channels.

Unlike previous methods, GCT first collects global information by computing the $l_{2}$-norm of each channel. 
Next, a learnable vector $\alpha$ is applied to scale the feature.
Then a competition mechanism is adopted by 
channel normalization to interact between channels. 
Like other common normalization methods, 
a learnable scale parameter $\gamma$ and bias $\beta$ are applied to 
rescale the normalization.
However, unlike previous methods,
GCT adopts tanh activation to control the attention vector.
Finally, it not only multiplies the input by the attention vector but also adds an identity connection. GCT can be written as: 
\begin{align}
    s = F_\text{gct}(X, \theta) & = \tanh (\gamma CN(\alpha \text{Norm}(X)) + \beta)
\end{align}
\begin{align}
    Y & = s  X + X
\end{align}

where $\alpha$, $\beta$ and $\gamma$ are trainable parameters. $\text{Norm}(\cdot)$ indicates the $L2$-norm of each channel. $CN$ is  channel normalization.

A GCT block has fewer parameters than an SE block, and as it is  lightweight, 
 can be added after each convolutional layer of a CNN.

Gated Channel Transformation for Visual Recognition

**M2Det** is a one-stage object detection model that utilises a Multi-Level Feature Pyramid Network ([MLFPN](https://paperswithcode.com/method/mlfpn)) to extract features from the input image, and then similar to [SSD](https://paperswithcode.com/method/ssd), produces dense bounding boxes and category scores based on the learned features, followed by the non-maximum suppression (NMS) operation to produce the final results.

M2Det

M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network

A **QRNN**, or **Quasi-Recurrent Neural Network**, is a type of recurrent neural network that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Due to their increased parallelism, they can be up to 16 times faster at train and test time than [LSTMs](https://paperswithcode.com/method/lstm).

Given an input sequence $\mathbf{X} \in \mathbb{R}^{T\times{n}}$ of $T$ n-dimensional vectors $\mathbf{x}\_{1}, \dots, \mathbf{x}\_{T}$, the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of $m$ filters, producing a sequence $\mathbf{Z} \in \mathbb{R}^{T\times{m}}$ of m-dimensional candidate vectors $\mathbf{z}\_{t}$. Masked convolutions are used so filters can not access information from future timesteps (implementing with left padding).

Additional convolutions are applied with separate filter banks to obtain sequences of vectors for the
elementwise gates that are needed for the pooling function. While the candidate vectors are passed
through a $\tanh$ nonlinearity, the gates use an elementwise sigmoid. If the pooling function requires a
forget gate $f\_{t}$ and an output gate $o\_{t}$ at each timestep, the full set of computations in the convolutional component is then:

$$ \mathbf{Z} = \tanh\left(\mathbf{W}\_{z} ∗ \mathbf{X}\right) $$
$$ \mathbf{F} = \sigma\left(\mathbf{W}\_{f} ∗ \mathbf{X}\right) $$
$$ \mathbf{O} = \sigma\left(\mathbf{W}\_{o} ∗ \mathbf{X}\right) $$

where $\mathbf{W}\_{z}$, $\mathbf{W}\_{f}$, and $\mathbf{W}\_{o}$, each in $\mathbb{R}^{k×n×m}$, are the convolutional filter banks and ∗ denotes a [masked convolution](https://paperswithcode.com/method/masked-convolution) along the timestep dimension.  Dynamic [average pooling](https://paperswithcode.com/method/average-pooling) by Balduzzi & Ghifary (2016) is used, which uses only a forget gate:

$$ \mathbf{h}\_{t} = \mathbf{f}\_{t} \odot{\mathbf{h}\_{t−1}} + \left(1 − \mathbf{f}\_{t}\right) \odot{\mathbf{z}\_{t}} $$ 

Which is denoted f-pooling. The function may also include an output gate:

$$ \mathbf{c}\_{t} = \mathbf{f}\_{t} \odot{\mathbf{c}\_{t−1}} + \left(1 − \mathbf{f}\_{t}\right) \odot{\mathbf{z}\_{t}} $$ 

$$ \mathbf{h}\_{t} = \mathbf{o}\_{t} \odot{\mathbf{c}\_{t}} $$

Which is denoted fo-pooling. Or the recurrence relation may include an independent input and forget gate:

$$ \mathbf{c}\_{t} = \mathbf{f}\_{t} \odot{\mathbf{c}\_{t−1}} + \mathbf{i}\_{t}\odot{\mathbf{z}\_{t}} $$ 

$$ \mathbf{h}\_{t} = \mathbf{o}\_{t} \odot{\mathbf{c}\_{t}} $$

Which is denoted ifo-pooling. In each case $h$ or $c$ is initialized to zero. The recurrent part sof these functions must be calculated for each timestep in the sequence, but parallelism along feature dimensions means evaluating them even over long sequences requires a negligible amount of computation time.

A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features. As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions.

QRNN

Quasi-Recurrent Neural Networks

**DiCENet** is a convolutional neural network architecture that utilizes dimensional convolutions (and dimension-wise fusion). The dimension-wise convolutions apply light-weight convolutional filtering across each dimension of the input tensor while dimension-wise fusion efficiently combines these dimension-wise representations; allowing the [DiCE Unit](https://paperswithcode.com/method/dice-unit) in the network to efficiently encode spatial and channel-wise information contained in the input tensor.

DiCENet

DiCENet: Dimension-wise Convolutions for Efficient Networks

**DeepSIM** is a generative model for conditional image manipulation based on a single image. The network learns to map between a primitive representation of the image to the image itself. At manipulation time, the generator allows for making complex image changes by modifying the primitive input representation and mapping it through the network. The choice of a primitive representations has an impact on the ease and expressiveness of the manipulations and can be automatic (e.g. edges), manual, or hybrid such as edges on top of segmentations.

DeepSIM

Image Shape Manipulation from a Single Augmented Training Sample

A **Local Relation Layer** is an image feature extractor that is an alternative to a [convolution](https://paperswithcode.com/method/convolution) operator. The intuition is that aggregation in convolution is basically a pattern matching process that applies fixed filters, which can be inefficient at modeling visual elements with varying spatial distributions. The local relation layer adaptively determines aggregation weights based on the compositional relationship of local pixel pairs. It is argued that, with this relational approach, it can composite visual elements into higher-level entities in a more efficient manner that benefits semantic inference.

Local Relation Layer

Local Relation Networks for Image Recognition

**DeepWalk** learns embeddings (social representations) of a graph's vertices, by modeling a stream of short random walks. Social representations are latent features of the vertices that capture neighborhood similarity and community membership. These latent representations encode social relations in a continuous vector space with a relatively small number of dimensions. It generalizes neural language models to process a special language composed of a set of randomly-generated walks. 

The goal is to learn a latent representation, not only a probability distribution of node co-occurrences, and so as to introduce a mapping function $\Phi \colon v \in V \mapsto \mathbb{R}^{|V|\times d}$.
This mapping $\Phi$ represents the latent social representation associated with each vertex $v$ in the graph. In practice, $\Phi$ is represented by a $|V| \times d$ matrix of free parameters.

DeepWalk

DeepWalk: Online Learning of Social Representations

A **Bidirectional GRU**, or **BiGRU**, is a sequence processing model that consists of two [GRUs](https://paperswithcode.com/method/gru). one taking the input in a forward direction, and the other in a backwards direction. It is a bidirectional recurrent neural network with only the input and forget gates.

Image Source: *Rana R (2016). Gated Recurrent Unit (GRU) for Emotion Classification from Noisy Speech.*

BiGRU

**Monte-Carlo Tree Search** is a planning algorithm that accumulates value estimates obtained from Monte Carlo simulations in order to successively direct simulations towards more highly-rewarded trajectories. We execute MCTS after encountering each new state to select an agent's action for that state: it is executed again to select the action for the next state. Each execution is an iterative process that simulates many trajectories starting from the current state to the terminal state. The core idea is to successively focus multiple simulations starting at the current state by extending the initial portions of trajectories that have received high evaluations from earlier simulations.

Source: Sutton and Barto, Reinforcement Learning (2nd Edition)

Image Credit: [Chaslot et al](https://www.aaai.org/Papers/AIIDE/2008/AIIDE08-036.pdf)

Monte-Carlo Tree Search

**SNet** is a convolutional neural network architecture and object detection backbone used for the [ThunderNet](https://paperswithcode.com/method/thundernet) two-stage object detector. SNet uses ShuffleNetV2 basic blocks but replaces all 3×3 depthwise convolutions with 5×5 depthwise convolutions.

SNet

ThunderNet: Towards Real-time Generic Object Detection

**VGG** is a classical convolutional neural network architecture. It was based on an analysis of how to increase the depth of such networks. The network utilises small 3 x 3 filters. Otherwise the network is characterized by its simplicity: the only other components being pooling layers and a fully connected layer.

Image: [Davi Frossard](https://www.cs.toronto.edu/frossard/post/vgg16/)

Very Deep Convolutional Networks for Large-Scale Image Recognition

**PointNet** provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. It directly takes point clouds as input and outputs either class labels for the entire input or per point segment/part labels for each point of the input.

Source: [Qi et al.](https://arxiv.org/pdf/1612.00593v2.pdf)

Image source: [Qi et al.](https://arxiv.org/pdf/1612.00593v2.pdf)

PointNet

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

**DDParser**, or **Baidu Dependency Parser**, is a Chinese dependency parser trained on a large-scale manually labeled dataset called Baidu Chinese Treebank (DuCTB).

For inputs, for the $i$ th word, its input vector $e_{i}$ is the concatenation of the word embedding and character-level representation:

$$
e\_{i}=e\_{i}^{w o r d} \oplus C h a r L S T M\left(w\_{i}\right)
$$

Where $\operatorname{CharLSTM}\left(w_{i}\right)$ is the output vectors after feeding the character sequence into a [BiLSTM](https://paperswithcode.com/method/bilstm) layer. The experimental results on DuCTB dataset show that replacing POS tag embeddings with $\operatorname{CharLSTM}\left(w_{i}\right)$ leads to the improvement.

For the BiLSTM encoder, three BiLSTM layers are employed over the input vectors for context encoding. Denote $r\_{i}$ the output vector of the top-layer BiLSTM for $w\_{i}$

The dependency parser of [Dozat and Manning](https://arxiv.org/abs/1611.01734) is used. Dimension-reducing MLPs are applied to each recurrent output vector $r\_{i}$ before applying the biaffine transformation. Applying smaller MLPs to the recurrent output states before the biaffine classifier has the advantage of stripping away information not relevant to the current decision. Then biaffine attention is used both in the dependency arc classifier and relation classifier. The computations of all symbols in the Figure are shown below:

$$
h_{i}^{d-a r c}=M L P^{d-a r c}\left(r_{i}\right)
$$
$$
h_{i}^{h-a r c}=M L P^{h-a r c}\left(r_{i}\right) \\
$$
$$
h_{i}^{d-r e l}=M L P^{d-r e l}\left(r_{i}\right) \\
$$
$$
h_{i}^{h-r e l}=M L P^{h-r e l}\left(r_{i}\right) \\
$$
$$
S^{a r c}=\left(H^{d-a r c} \oplus I\right) U^{a r c} H^{h-a r c} \\
$$
$$
S^{r e l}=\left(H^{d-r e l} \oplus I\right) U^{r e l}\left(\left(H^{h-r e l}\right)^{T} \oplus I\right)^{T}
$$

For the decoder, the first-order Eisner algorithm is used to ensure that the output is a projection tree. Based on the dependency tree built by biaffine parser, we get a word sequence through the in-order traversal of the tree. The output is a projection tree only if the word sequence is in order.

DDParser

A Practical Chinese Dependency Parser Based on A Large-scale Dataset

The main control tasks in autonomous vehicles are steering (lateral) and speed (longitudinal) control. PID controllers are widely used in the industry because of their simplicity and good performance, but they are difficult to tune and need additional adaptation to control nonlinear systems with varying parameters. In this paper, the longitudinal control task is addressed by implementing adaptive PID control
using two different approaches: Genetic Algorithms (GA-PID) and then Neural Networks (NN-PID) respectively. The vehicle
nonlinear longitudinal dynamics are modeled using Powertrain blockset library. Finally, simulations are performed to assess
and compare the performance of the two controllers subject to external disturbances.

Viet-Anh on Software

AI Glossary

Tech Terminology