**$n$-step Returns** are used for value function estimation in reinforcement learning. Specifically, for $n$ steps we can write the complete return as:

$$ R\_{t}^{(n)} = r\_{t+1} + \gamma{r}\_{t+2} + \cdots + \gamma^{n-1}\_{t+n} + \gamma^{n}V\_{t}\left(s\_{t+n}\right) $$

We can then write an $n$-step backup, in the style of TD learning, as:

$$ \Delta{V}\_{r}\left(s\_{t}\right) = \alpha\left[R\_{t}^{(n)} - V\_{t}\left(s\_{t}\right)\right] $$

Multi-step returns often lead to faster learning with suitably tuned $n$.

Image Credit: Sutton and Barto, Reinforcement Learning

N-step Returns

**Point-GNN** is a graph neural network for detecting objects from a LiDAR point cloud. It predicts the category and shape of the object that each vertex in the graph belongs to. In Point-GNN, there is an auto-registration mechanism to reduce translation variance, as well as a box merging and scoring operation to combine detections from multiple vertices accurately.

Point-GNN

Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud

A **R(2+1)D** convolutional neural network is a network for action recognition that employs [R(2+1)D](https://paperswithcode.com/method/2-1-d-convolution) convolutions in a [ResNet](https://paperswithcode.com/method/resnet) inspired architecture. The use of these convolutions over regular [3D Convolutions](https://paperswithcode.com/method/3d-convolution) reduces computational complexity, prevents overfitting, and introduces more non-linearities that allow for a better functional relationship to be modeled.

R(2+1)D

A Closer Look at Spatiotemporal Convolutions for Action Recognition

The **NVAE Encoder Residual Cell** is a [residual connection](https://paperswithcode.com/method/residual-connection) block used in the [NVAE](https://paperswithcode.com/method/nvae) architecture for the encoder. It applies two series of BN-[Swish](https://paperswithcode.com/method/swish)-Conv layers without changing the number of channels.

NVAE Encoder Residual Cell

NVAE: A Deep Hierarchical Variational Autoencoder

**Contrastive Multiview Coding (CMC)** is a self-supervised learning approach, based on [CPC](https://paperswithcode.com/method/contrastive-predictive-coding), that  learns representations that capture information shared between multiple sensory views. The core idea is to set an anchor view and the sample positive and negative data points from the other view and maximise agreement between positive pairs in learning from two views. Contrastive learning is used to build the embedding.

Contrastive Multiview Coding

**NT-Xent**, or **Normalized Temperature-scaled Cross Entropy Loss**, is a loss function. Let $\text{sim}\left(\mathbf{u}, \mathbf{v}\right) = \mathbf{u}^{T}\mathbf{v}/||\mathbf{u}|| ||\mathbf{v}||$ denote the cosine similarity between two vectors $\mathbf{u}$ and $\mathbf{v}$. Then the loss function for a positive pair of examples $\left(i, j\right)$ is :

$$ \mathbb{l}\_{i,j} = -\log\frac{\exp\left(\text{sim}\left(\mathbf{z}\_{i}, \mathbf{z}\_{j}\right)/\tau\right)}{\sum^{2N}\_{k=1}\mathcal{1}\_{[k\neq{i}]}\exp\left(\text{sim}\left(\mathbf{z}\_{i}, \mathbf{z}\_{k}\right)/\tau\right)}$$

where $\mathcal{1}\_{[k\neq{i}]} \in ${$0, 1$} is an indicator function evaluating to $1$ iff $k\neq{i}$ and $\tau$ denotes a temperature parameter. The final loss is computed across all positive pairs, both $\left(i, j\right)$ and $\left(j, i\right)$, in a mini-batch.

Source: [SimCLR](https://paperswithcode.com/method/simclr)

NT-Xent

Improved Deep Metric Learning with Multi-class N-pair Loss Objective

**MatrixNet** is a scale and aspect ratio aware building block for object detection that seek to handle objects of different sizes and aspect ratios. They have several matrix layers, each layer handles an object of specific size and aspect ratio. They can be seen as an alternative to [FPNs](https://paperswithcode.com/method/fpn). While FPNs are capable of handling objects of different sizes, they do not have a solution for objects of different aspect ratios. Objects such as a high tower, a giraffe, or a knife introduce a design difficulty for FPNs: does one map these objects to layers according to their width or height? Assigning the object to a layer according to its larger dimension would result in loss of information along the smaller dimension due to aggressive downsampling, and vice versa. 

MatrixNets assign objects of different sizes and aspect ratios to layers such that object sizes within their assigned layers are close to uniform. This assignment allows a square output [convolution](https://paperswithcode.com/method/convolution) kernel to equally gather information about objects of all aspect ratios and scales. MatrixNets can be applied to any backbone, similar to FPNs. We denote this by appending a "-X" to the backbone, i.e. ResNet50-X.

MatrixNet

MatrixNets: A New Scale and Aspect Ratio Aware Architecture for Object Detection

Just as [dropout](https://paperswithcode.com/method/dropout) prevents co-adaptation of activations, **DropPath** prevents co-adaptation of parallel paths in networks such as [FractalNets](https://paperswithcode.com/method/fractalnet) by randomly dropping operands of the join layers. This
discourages the network from using one input path as an anchor and another as a corrective term (a
configuration that, if not prevented, is prone to overfitting). Two sampling strategies are:

- **Local**: a join drops each input with fixed probability, but we make sure at least one survives.
- **Global**: a single path is selected for the entire network. We restrict this path to be a single
column, thereby promoting individual columns as independently strong predictors.

DropPath

FractalNet: Ultra-Deep Neural Networks without Residuals

**Compact Convolutional Transformers** utilize sequence pooling and replace the patch embedding with a convolutional embedding, allowing for better inductive bias and making positional embeddings optional. CCT achieves better accuracy than ViT-Lite (smaller ViTs) and increases the flexibility of the input parameters.

Escaping the Big Data Paradigm with Compact Transformers

**Flow Alignment Module**, or **FAM**, is a flow-based align module for scene parsing to learn Semantic Flow between feature maps of adjacent levels and broadcast high-level features to high resolution features effectively and efficiently. The concept of Semantic Flow is inspired from optical flow, which is widely used in video processing task to represent the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by relative motion. The authors postulate that the relatinship between two feature maps of arbitrary resolutions from the same image can also be represented with the “motion” of every pixel from one feature map to the other one. Once precise Semantic Flow is obtained, the network is able to propagate semantic features with minimal information loss.

In the FAM module, the transformed high-resolution feature map are combined with the low-resolution feature map to generate the semantic flow field, which is utilized to warp the low-resolution feature map to high-resolution feature map.

Flow Alignment Module

Semantic Flow for Fast and Accurate Scene Parsing

CharacterBERT is a variant of [BERT](https://paperswithcode.com/method/bert) that **drops the wordpiece system** and **replaces it with a CharacterCNN module** just like the one [ELMo](https://paperswithcode.com/method/elmo) uses to produce its first layer representation. This allows CharacterBERT to represent any input token without splitting it into wordpieces. Moreover, this frees BERT from the burden of a domain-specific wordpiece vocabulary which may not be suited to your domain of interest (e.g. medical domain). Finally, it allows the model to be more robust to noisy inputs.

CharacterBERT

CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

**DeepLabv2** is an architecture for semantic segmentation that build on [DeepLab](https://paperswithcode.com/method/deeplab) with an atrous [spatial pyramid pooling](https://paperswithcode.com/method/spatial-pyramid-pooling) scheme. Here we have parallel dilated convolutions with different rates applied in the input feature map, which are then fused together. As objects of the same class can have different sizes in the image, [ASPP](https://paperswithcode.com/method/aspp) helps to account for different object sizes.

DeepLabv2

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Liu et al. presented self-calibrated convolution as a means to enlarge the receptive field at each spatial location. 

Self-calibrated convolution is used together with a standard convolution. It first divides the input feature $X$ into $X_{1}$ and $X_{2}$ in the channel domain. The self-calibrated convolution first uses average pooling to reduce the input size and enlarge the receptive field:
\begin{align}
T_{1} = AvgPool_{r}(X_{1}) 
\end{align}
where $r$ is the filter size and stride. Then a convolution is used to model the channel relationship and a bilinear interpolation operator $Up$ is used to upsample the feature map: 

\begin{align}
X'_{1} = \text{Up}(Conv_2(T_1))
\end{align}

Next, element-wise multiplication finishes the self-calibrated process:

\begin{align}
Y'_{1} = Conv_3(X_1) \sigma(X_1 + X'_1)
\end{align}

Finally, the output feature map of is formed:
\begin{align}
Y_{1} &= Conv_4(Y'_{1})
\end{align}
\begin{align}
Y_2 &= Conv_1(X_2)
\end{align}
\begin{align}
Y &= [Y_1; Y_2]
\end{align}
Such self-calibrated convolution can enlarge the receptive field of a network and improve its adaptability. It achieves excellent results in image classification and certain downstream tasks such as instance segmentation, object detection and keypoint detection.

Self-Calibrated Convolutions

Improving Convolutional Networks With Self-Calibrated Convolutions

**TD_INLINE_MATH_1** is a generalisation of **TD_INLINE_MATH_2** reinforcement learning algorithms, but it employs an [eligibility trace](https://paperswithcode.com/method/eligibility-trace) $\lambda$ and $\lambda$-weighted returns. The eligibility trace vector is initialized to zero at the beginning of the episode, and it is incremented on each time step by the value gradient, and then fades away by $\gamma\lambda$:

$$ \textbf{z}\_{-1} = \mathbf{0} $$
$$ \textbf{z}\_{t} = \gamma\lambda\textbf{z}\_{t-1} + \nabla\hat{v}\left(S\_{t}, \mathbf{w}\_{t}\right), 0 \leq t \leq T$$

The eligibility trace keeps track of which components of the weight vector contribute to recent state valuations. Here $\nabla\hat{v}\left(S\_{t}, \mathbf{w}\_{t}\right)$ is the feature vector.

The TD error for state-value prediction is:

$$ \delta\_{t} = R\_{t+1} + \gamma\hat{v}\left\(S\_{t+1}, \mathbf{w}\_{t}\right) - \hat{v}\left(S\_{t}, \mathbf{w}\_{t}\right) $$

In **TD_INLINE_MATH_1**, the weight vector is updated on each step proportional to the scalar TD error and the vector eligibility trace:

$$ \mathbf{w}\_{t+1} = \mathbf{w}\_{t} + \alpha\delta\mathbf{z}\_{t}  $$

Source: Sutton and Barto, Reinforcement Learning, 2nd Edition

TD Lambda

**Deformable convolutions** add 2D offsets to the regular grid sampling locations in the standard [convolution](https://paperswithcode.com/method/convolution). It enables free form deformation of the sampling grid. The offsets are learned from the preceding feature maps, via additional convolutional layers. Thus, the deformation is conditioned on the input features in a local, dense, and adaptive manner.

Deformable Convolution

Deformable Convolutional Networks

**GAGNN**, or **Group-aware Graph Neural Network**, is a hierarchical model for nationwide city air quality forecasting. The model constructs a city graph and a city group graph to model the spatial and latent dependencies between cities, respectively. GAGNN introduces differentiable grouping network to discover the latent dependencies among cities and generate city groups. Based on the generated city groups, a group correlation encoding module is introduced to learn the correlations between them, which can effectively capture the dependencies between city groups. After the graph construction, GAGNN implements message passing mechanism to model the dependencies between cities and city groups.

GAGNN

Group-Aware Graph Neural Network for Nationwide City Air Quality Forecasting

**DeepIR**, or **Deep InfraRed image processing**, is a thermal image processing framework for recovering high quality images from a very small set of images captured with camera motion. Enhancement is achieved by noting that camera motion, which is usually a hinderance, can be exploited to our advantage to separate a sequence of images into the scene-dependent radiant flux, and a slowly changing scene-independent non-uniformity. DeepIR combines the physics of microbolometer sensors, with powerful regularization capabilities by neural network-based representations. DeepIR relies on the key observation that jittering a camera, while unwanted in visible domain, is highly desirable in the thermal domain as it allows an accurate separation of the sensor-specific non-uniformities from the scene’s radiant flux.

DeepIR

Thermal Image Processing via Physics-Inspired Deep Networks

**Decaying Momentum**, or **Demon**, is a stochastic optimizer motivated by decaying the total contribution of a gradient to all future updates. By decaying the momentum parameter, the total contribution of a gradient to all future updates is decayed. A particular gradient term $g\_{t}$ contributes a total of  $\eta\sum\_{i}\beta^{i}$ of its "energy" to all future gradient updates, and this results in the geometric sum, $\sum^{\infty}\_{i=1}\beta^{i} = \beta\sum^{\infty}\_{i=0}\beta^{i} = \frac{\beta}{\left(1-\beta\right)}$. Decaying this sum results in the Demon algorithm. Letting $\beta\_{init}$ be the initial $\beta$; then at the current step $t$ with total $T$ steps, the decay routine is given by solving the below for $\beta\_{t}$:

$$ \frac{\beta\_{t}}{\left(1-\beta\_{t}\right)} =  \left(1-t/T\right)\beta\_{init}/\left(1-\beta\_{init}\right)$$

Where $\left(1-t/T\right)$ refers to the proportion of iterations remaining. Note that Demon typically requires no hyperparameter tuning as it is usually decayed to $0$ or a small negative value at time 
$T$. Improved performance is observed by delaying the decaying. Demon can be applied to any gradient descent algorithm with a momentum parameter.

Viet-Anh on Software

AI Glossary

Tech Terminology