**Mobile Video Network**, or **MoViNet**, is a type of computation and memory efficient video network that can operate on streaming video for online inference. Three techniques are used to improve efficiency while reducing the peak memory usage of 3D CNNs. First, a video network search space is designed and [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) employed to generate efficient and diverse 3D CNN architectures. Second, a Stream Buffer technique is introduced that decouples memory from video clip duration, allowing 3D CNNs to embed arbitrary-length streaming video sequences for both training and inference with a small constant memory footprint. Third, a simple ensembling technique is used to improve accuracy further without sacrificing efficiency.

MoViNet

MoViNets: Mobile Video Networks for Efficient Video Recognition

**Position-Wise Feed-Forward Layer** is a type of [feedforward layer](https://www.paperswithcode.com/method/category/feedforwad-networks) consisting of two [dense layers](https://www.paperswithcode.com/method/dense-connections) that applies to the last dimension, which means the same dense layers are used for each position item in the sequence, so called position-wise.

Position-Wise Feed-Forward Layer

Attention Is All You Need

**CrossViT** is a type of [vision transformer](https://paperswithcode.com/method/vision-transformer) that uses a dual-branch architecture to extract multi-scale feature representations for image classification. The architecture combines image patches (i.e. tokens in a [transformer](https://paperswithcode.com/method/transformer)) of different sizes to produce stronger visual features for image classification. It processes small and large patch tokens with two separate branches of different computational complexities and these tokens are fused together multiple times to complement each other.

Fusion is achieved by an efficient [cross-attention module](https://paperswithcode.com/method/cross-attention-module), in which each transformer branch creates a non-patch token as an agent to exchange information with the other branch by attention. This allows for linear-time generation of the attention map in fusion instead of quadratic time otherwise.

CrossViT

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

**Cycle-CenterNet** is a table structure parsing approach built on [CenterNet](https://paperswithcode.com/method/centernet) that uses a cycle-pairing module to simultaneously detect and group tabular cells into structured tables. It also utilizes a pairing loss which enables the grouping of discrete cells into the structured tables.

Cycle-CenterNet

Parsing Table Structures in the Wild

**GreedyNAS** is a one-shot [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) method. Previous methods held the assumption that a supernet should give a reasonable ranking over all paths. They thus treat all paths equally, and spare much effort to train paths. However, it is harsh for a single supernet to evaluate accurately on such a huge-scale search space (eg, $7^{21}$). GreedyNAS eases the burden of supernet by encouraging focus more on evaluation of potentially-good candidates, which are identified using a surrogate portion of validation data. 

Concretely, during training, GreedyNAS utilizes a multi-path sampling strategy with rejection, and greedily filters the weak paths. The training efficiency is thus boosted since the training space has been greedily shrunk from all paths to those potentially-good ones. An exploration and exploitation policy is adopted by introducing an empirical candidate path pool.

GreedyNAS

GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet

**VDO-SLAM** is a feature-based stereo/RGB-D dynamic SLAM system that leverages image-based semantic information to simultaneously localise the robot, map the static and dynamic structure, and track motions of rigid objects in the scene. Input images are first pre-processed to generate instance-level object segmentation and dense optical flow. These are then used to track features on static background structure and dynamic objects. Camera poses and object motions estimated from feature tracks are then refined in a global batch optimisation, and a local map is maintained and updated with every new frame.

VDO-SLAM

VDO-SLAM: A Visual Dynamic Object-aware SLAM System

Temporal attention can be seen as a dynamic time selection mechanism determining when to pay attention, and is thus usually used for video processing.

Temporal attention

Jointly Attentive Spatial-Temporal Pooling Networks for Video-based Person Re-Identification

**Segmentation Transformer**, or **SETR**, is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based segmentation model. The transformer-alone encoder treats an input image as a sequence of image patches represented by learned patch embedding, and transforms the sequence with global self-attention modeling for discriminative feature representation learning. Concretely, we first decompose an image into a grid of fixed-sized patches, forming a sequence of patches. With a linear embedding layer applied to the flattened pixel vectors of every patch, we then obtain a sequence of feature embedding vectors as the input to a transformer. Given the learned features from the encoder
transformer, a decoder is then used to recover the original image resolution. Crucially there is no downsampling in spatial resolution but global context modeling at every layer of the encoder transformer.

SETR

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

TWEC is a method to generate temporal word embeddings: this method is efficient and it is based on a simple heuristic: we train an atemporal word embedding, the compass and we use this embedding to freeze one of the layers of the CBOW architecture. The frozen architecture is then used to train time-specific slices that are all comparable after training.

TWEC

Training Temporal Word Embeddings with a Compass

CondInst is a simple yet effective instance segmentation framework. It eliminates ROI cropping and feature alignment with the instance-aware mask heads. As a result, CondInst can solve instance segmentation with fully convolutional networks. CondInst is able to produce high-resolution instance masks without longer computational time. Extensive experiments show that CondInst can achieve even better performance and inference speed than [Mask R-CNN](https://paperswithcode.com/method/mask-r-cnn). It can be a strong alternative to previous ROI-based instance segmentation methods. Code is at https://github.com/aim-uofa/AdelaiDet.

CondInst

Conditional Convolutions for Instance Segmentation

**Disp R-CNN** is a 3D object detection system for stereo images. It utilizes an instance disparity estimation network (iDispNet) that predicts disparity only for pixels on objects of interest and learns a category-specific shape prior for more accurate disparity estimation. To address the challenge from scarcity of disparity annotation in training, a statistical shape model is used to generate dense disparity pseudo-ground-truth without the need of LiDAR point clouds.

Disp R-CNN

Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimation

**Meta Pseudo Labels** is a semi-supervised learning method that uses a teacher network to generate pseudo labels on unlabeled data to teach a student network. The teacher receives feedback from the student to inform the teacher to generate better pseudo labels. This feedback signal is used as a reward to train the teacher throughout the course of the student’s learning.

Meta Pseudo Labels

**Random Scaling** is a type of image data augmentation where we randomly change the scale the image between a specified range.

Random Scaling

A variant of the popular k-means algorithm that integrates constraint satisfaction into its objective function.

Original paper : Active Semi-Supervision for Pairwise Constrained Clustering, Basu et al. 2004

PCKMeans

This method combines multiple optimization techniques like [ADAM](https://paperswithcode.com/method/adam) and [SGD](https://paperswithcode.com/method/sgd) or PADAM. This method can be applied to any couple of optimizers.

Image credit: [Combining Optimization Methods Using an Adaptive Meta Optimizer](https://www.mdpi.com/1999-4893/14/6/186)

ATMO

Combining Optimization Methods Using an Adaptive Meta Optimizer

**Local Response Normalization** is a normalization layer that implements the idea of lateral inhibition. Lateral inhibition is a concept in neurobiology that refers to the phenomenon of an excited neuron inhibiting its neighbours: this leads to a peak in the form of a local maximum, creating contrast in that area and increasing sensory perception. In practice, we can either normalize within the same channel or normalize across channels when we apply LRN to convolutional neural networks.

$$ b_{c} = a_{c}\left(k + \frac{\alpha}{n}\sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta} $$

Where the size is the number of neighbouring channels used for normalization, $\alpha$ is multiplicative factor, $\beta$ an exponent and $k$ an additive factor

Local Response Normalization

ImageNet Classification with Deep Convolutional Neural Networks

**Inception-ResNet-v2-C** is an image model block for an 8 x 8 grid used in the [Inception-ResNet-v2](https://paperswithcode.com/method/inception-resnet-v2) architecture. It largely follows the idea of Inception modules - and grouped convolutions - but also includes residual connections.

Inception-ResNet-v2-C

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

**Pointwise Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) that uses a 1x1 kernel: a kernel that iterates through every single point. This kernel has a depth of however many channels the input image has. It can be used in conjunction with [depthwise convolutions](https://paperswithcode.com/method/depthwise-convolution) to produce an efficient class of convolutions known as [depthwise-separable convolutions](https://paperswithcode.com/method/depthwise-separable-convolution).

Image Credit: [Chi-Feng Wang](https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728)

Viet-Anh on Software

AI Glossary

Tech Terminology