**Continuous Bag-of-Words Word2Vec** is an architecture for creating word embeddings that uses $n$ future words as well as $n$ past words to create a word embedding. The objective function for CBOW is:

$$ J\_\theta = \frac{1}{T}\sum^{T}\_{t=1}\log{p}\left(w\_{t}\mid{w}\_{t-n},\ldots,w\_{t-1}, w\_{t+1},\ldots,w\_{t+n}\right) $$

In the CBOW model, the distributed representations of context are used to predict the word in the middle of the window. This contrasts with [Skip-gram Word2Vec](https://paperswithcode.com/method/skip-gram-word2vec) where the distributed representation of the input word is used to predict the context.

CBoW Word2Vec

Efficient Estimation of Word Representations in Vector Space

**ViP-DeepLab** is a model for depth-aware video panoptic segmentation. It extends Panoptic-[DeepLab](https://paperswithcode.com/method/deeplab) by adding a depth prediction head to perform monocular depth estimation and a next-frame instance branch which regresses to the object centers in frame $t$ for frame $t + 1$.  This allows the model to jointly perform video panoptic segmentation and monocular depth estimation.

ViP-DeepLab

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

**Anti-Alias Downsampling (AA)** aims to improve the shift-equivariance of deep networks. Max-pooling is inherently composed of two operations. The first operation is to densely evaluate the max operator and second operation is naive subsampling. AA is proposed as a low-pass filter between them to achieve practical anti-aliasing in any existing strided layer such as strided [convolution](https://paperswithcode.com/method/convolution). The smoothing factor can be adjusted by changing the blur kernel filter size, where a larger filter size results in increased blur.

Anti-Alias Downsampling

Making Convolutional Networks Shift-Invariant Again

**GBST**, or **Gradient-based Subword Tokenization Module**, is a soft gradient-based subword tokenization module that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network.  

GBST learns a position-wise soft selection over candidate subword blocks by scoring them with a scoring network. In contrast to prior tokenization-free methods, GBST learns interpretable latent subwords, which enables easy inspection of lexical representations and is more efficient than other byte-based models.

Gradient-Based Subword Tokenization

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Spatial CNN with UNet based Encoder-decoder and ConvLSTM

SCNN_UNet_ConvLSTM

A Hybrid Spatial-temporal Deep Learning Architecture for Lane Detection

Decoder architecture inspired on the [UNet++](https://paperswithcode.com/method/unet) structure and the [EfficientNet](https://paperswithcode.com/method/efficientnet) building blocks. Keeping the UNet++ structure, the EfficientUNet++ achieves higher performance and significantly lower computational complexity through two simple modifications:

* Replaces the 3x3 convolutions of the UNet++ with residual bottleneck blocks with depthwise convolutions
* Applies channel and spatial attention to the bottleneck feature maps using [concurrent spatial and channel squeeze & excitation (scSE)](https://paperswithcode.com/method/scse) blocks

EfficientUNet++

Encoder-Decoder Architectures for Clinically Relevant Coronary Artery Segmentation

AlphaFold is a deep learning based algorithm for accurate protein structure prediction. AlphaFold incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm.

Description from: [Highly accurate protein structure prediction with AlphaFold](https://paperswithcode.com/paper/highly-accurate-protein-structure-prediction)

Image credit: [DeepMind](https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology)

AlphaFold

Highly accurate protein structure prediction with AlphaFold

**Jukebox** is a model that generates music with singing in the raw audio domain. It tackles the long context of raw audio using a multi-scale [VQ-VAE](https://paperswithcode.com/method/vq-vae) to compress it to discrete codes, and modeling those using [autoregressive Transformers](https://paperswithcode.com/methods/category/autoregressive-transformers). It can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable.

Three separate VQ-VAE models are trained with different temporal resolutions. At each level, the input audio is segmented and encoded into latent vectors $\mathbf{h}\_{t}$, which are then quantized to the closest codebook vectors $\mathbf{e}\_{z\_{t}}$. The code $z\_{t}$ is a discrete representation of the audio that we later train our prior on. The decoder takes the sequence of codebook vectors and reconstructs the audio. The top level learns the highest degree of abstraction, since it is encoding longer audio per token while keeping the codebook size the same. Audio can be reconstructed using the codes at any one of the abstraction levels, where the least abstract bottom-level codes result in the highest-quality audio.

Jukebox

Jukebox: A Generative Model for Music

**IMPALA**, or the **Importance Weighted Actor Learner Architecture**, is an off-policy actor-critic framework that decouples acting from learning and learns from experience trajectories using [V-trace](https://paperswithcode.com/method/v-trace). Unlike the popular [A3C](https://paperswithcode.com/method/a3c)-based agents, in which workers communicate gradients with respect to the parameters of the policy to a central parameter server, IMPALA actors communicate trajectories of experience (sequences of states, actions, and rewards) to a centralized learner. Since the learner in IMPALA has access to full trajectories of experience we use a GPU to perform updates on mini-batches of trajectories while aggressively parallelising all time independent operations. 

This type of decoupled architecture can achieve very high throughput. However, because the policy used to generate a trajectory can lag behind the policy on the learner by several updates at the time of gradient calculation, learning becomes off-policy. The V-trace off-policy actor-critic algorithm is used to correct for this harmful discrepancy.

IMPALA

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

learn to explore the sample relationships via transformer networks

BatchFormer

BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning

ComplEx model trained with a nuclear norm regularizer; A relation prediction objective is added on top of the commonly used 1vsAll objective.

ComplEx-N3-RP

Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations

A **Graph Attention Network (GAT)** is a neural network architecture that operates on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods’ features, a GAT enables (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront.

See [here](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html) for an explanation by DGL.

Graph Attention Networks

**VGG Loss** is a type of content loss introduced in the [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://paperswithcode.com/paper/perceptual-losses-for-real-time-style) super-resolution and style transfer framework. It is an alternative to pixel-wise losses; VGG Loss attempts to be closer to perceptual similarity. The [VGG](https://paperswithcode.com/method/vgg) loss is based on the [ReLU](https://paperswithcode.com/method/relu) activation layers of the pre-trained 19 layer VGG network. With $\phi\_{i,j}$ we indicate the feature map obtained by the $j$-th [convolution](https://paperswithcode.com/method/convolution) (after activation) before the $i$-th maxpooling layer within the VGG19 network, which we consider given. We then define the VGG loss as the euclidean distance between the feature representations of a reconstructed image $G\_{\theta\_{G}}\left(I^{LR}\right)$ and the reference image $I^{HR}$:

$$ l\_{VGG/i.j} = \frac{1}{W\_{i,j}H\_{i,j}}\sum\_{x=1}^{W\_{i,j}}\sum\_{y=1}^{H\_{i,j}}\left(\phi\_{i,j}\left(I^{HR}\right)\_{x, y} - \phi\_{i,j}\left(G\_{\theta\_{G}}\left(I^{LR}\right)\right)\_{x, y}\right)^{2}$$ 

Here $W\_{i,j}$ and $H\_{i,j}$ describe the dimensions of the respective feature maps within the VGG network.

VGG Loss

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

**Local SGD** is a distributed training technique that runs [SGD](https://paperswithcode.com/method/sgd) independently in parallel on different workers and averages the sequences only once in a while.

Local SGD

Local SGD Converges Fast and Communicates Little

**Attention Mesh** is a neural network architecture for 3D face mesh prediction that uses attention to semantically meaningful regions. Specifically region-specific heads are employed that transform the feature maps with spatial transformers.

Attention Mesh

Attention Mesh: High-fidelity Face Mesh Prediction in Real-time

**Mirror Descent Policy Optimization (MDPO)** is a policy gradient algorithm based on the idea of iteratively solving a trust-region problem that minimizes a sum of two terms: a linearization of the standard RL objective function and a proximity term that restricts two consecutive updates to be close to each other. It is based on Mirror Descent, which is a general trust region method that
attempts to keep consecutive iterates close to each other.

MDPO

Mirror Descent Policy Optimization

**Gumbel-Softmax** is a continuous distribution that has the property that it can be smoothly annealed into a categorical distribution, and whose parameter gradients can be easily computed via the reparameterization trick.

Gumbel Softmax

Categorical Reparameterization with Gumbel-Softmax

**Local Prior Matching** is a semi-supervised objective for speech recognition that distills knowledge from a strong prior (e.g. a language model) to provide learning signal to a discriminative model trained on unlabeled speech. The LPM objective minimizes the cross entropy between the local prior and the model distribution, and is minimized when $q\_{y\mid{x}} = \bar{p}\_{y\mid{x}}$. Intuitively, LPM encourages the ASR model to assign posterior probabilities proportional to the linguistic probabilities of the proposed hypotheses.

Viet-Anh on Software

AI Glossary

Tech Terminology

Continuous Bag-of-Words Word2Vec