A **Neural Cache**, or a **Continuous Cache**, is a module for language modelling which stores previous hidden states in memory cells. They are then used as keys to retrieve their corresponding word, that is the next word. There is no transformation applied to the storage during writing and reading.

More formally it exploits the hidden representations $h\_{t}$ to define a probability distribution over the words in the cache. As
illustrated in the Figure, the cache stores pairs $\left(h\_{i}, x\_{i+1}\right)$ of a hidden representation, and the word which was generated based on this representation (the vector $h\_{i}$ encodes the history $x\_{i}, \dots, x\_{1}$). At time $t$, we then define a probability distribution over words stored in the cache based on the stored hidden representations and the current one $h\_{t}$ as:

$$ p\_{cache}\left(w | h\_{1\dots{t}}, x\_{1\dots{t}}\right) \propto \sum^{t-1}\_{i=1}\mathcal{1}\_{\text{set}\left(w=x\_{i+1}\right)} \exp\left(θ\_{h}>h\_{t}^{T}h\_{i}\right) $$

where the scalar $\theta$ is a parameter which controls the flatness of the distribution. When $\theta$ is equal to zero, the probability distribution over the history is uniform, and the model is equivalent to a unigram cache model.

Neural Cache

Improving Neural Language Models with a Continuous Cache

A **Compact Global Descriptor** is an image model block for modelling interactions between positions across different dimensions (e.g., channels, frames). This descriptor enables subsequent convolutions to access the informative global features. It is a form of attention.

Compact Global Descriptor

Compact Global Descriptor for Neural Networks

**SentencePiece** is a subword tokenizer and detokenizer for natural language processing. It performs subword segmentation, supporting the byte-pair-encoding ([BPE](https://paperswithcode.com/method/bpe)) algorithm and unigram language model, and then converts this text into an id sequence guarantee perfect reproducibility of the normalization and subword segmentation.

SentencePiece

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

**SKEP** is a self-supervised pre-training method for sentiment analysis. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair.

SKEP contains two parts: (1) Sentiment masking recognizes the sentiment information of an input sequence based on automatically-mined sentiment knowledge, and produces a corrupted version by removing these informations. (2) Sentiment pre-training objectives require the transformer to recover the removed information from the corrupted version. The three prediction objectives on top are jointly optimized: Sentiment Word (SW) prediction (on $\left.\mathrm{x}\_{9}\right)$, Word Polarity (SP) prediction (on $\mathrm{x}\_{6}$ and $\mathbf{x}\_{9}$ ), Aspect-Sentiment pairs (AP) prediction (on $\mathbf{x}\_{1}$ ). Here, the smiley denotes positive polarity. Notably, on $\mathrm{x}\_{6}$, only SP is calculated without SW, as its original word has been predicted in the pair prediction on $\mathbf{x}\_{1}$.

SKEP

SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

**ALDEN**, or **Active Learning with DivErse iNterpretations**, is an active learning approach for text classification. With local interpretations in DNNs, ALDEN identifies linearly separable regions of samples. Then, it selects samples according to their diversity of local interpretations and queries their labels.

Specifically, we first calculate the local interpretations in DNN for each sample as the gradient backpropagated from the final
predictions to the input features. Then, we use the most diverse interpretation of words in a sample to measure its diverseness. Accordingly, we select unlabeled samples with the maximally diverse interpretations for labeling and retrain the model with these
labeled samples.

ALDEN

Deep Active Learning for Text Classification with Diverse Interpretations

A large amount of information is stored in data tables. Users can search for data tables using a keyword-based query. A table is composed primarily of data values that are organized in rows and columns providing implicit structural information. A table is usually accompanied by secondary information such as the caption, page title, etc., that form the textual information. Understanding the connection between the textual and structural information is an important yet neglected aspect in table retrieval as previous methods treat each source of information independently. In addition, users can search for data tables that are similar to an existing table, and this setting can be seen as a content-based table retrieval. In this paper, we propose StruBERT, a structure-aware BERT model that fuses the textual and structural information of a data table to produce context-aware representations for both textual and tabular content of a data table. StruBERT features are integrated in a new end-to-end neural ranking model to solve three table-related downstream tasks: keyword- and content-based table retrieval, and table similarity. We evaluate our approach using three datasets, and we demonstrate substantial improvements in terms of retrieval and classification metrics over state-of-the-art methods.

StruBERT

StruBERT: Structure-aware BERT for Table Search and Matching

**Skip-gram Word2Vec** is an architecture for computing word embeddings. Instead of using surrounding words to predict the center word, as with CBow Word2Vec, Skip-gram Word2Vec uses the central word to predict the surrounding words.

The skip-gram objective function sums the log probabilities of the surrounding $n$ words to the left and right of the target word $w\_{t}$ to produce the following objective:

$$J\_\theta = \frac{1}{T}\sum^{T}\_{t=1}\sum\_{-n\leq{j}\leq{n}, \neq{0}}\log{p}\left(w\_{j+1}\mid{w\_{t}}\right)$$

Skip-gram Word2Vec

Efficient Estimation of Word Representations in Vector Space

**Big-Little Net** is a convolutional neural network architecture for learning multi-scale feature representations. This is achieved by using a multi-branch network, which has different computational complexity at different branches with different resolutions. Through frequent merging of features from branches at distinct scales, the model obtains multi-scale features while using less computation.

It consists of Big-Little Modules, which have two branches: each of which represents a separate block from a deep model and a less deep counterpart. The two branches are fused with linear combination + unit weights. These two branches are known as Big-Branch (more layers and channels at low resolutions) and Little-Branch (fewer layers and channels at high resolution).

Big-Little Net

Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition

Method for one-shot visual search of object parts / one-shot semantic part correspondence. Given a single reference image of an object with annotated affordance regions, it segments semantically corresponding parts within a target scene. AffCorrs is used to find corresponding affordances both for intra- and inter-class one-shot part segmentation.

AffCorrs

A **Multiscale Dilated Convolution Block** is an Inception-style convolutional block motivated by the ideas that image features naturally occur at multiple scales, that a network’s expressivity is proportional to the range of functions it can represent divided by its total number of parameters, and by the desire to efficiently expand a network’s receptive field. The Multiscale [Dilated Convolution](https://paperswithcode.com/method/dilated-convolution) (MDC) block applies a single $F\times{F}$ filter at multiple dilation factors, then performs a weighted elementwise sum of each dilated filter’s output, allowing the network to simultaneously learn a set of features and the relevant scales at which those features occur with a minimal increase in parameters. This also rapidly expands the network’s receptive field without requiring an increase in depth or the number of parameters.

Multiscale Dilated Convolution Block

Neural Photo Editing with Introspective Adversarial Networks

**Seq2Seq**, or **Sequence To Sequence**, is a model used in sequence prediction tasks, such as language modelling and machine translation. The idea is to use one [LSTM](https://paperswithcode.com/method/lstm), the *encoder*, to read the input sequence one timestep at a time, to obtain a large fixed dimensional vector representation (a context vector), and then to use another LSTM, the *decoder*, to extract the output sequence
from that vector. The second LSTM is essentially a recurrent neural network language model except that it is conditioned on the input sequence.

(Note that this page refers to the original seq2seq not general sequence-to-sequence models)

Seq2Seq

Sequence to Sequence Learning with Neural Networks

Inspired on the widely known [spatial squeeze and channel excitation (SE)](https://paperswithcode.com/method/squeeze-and-excitation-block) block, the sSE block performs channel squeeze and spatial excitation, to recalibrate the feature maps spatially and achieve more fine-grained image segmentation.

Channel Squeeze and Spatial Excitation

Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks

**Region of Interest Warping**, or **RoIWarp**, is a form of [RoIPool](https://paperswithcode.com/method/roi-pooling) that is differentiable with respect to the box position. In practice, this takes the form of a RoIWarp layer followed by a standard [Max Pooling](https://paperswithcode.com/method/max-pooling) layer. The RoIWarp layer crops a feature map region and warps it into a target size by interpolation.

RoIWarp

Instance-aware Semantic Segmentation via Multi-task Network Cascades

ZeRO-Offload is a sharded data parallel method for distributed training. It exploits both CPU memory and compute for offloading, while offering a clear path towards efficiently scaling on multiple GPUs by working with [ZeRO-powered data parallelism](https://www.paperswithcode.com/method/zero). The symbiosis allows ZeRO-Offload to maintain a single copy of the optimizer states on the CPU memory regardless of the data parallel degree. Furthermore, it keeps the aggregate communication volume between GPU and CPU, as well as the aggregate CPU computation a constant regardless of data parallelism, allowing ZeRO-Offload to effectively utilize the linear increase in CPU compute with the increase in the data parallelism degree.

ZeRO-Offload

ZeRO-Offload: Democratizing Billion-Scale Model Training

**Rectified Linear Units**, or **ReLUs**, are a type of activation function that are linear in the positive dimension, but zero in the negative dimension. The kink in the function is the source of the non-linearity. Linearity in the positive dimension has the attractive property that it prevents non-saturation of gradients (contrast with [sigmoid activations](https://paperswithcode.com/method/sigmoid-activation)), although for half of the real line its gradient is zero.

$$ f\left(x\right) = \max\left(0, x\right) $$

ReLU

**InstaBoost** is a data augmentation technique for instance segmentation that utilises existing instance mask annotations.

Intuitively in a small neighbor area of $(x_0, y_0, 1, 0)$, the probability map $P(x, y, s, r)$ should be high-valued since images are usually continuous and redundant in pixel level. Based on this, InstaBoost is a form of augmentation where we apply object jittering that randomly samples transformation tuples from the neighboring space of identity transform $(x_0, y_0, 1, 0)$ and paste the cropped object following affine transform $\mathbf{H}$.

InstaBoost

InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting

A **LAPGAN**, or **Laplacian Generative Adversarial Network**, is a type of generative adversarial network that has a [Laplacian pyramid](https://paperswithcode.com/method/laplacian-pyramid) representation. In the sampling procedure following training, we have a set of generative convnet models {$G\_{0}, \dots , G\_{K}$}, each of which captures the distribution of coefficients $h\_{k}$ for natural images at a different level of the Laplacian pyramid. Sampling an image is akin to a reconstruction procedure, except that the generative
models are used to produce the $h\_{k}$’s:

$$ \tilde{I}\_{k} = u\left(\tilde{I}\_{k+1}\right) + \tilde{h}\_{k} = u\left(\tilde{I}\_{k+1}\right) + G\_{k}\left(z\_{k}, u\left(\tilde{I}\_{k+1}\right)\right)$$

The recurrence starts by setting $\tilde{I}\_{K+1} = 0$ and using the model at the final level $G\_{K}$ to generate a residual image $\tilde{I}\_{K}$ using noise vector $z\_{K}$: $\tilde{I}\_{K} = G\_{K}\left(z\_{K}\right)$. Models at all levels except the final are conditional generative models that take an upsampled version of the current image $\tilde{I}\_{k+1}$ as a conditioning variable, in addition to the noise vector $z\_{k}$.

The generative models {$G\_{0}, \dots, G\_{K}$} are trained using the CGAN approach at each level of the pyramid. Specifically, we construct a Laplacian pyramid from each training image $I$. At each level we make a stochastic choice (with equal probability) to either (i) construct the coefficients $h\_{k}$ either using the standard Laplacian pyramid coefficient generation procedure or (ii) generate them using $G\_{k}:

$$ \tilde{h}\_{k} = G\_{k}\left(z\_{k}, u\left(I\_{k+1}\right)\right) $$

Here $G\_{k}$ is a convnet which uses a coarse scale version of the image $l\_{k} = u\left(I\_{k+1}\right)$ as an input, as well as noise vector $z\_{k}$. $D\_{k}$ takes as input $h\_{k}$ or $\tilde{h}\_{k}$, along with the low-pass image $l\_{k}$ (which is explicitly added to $h\_{k}$ or $\tilde{h}\_{k}$ before the first [convolution](https://paperswithcode.com/method/convolution) layer), and predicts if the image was real or
generated. At the final scale of the pyramid, the low frequency residual is sufficiently small that it
can be directly modeled with a standard [GAN](https://paperswithcode.com/method/gan): $\tilde{h}\_{K} = G\_{K}\left(z\_{K}\right)$ and $D\_{K}$ only has $h\_{K}$ or $\tilde{h}\_{K}$ as input.

Breaking the generation into successive refinements is the key idea. We give up any “global” notion of fidelity; an attempt is never made to train a network to discriminate between the output of a cascade and a real image and instead the focus is on making each step plausible.

LAPGAN

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

**VOS** is a type of video object segmentation model consisting of two network components. The target appearance model consists of a light-weight module, which is learned during the inference stage using fast optimization techniques to predict a coarse but robust target segmentation. The segmentation model is exclusively trained offline, designed to process the coarse scores into high quality segmentation masks.

AI Glossary