**PIoU Loss** is a loss function for oriented object detection which is formulated to exploit both the angle and IoU for accurate oriented bounding box regression. The PIoU loss is derived from IoU metric with a pixel-wise form.

PIoU Loss

PIoU Loss: Towards Accurate Oriented Object Detection in Complex Environments

**Grab** is a sensor processing system for cashier-free shopping. Grab needs to accurately identify and track customers, and associate each shopper with items he or she retrieves from shelves. To do this, it uses a keypoint-based pose tracker as a building block for identification and tracking, develops robust feature-based face trackers, and algorithms for associating and tracking arm movements. It also uses a probabilistic framework to fuse readings from camera, weight and RFID sensors in order to accurately assess which shopper picks up which item.

Grab

Grab: Fast and Accurate Sensor Processing for Cashier-Free Shopping

**Multiple Random Window Discriminator** is a discriminator used for the [GAN-TTS](https://paperswithcode.com/method/gan-tts) text-to-speech architecture. These discriminators operate on randomly sub-sampled fragments of the real or generated samples. The ensemble allows for the evaluation of audio in different complementary ways, and is obtained by taking
a Cartesian product of two parameter spaces: (i) the size of the random windows fed into the discriminator; (ii) whether a discriminator is conditioned on linguistic and pitch features. For example,
in the authors' best-performing model, they consider five window sizes (240, 480, 960, 1920, 3600 samples), which yields 10 discriminators in total. 

Using random windows of different size, rather than the full generated sample, has a data augmentation effect and also reduces the computational complexity of RWDs. In the first layer of each discriminator, the MRWD reshapes (downsamples) the input raw waveform to a constant
temporal dimension $\omega = 240$ by moving consecutive blocks of samples into the channel dimension, i.e. from $\left[\omega\_{k}, 1\right]$ to $\left[\omega, k\right]$, where $k$ is the downsampling factor (e.g. $k = 8$ for input window size $1920$). This way, all the RWDs have the same architecture and similar computational complexity despite different window sizes. 

The conditional discriminators have access to linguistic and pitch features, and can measure whether
the generated audio matches the input conditioning. This means that random windows in conditional
discriminators need to be aligned with the conditioning frequency to preserve the correspondence
between the waveform and linguistic features within the sampled window. This limits the valid sampling to that of the frequency of the conditioning signal (200Hz, or every 5ms). The unconditional
discriminators, on the contrary, only evaluate whether the generated audio sounds realistic regardless
of the conditioning. The random windows for these discriminators are sampled without constraints
at full 24kHz frequency, which further increases the amount of training data. 

For the architecture, the discriminators consists of blocks (DBlocks) that are similar to the [GBlocks](https://paperswithcode.com/method/gblock) used in the generator, but without batch normalisation. Unconditional RWDs are composed entirely of DBlocks. In conditional RWDs, the input waveform is gradually downsampled by DBlocks, until the temporal dimension of the activation is equal to that of the conditioning, at which point a conditional [DBlock](https://paperswithcode.com/method/dblock) is used. This joint information is then passed to the remaining DBlocks, whose final output is average-pooled to obtain a scalar. The dilation factors in the DBlocks’ convolutions follow the pattern 1, 2, 1, 2 – unlike the generator, the discriminator operates on a relatively small window, and the authors did not observe any benefit from using larger dilation factors.

Multiple Random Window Discriminator

High Fidelity Speech Synthesis with Adversarial Networks

In this paper, we propose a method for obtaining sentence-level embeddings. While the problem of obtaining word-level embeddings is very well studied, we propose a novel method for obtaining sentence-level embeddings. This is obtained by a simple method in the context of solving the paraphrase generation task. If we use a sequential encoder-decoder model for generating paraphrase, we would like the generated paraphrase to be semantically close to the original sentence. One way to ensure this is by adding constraints for true paraphrase embeddings to be close and unrelated paraphrase candidate sentence embeddings to be far. This is ensured by using a sequential pair-wise discriminator that shares weights with the encoder. This discriminator is trained with a suitable loss function. Our loss function penalizes paraphrase sentence embedding distances from being too large. This loss is used in combination with a sequential encoder-decoder network. We also validate our method by evaluating the obtained embeddings for a sentiment analysis task. The proposed method results in semantic embeddings and provide competitive results on the paraphrase generation and sentiment analysis task on standard dataset. These results are also shown to be statistically significant.




Github Link:https://github.com/dev-chauhan/PQG-pytorch.

2
The PQG dataset is available on this link: https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs.

3
website: https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs.

4
we report same baseline results as mentioned in [10]

5
website: www.kaggle.com/c/sentiment-analysis-on-movie-reviews.

6
Code: https://github.com/dev-chauhan/PQG-pytorch.

EDLPS

**Herring** is a parameter server based distributed training method. It combines AWS's Elastic Fabric [Adapter](https://paperswithcode.com/method/adapter) (EFA) with a novel parameter sharding technique that makes better use of the available network bandwidth.  Herring uses EFA and balanced fusion buffer to optimally use the total bandwidth available across all nodes in the cluster. Herring reduces gradients hierarchically, reducing them inside the node first and then reducing across nodes. This enables more efficient use of PCIe bandwidth in the node and helps keep the gradient averaging related burden on GPU low.

Herring

**Greedy Policy Search** (GPS) is a simple algorithm that learns a policy for test-time data augmentation based on the predictive performance on a validation set. GPS starts with an empty policy and builds it in an iterative fashion. Each step selects a sub-policy that provides the largest improvement in calibrated log-likelihood of ensemble predictions and adds it to the current policy.

Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation

**Prediction-aware One-To-One**, or **POTO**, is an assignment rule for object detection which dynamically assigns the foreground samples according to the quality of classification and regression simultaneously.

POTO

End-to-End Object Detection with Fully Convolutional Network

**TaBERT** is a pretrained language model (LM) that jointly learns representations for natural language sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. 

In summary, TaBERT's process for learning representations for NL sentences is as follows: Given an utterance $u$ and a table $T$, TaBERT first creates a content snapshot of $T$. This snapshot consists of sampled rows that summarize the information in $T$ most relevant to the input utterance. The model then linearizes each row in the snapshot, concatenates each linearized row with the utterance, and uses the concatenated string as input to a Transformer model, which outputs row-wise encoding vectors of utterance tokens and cells. The encodings for all the rows in the snapshot are fed into a series of vertical self-attention layers, where a cell representation (or an utterance token representation) is computed by attending to vertically-aligned vectors of the same column (or the same NL token). Finally, representations for each utterance token and column are generated from a pooling layer.

TaBERT

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

A **Scatter Connection** is a type of connection that allows a vector to be "scattered" onto a layer representing a map, so that a vector at a specific location corresponds to objects of interest at that location (e.g. units in Starcraft II). This allows for the integration of spatial and non-spatial features.

Scatter Connection

**Sigmoid Activations** are a type of activation function for neural networks:

$$f\left(x\right) = \frac{1}{\left(1+\exp\left(-x\right)\right)}$$

Some drawbacks of this activation that have been noted in the literature are: sharp damp gradients during backpropagation from deeper hidden layers to inputs, gradient saturation, and slow convergence.

Sigmoid Activation

The **Two Time-scale Update Rule (TTUR)** is an update rule for generative adversarial networks trained with stochastic gradient descent. TTUR has an individual learning rate for both the discriminator and the generator. The main premise is that the discriminator converges to a local minimum when the generator is fixed. If the generator changes slowly enough, then the discriminator still converges, since the generator perturbations are small. Besides ensuring convergence, the performance may also improve since the discriminator must first learn new patterns before they are transferred to the generator. In contrast, a generator which is overly fast, drives the discriminator steadily into new regions without capturing its gathered information.

TTUR

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

The  overhead  cost  of  training  multiple  deep  neural networks  could  be  very  high  in  terms  of  the  training  time, hardware, and computational resource requirement and often acts  as  obstacle  for  creating  deep  ensembles.  To  overcome these barriers Huang et al. proposed a unique method to create  ensemble  which  at  the  cost  of  training  one  model, yields  multiple  constituent  model  snapshots  that  can  be ensembled together to create a strong learner. The core idea behind the concept is to make the model converge to several local minima along the optimization path and save the model parameters at these local minima points. During the training phase, a neural network would traverse through many such points. The lowest of all such local minima is known as the Global Minima. The larger the model, more are the number of parameters and larger the number of local minima points. This implies, there are discrete sets of weights and biases, at which  the  model  is  making  fewer  errors.  So,  every  such minimum  can  be  considered a  weak  but  a  potential learner model for the problem being solved. Multiple such snapshot of  weights  and  biases  are  recorded  which  can  later  be ensembled to get a better generalized model which makes the least amount of mistakes.

Snapshot Ensembles

Snapshot Ensembles: Train 1, get M for free

**Attention Free Transformer**, or **AFT**, is an efficient variant of a [multi-head attention module](https://paperswithcode.com/method/multi-head-attention) that eschews [dot product self attention](https://paperswithcode.com/method/scaled). In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes.

Given the input $X$, AFT first linearly transforms them into $Q=X W^{Q}, K=X W^{K}, V=X W^{V}$, then performs following operation:

$$
Y=f(X) ; Y\_{t}=\sigma\_{q}\left(Q\_{t}\right) \odot \frac{\sum\_{t^{\prime}=1}^{T} \exp \left(K\_{t^{\prime}}+w\_{t, t^{\prime}}\right) \odot V\_{t^{\prime}}}{\sum\_{t^{\prime}=1}^{T} \exp \left(K\_{t^{\prime}}+w\_{t, t^{\prime}}\right)}
$$

where $\odot$ is the element-wise product; $\sigma\_{q}$ is the nonlinearity applied to the query with default being sigmoid; $w \in R^{T \times T}$ is the learned pair-wise position biases.

Explained in words, for each target position $t$, AFT performs a weighted average of values, the result of which is combined with the query with element-wise multiplication. In particular, the weighting is simply composed of the keys and a set of learned pair-wise position biases. This provides the immediate advantage of not needing to compute and store the expensive attention matrix, while maintaining the global interactions between query and values as MHA does.

Attention Free Transformer

An Attention Free Transformer

**Self-Adversarial Negative Sampling** is a negative sampling technique used for methods like [word embeddings](https://paperswithcode.com/methods/category/word-embeddings) and [knowledge graph embeddings](https://paperswithcode.com/methods/category/graph-embeddings). The traditional negative sampling loss from word2vec for optimizing distance-based models be written as:

$$ L = −\log\sigma\left(\gamma − d\_{r}\left(\mathbf{h}, \mathbf{t}\right)\right) − \sum^{n}\_{i=1}\frac{1}{k}\log\sigma\left(d\_{r}\left(\mathbf{h}^{'}\_{i}, \mathbf{t}^{'}\_{i}\right) - \gamma\right) $$

where $\gamma$ is a fixed margin, $\sigma$ is the sigmoid function, and $\left(\mathbf{h}^{'}\_{i}, r, \mathbf{t}^{'}\_{i}\right)$ is the $i$-th negative triplet. 

The negative sampling loss samples the negative triplets in a uniform way. Such a uniform negative sampling suffers the problem of inefficiency since many samples are obviously false as training goes on, which does not provide any meaningful information. Therefore, the authors propose an approach called self-adversarial negative sampling, which samples negative triples according to the current embedding model. Specifically, we sample negative triples from the following distribution:

$$ p\left(h^{'}\_{j}, r, t^{'}\_{j} | \text{set}\left(h\_{i}, r\_{i}, t\_{i} \right) \right) = \frac{\exp\alpha{f}\_{r}\left(\mathbf{h}^{'}\_{j}, \mathbf{t}^{'}\_{j}\right)}{\sum\_{i=1}\exp\alpha{f}\_{r}\left(\mathbf{h}^{'}\_{i}, \mathbf{t}^{'}\_{i}\right)} $$

where $\alpha$ is the temperature of sampling. Moreover, since the sampling procedure may be costly, the authors treat the above probability as the weight of the negative sample. Therefore, the final negative sampling loss with self-adversarial training takes the following form:

$$ L = −\log\sigma\left(\gamma − d\_{r}\left(\mathbf{h}, \mathbf{t}\right)\right) − \sum^{n}\_{i=1}p\left(h^{'}\_{i}, r, t^{'}\_{i}\right)\log\sigma\left(d\_{r}\left(\mathbf{h}^{'}\_{i}, \mathbf{t}^{'}\_{i}\right) - \gamma\right) $$

Self-Adversarial Negative Sampling

RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

**SCCL**, or **Supporting Clustering with Contrastive Learning**, is a framework to leverage contrastive learning to promote better separation in unsupervised clustering. It combines the top-down clustering with the bottom-up instance-wise contrastive learning to achieve better inter-cluster distance and intra-cluster distance. During training, we jointly optimize a clustering loss over the original data instances and an instance-wise contrastive loss over the associated augmented pairs.

SCCL

Supporting Clustering with Contrastive Learning

**Global Sub-Sampled Attention**, or **GSA**, is a local [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) used in the [Twins-SVT](https://paperswithcode.com/method/twins-svt) architecture. 

A single representative is used to summarize the key information for each of $m \times n$ subwindows and the representative is used to communicate with other sub-windows (serving as the key in self-attention), which can reduce the cost to $\mathcal{O}(m n H W d)=\mathcal{O}\left(\frac{H^{2} W^{2} d}{k\_{1} k\_{2}}\right)$. This is essentially equivalent to using the sub-sampled feature maps as the key in attention operations, and thus it is termed global sub-sampled attention (GSA). 

If we alternatively use the [LSA](https://paperswithcode.com/method/locally-grouped-self-attention) and GSA like [separable convolutions](https://paperswithcode.com/method/depthwise-separable-convolution) (depth-wise + point-wise). The total computation cost is $\mathcal{O}\left(\frac{H^{2} W^{2} d}{k\_{1} k\_{2}}+k\_{1} k\_{2} H W d\right) .$ We have:

$$\frac{H^{2} W^{2} d}{k\_{1} k\_{2}}+k_{1} k_{2} H W d \geq 2 H W d \sqrt{H W} $$ 

The minimum is obtained when $k\_{1} \cdot k\_{2}=\sqrt{H W}$. Note that $H=W=224$ is popular in classification. Without loss of generality, square sub-windows are used, i.e., $k\_{1}=k\_{2}$. Therefore, $k\_{1}=k\_{2}=15$ is close to the global minimum for $H=W=224$. However, the network is designed to include several stages with variable resolutions. Stage 1 has feature maps of $56 \times 56$, the minimum is obtained when $k\_{1}=k\_{2}=\sqrt{56} \approx 7$. Theoretically, we can calibrate optimal $k\_{1}$ and $k\_{2}$ for each of the stages. For simplicity, $k\_{1}=k\_{2}=7$ is used everywhere. As for stages with lower resolutions, the summarizing window-size of GSA is controlled to avoid too small amount of generated keys. Specifically, the sizes of 4,2 and 1 are used for the last three stages respectively.

Global Sub-Sampled Attention

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

1D Convolutional Neural Networks are similar to well known and more established 2D Convolutional Neural Networks. 1D Convolutional Neural Networks are used mainly used on text and 1D signals.

1D CNN

Convolutional Neural Network and Rule-Based Algorithms for Classifying 12-lead ECGs

**Inception-A** is an image model block used in the [Inception-v4](https://paperswithcode.com/method/inception-v4) architecture.

Viet-Anh on Software

AI Glossary

Tech Terminology

PIoU Loss

Grab

Multiple Random Window Discriminator

Encoder-Decoder model with local and pairwise loss along with shared encoder and discriminator network (EDLPS)

Herring

Greedy Policy Search

Prediction-aware One-To-One