A **Focal Loss** function addresses class imbalance during training in tasks like object detection. Focal loss applies a modulating term to the cross entropy loss in order to focus learning on hard misclassified examples. It is a dynamically scaled cross entropy loss, where the scaling factor decays to zero as confidence in the correct class increases. Intuitively, this scaling factor can automatically down-weight the contribution of easy examples during training and rapidly focus the model on hard examples. 

Formally, the Focal Loss adds a factor $(1 - p\_{t})^\gamma$ to the standard cross entropy criterion. Setting $\gamma>0$ reduces the relative loss for well-classified examples ($p\_{t}>.5$), putting more focus on hard, misclassified examples. Here there is tunable *focusing* parameter $\gamma \ge 0$. 

$$ {\text{FL}(p\_{t}) = - (1 - p\_{t})^\gamma \log\left(p\_{t}\right)} $$

Focal Loss

Focal Loss for Dense Object Detection

**Good Feature Matching** is an active map-to-frame feature matching method. Feature matching effort is tied to submatrix selection, which has combinatorial time complexity and requires choosing a scoring metric. Via simulation, the Max-logDet matrix revealing metric is shown to perform best.

Good Feature Matching

Good Feature Matching: Towards Accurate, Robust VO/VSLAM with Low Latency

**Singular Value Clipping (SVC)** is an adversarial training technique used by [TGAN](https://paperswithcode.com/method/tgan) to enforce the 1-Lipschitz constraint of the [WGAN](https://paperswithcode.com/method/wgan) objective. It is a constraint to all linear layers in the discriminator that satisfies the spectral norm of weight parameter $W$ is equal or less than one. This
means that the singular values of weight matrix are all one or less. Therefore singular value decomposition (SVD) is performed after a parameter update, replacing all the singular values larger than one with one, and the parameters are reconstructed with them. The same operation is applied to convolutional layers by interpreting a higher order tensor in weight parameter as a matrix $\hat{W}$.

Singular Value Clipping

Temporal Generative Adversarial Nets with Singular Value Clipping

This work introduces a novel activation unit that can be efficiently employed in deep neural nets (DNNs) and performs significantly better than the traditional Rectified Linear Units ([ReLU](https://paperswithcode.com/method/relu)). The function developed is a two parameter version of the specialized Richard's Curve and we call it Adaptive Richard's Curve weighted Activation (ARiA). This function is non-monotonous, analogous to the newly introduced [Swish](https://paperswithcode.com/method/swish), however allows a precise control over its non-monotonous convexity by varying the hyper-parameters. We first demonstrate the mathematical significance of the two parameter ARiA followed by its application to benchmark problems such as MNIST, CIFAR-10 and CIFAR-100, where we compare the performance with ReLU and Swish units. Our results illustrate a significantly superior performance on all these datasets, making ARiA a potential replacement for ReLU and other activations in DNNs.

ARiA

ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

**DeCLUTR** is an approach for learning universal sentence embeddings that utilizes a self-supervised objective that does not require labelled training data. The objective learns universal sentence embeddings by training an encoder to minimize the distance between the embeddings of textual segments randomly sampled from nearby in the same document.

DeCLUTR

DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

**WaveTTS** is a [Tacotron](https://paperswithcode.com/method/tacotron)-based text-to-speech architecture that has two loss functions: 1) time-domain loss, denoted as the waveform loss, that measures the distortion between the natural and generated waveform; and 2) frequency-domain loss, that measures the Mel-scale acoustic feature loss between the natural and generated acoustic features.

The motivation arises from [Tacotron 2](https://paperswithcode.com/method/tacotron-2). Here its feature prediction network is trained independently of the [WaveNet](https://paperswithcode.com/method/wavenet) vocoder. At run-time, the feature prediction network and WaveNet vocoder are artificially joined together. As a result, the framework suffers from the mismatch between frequency-domain acoustic features and time-domain waveform. To overcome such mismatch, WaveTTS uses a joint time-frequency domain loss for TTS that effectively improves the synthesized voice quality.

WaveTTS

WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss

**TSRUp**, or **Transformation-based Spatial Recurrent Unit p**, is a modification of a [ConvGRU](https://paperswithcode.com/method/cgru) used in the [TriVD-GAN](https://paperswithcode.com/method/trivd-gan) architecture for video generation.

It largely follows [TSRUc](https://paperswithcode.com/method/tsruc), but computes $\theta$, $u$ and $c$ in parallel given $x\_{t}$ and $h\_{t−1}$, yielding the following replacement for the $c$ update equation:

$$ c = \rho\left(W\_{c} \star\_{n}\left[h\_{t-1}; x\_{t}\right] + b\_{c} \right) $$

In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and [ReLU](https://paperswithcode.com/method/relu) functions respectively and the $\star\_{n}$ represents a [convolution](https://paperswithcode.com/method/convolution) with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUp

Transformation-based Adversarial Video Prediction on Large-Scale Data

**Panoptic-PolarNet** is a point cloud segmentation framework for LiDAR point clouds. It learns both semantic segmentation and class-agnostic instance clustering in a single inference network using a polar Bird's Eye View (BEV) representation, enabling the authors to circumvent the issue of occlusion among instances in urban street scenes. We first encode the raw point cloud data with $K$ features into a fixed-size representation on the polar BEV map. Next, we use a single backbone encoder-decoder network to generate semantic prediction, center [heatmap](https://paperswithcode.com/method/heatmap) and offset regression. Finally, we merge these outputs via a voting-based fusion to yield the panoptic segmentation result.

Panoptic-PolarNet

Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation

**GPT-2** is a [Transformer](https://paperswithcode.com/methods/category/transformers) architecture that was notable for its size (1.5 billion parameters) on its release. The model is pretrained on a WebText dataset - text from 45 million website links. It largely follows the previous [GPT](https://paperswithcode.com/method/gpt) architecture with some modifications:

- [Layer normalization](https://paperswithcode.com/method/layer-normalization) is moved to the input of each sub-block, similar to a
pre-activation residual network and an additional layer normalization was added after the final self-attention block. 

- A modified initialization which accounts for the accumulation on the residual path with model depth
is used. Weights of residual layers are scaled at initialization by a factor of $1/\sqrt{N}$ where $N$ is the number of residual layers. 

- The vocabulary is expanded to 50,257. The context size is expanded from 512 to 1024 tokens and
a larger batch size of 512 is used.

GPT-2

Language Models are Unsupervised Multitask Learners

**VirText**, or **Visual representations from Textual annotations** is a pretraining approach using semantically dense captions to learn visual representations. First a ConvNet and [Transformer](https://paperswithcode.com/method/transformer) are jointly trained from scratch to generate natural language captions for images. Then, the learned features are transferred to downstream visual recognition tasks.

VirTex

VirTex: Learning Visual Representations from Textual Annotations

OODformer is a [transformer](https://paperswithcode.com/method/transformer)-based OOD detection architecture that leverages the contextualization capabilities of the transformer. Incorporating the transformer as the principal feature extractor allows to exploit the object concepts and their discriminate attributes along with their co-occurrence via [visual attention](https://paperswithcode.com/method/visual-attention). 

OODformer employs [ViT](method/vision-transformer) and its data efficient variant [DeiT](/method/deit). Each encoder layer consist of multi-head self attention and a multi-layer perception block. The combination of MSA and MLP layers in the encoder jointly encode the attributes' importance, associated correlation, and co-occurrence. The [class] token (a representative of an image $x$) consolidated multiple attributes and their related features via the global context. The [class] token from the final layer is used for OOD detection in two ways; first, it is passed to $
F_{\text {classifier }}\left(x_{\text {feat }}\right)$  for softmax confidence score, and second it is used for latent space distance calculation.

OODformer

OODformer: Out-Of-Distribution Detection Transformer

**Stacked Hourglass Networks** are a type of convolutional neural network for pose estimation. They are based on the successive steps of pooling and upsampling that are done to produce a final set of predictions.

Stacked Hourglass Network

Stacked Hourglass Networks for Human Pose Estimation

A **Residual GRU** is a [gated recurrent unit (GRU)](https://paperswithcode.com/method/gru) that incorporates the idea of residual connections from [ResNets](https://paperswithcode.com/method/resnet).

Residual GRU

Full Resolution Image Compression with Recurrent Neural Networks

**Circular Smooth Label** (CSL) is a classification-based rotation detection technique for arbitrary-oriented object detection. It is used for circularly distributed angle classification and addresses the periodicity of the angle and increases the error tolerance to adjacent angles.

On the Arbitrary-Oriented Object Detection: Classification based Approaches Revisited

**Sample Consistency Network (SCNet)** is a method for instance segmentation which ensures the IoU distribution of the samples at training time are as close to that at inference time. To this end, only the outputs of the last box stage are used for mask predictions at both training and inference. The Figure shows the IoU distribution of the samples going to the mask branch at training time with/without sample consistency compared to that at inference time.

SCNet

SCNet: Training Inference Sample Consistency for Instance Segmentation

PipeDream is an asynchronous pipeline parallel strategy for training large neural networks. It adds inter-batch pipelining to intra-batch parallelism to further improve parallel training throughput, helping to better overlap computation with communication and reduce the amount of communication when possible.

PipeDream

Branch attention can be seen as a dynamic branch selection mechanism: which to pay attention to, used with a multi-branch structure.

Branch attention

Training Very Deep Networks

A **Residual SRM** is a module for convolutional neural networks that uses a [Style-based Recalibration Module](https://paperswithcode.com/method/style-based-recalibration-module) within a [residual block](https://paperswithcode.com/method/residual-block) like structure. The Style-based Recalibration Module (SRM) adaptively recalibrates intermediate feature maps by exploiting their styles.

Viet-Anh on Software

AI Glossary

Tech Terminology