**STDC**, or **Short-Term Dense Concatenate**, is a module for semantic segmentation to extract deep features with scalable
receptive field and multi-scale information. It aims to remove structure redundancy in the BiSeNet architecture, specifically BiSeNet adds an extra path to encode spatial information which can be time-consuming,. Instead, STDC gradually reduces the dimension of feature maps and use the aggregation of them for image representation.

We concatenate response maps from multiple continuous layers, each of which encodes input image/feature in different scales and respective fields, leading to multi-scale feature representation. To speed up, the filter size of layers is gradually reduced with negligible loss in segmentation performance.

STDC

Rethinking BiSeNet For Real-time Semantic Segmentation

Self-supervised pre-training method to learn efficient representations without labels on histopathology medical images utilizing magnification factors.

Magnification Prior Contrastive Similarity

Magnification Prior: A Self-Supervised Method for Learning Representations on Breast Cancer Histopathological Images

**RepVGG** is a [VGG](https://paperswithcode.com/method/vgg)-style convolutional architecture. It has the following advantages:

- The model has a VGG-like plain (a.k.a. feed-forward) topology 1 without any branches. I.e., every layer takes
the output of its only preceding layer as input and feeds the output into its only following layer.
- The model’s body uses only 3 × 3 conv and [ReLU](https://paperswithcode.com/method/relu).
- The concrete architecture (including the specific depth and layer widths) is instantiated with no automatic
search, manual refinement, compound scaling, nor other heavy designs.

RepVGG

RepVGG: Making VGG-style ConvNets Great Again

reSGLD proposes to simulate a high-temperature particle for exploration and a low-temperature particle for exploitation and allows them to swap simultaneously. Moreover, a correction term is included to avoid biases.

reSGLD

Non-convex Learning via Replica Exchange Stochastic Gradient MCMC

**BS-Net** is an architecture for COVID-19 severity prediction based on clinical data from different modalities. The architecture comprises 1) a shared multi-task feature extraction backbone, 2) a lung segmentation branch, 3) an original registration mechanism that acts as a ”multi-resolution feature alignment” block operating on the encoding backbone , and 4) a multi-regional classification part for the final six-valued score estimation. 

All these blocks act together in the final training thanks to a loss specifically crated for this task. This loss guarantees also performance robustness, comprising a differentiable version of the target discrete metric. The learning phase operates in a weakly-supervised fashion. This is due to the fact that difficulties and pitfalls in the visual interpretation of the disease signs on CXRs (spanning from subtle findings to heavy lung impairment), and the lack of detailed localization information, produces unavoidable inter-rater variability among radiologists in assigning scores.

Specifically the architectural details are:

- The input image is processed with a convolutional backbone; the authors opt for a [ResNet](https://paperswithcode.com/method/resnet)-18.
- Segmentation is performed by a nested version of [U-Net](https://paperswithcode.com/method/u-net) (U-Net++).
- Alignment is estimated through the segmentation probability map produced by the U-Net++ decoder, which is achieved through a [spatial transformer network](https://paperswithcode.com/method/spatial-transformer) -- able to estimate the spatial transform matrix in order to center, rotate, and correctly zoom the lungs. After alignment at various scales, features are forward to a [ROIPool](https://paperswithcode.com/method/roi-pooling). 
- The alignment block is pre-trained on the synthetic alignment dataset in a weakly-supervised setting, using a Dice loss.
- The scoring head uses [FPNs](https://paperswithcode.com/method/fpn) for the combination of multi-scale feature maps. The multiresolution feature aligner produces input feature maps that are well focused on the specific area of interest. Eventually, the output of the FPN layer flows in a series of convolutional blocks to retrieve the output map. The classification is performed by a final [Global Average Pooling](https://paperswithcode.com/method/global-average-pooling) layer and a [SoftMax](https://paperswithcode.com/method/softmax) activation.
- The Loss function used for training is a sparse categorical cross entropy (SCCE) with a (differentiable) mean absolute error contribution.

BS-Net

BS-Net: learning COVID-19 pneumonia severity on a large Chest X-Ray dataset

**GPT-NeoX** is an autoregressive transformer decoder model whose architecture largely follows that of GPT-3, with a few notable deviations. The model has 20 billion parameters with 44 layers, a hidden dimension size of 6144, and 64 heads. The main difference with GPT-3 is the change in tokenizer, the addition of Rotary Positional Embeddings, the parallel computation of attention and feed-forward layers, and a different initialization scheme and hyperparameters.

GPT-NeoX

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

A **Neural Probablistic Language Model** is an early language modelling architecture. It involves a feedforward architecture that takes in input vector representations (i.e. word embeddings) of the previous $n$ words, which are looked up in a table $C$.

The word embeddings are concatenated and fed into a hidden layer which then feeds into a [softmax](https://paperswithcode.com/method/softmax) layer to estimate the probability of the word given the context.

Neural Probabilistic Language Model

SOHO (“See Out of tHe bOx”) that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. Text embeddings are used to extract textual embedding features. A trainable CNN is used to extract visual representations. SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in the proposed pre-training task Masked Visual Modeling (MVM).

SOHO

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Feature selection, also known as variable selection, attribute selection or variable subset selection, is the process of selecting a subset of relevant features (variables, predictors) for use in model construction.

Feature Selection

Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review

**LiteSeg** is a lightweight architecture for semantic segmentation that uses a deeper version of Atrous [Spatial Pyramid Pooling](https://paperswithcode.com/method/spatial-pyramid-pooling) module ([ASPP](https://paperswithcode.com/method/aspp)) and applies short and long residual connections, and [depthwise separable convolution](https://paperswithcode.com/method/depthwise-separable-convolution), resulting in a faster, more efficient model.

LiteSeg

LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation

**MoBY** is a self-supervised learning approach for [Vision Transformers](methods/category/vision-transformer). The approach is basically a combination of [MoCo v2](https://paperswithcode.com/method/moco-v2) and [BYOL](https://paperswithcode.com/method/byol). It inherits the momentum design, the key queue, and the contrastive loss used in MoCo v2, and inherits the asymmetric encoders, asymmetric data augmentations and the momentum scheduler in BYOL. It is named MoBY by picking the first two letters of each method.

The MoBY approach is illustrated in the Figure. There are two encoders: an online encoder and a target encoder. Both two encoders consist of a backbone and a projector head ([2-layer MLP](https://paperswithcode.com/method/feedforward-network)), and the online encoder introduces an additional prediction head (2-layer MLP), which makes the two encoders asymmetric. The online encoder is updated by gradients, and the target encoder is a moving average of the online encoder by momentum updating in each training iteration. A gradually increasing momentum updating strategy is applied for on the target encoder: the value of momentum term is gradually increased to 1 during the course of training. The default starting value is $0.99$.

A contrastive loss is applied to learn the representations. Specifically, for an online view $q$, its contrastive loss is computed as

$$
\mathcal{L}\_{q}=-\log \frac{\exp \left(q \cdot k\_{+} / \tau\right)}{\sum\_{i=0}^{K} \exp \left(q \cdot k\_{i} / \tau\right)}
$$

where $k\_{+}$is the target feature for the other view of the same image; $k\_{i}$ is a target feature in the key queue; $\tau$ is a temperature term; $K$ is the size of the key queue (4096 by default).

In training, like most [Transformer-based methods](https://paperswithcode.com/methods/category/transformers), the [AdamW](https://paperswithcode.com/method/adamw) optimizer is used, in contrast to previous [self-supervised learning approaches](https://paperswithcode.com/methods/category/self-supervised-learning) built on [ResNet](https://paperswithcode.com/method/resnet) backbone where usually [SGD](https://paperswithcode.com/method/sgd-with-momentum) or [LARS](https://paperswithcode.com/method/lars) $[4,8,19]$ is used. The authors also use a regularization method of asymmetric [drop path](https://paperswithcode.com/method/droppath) which proves important for the final performance.

In the experiments, the authors adopt a fixed learning rate of $0.001$ and a fixed weight decay of $0.05$, which performs stably well. Hyper-parameters are tuned of the key queue size $K$, the starting momentum value of the target branch, the temperature $\tau$, and the drop path rates.

MoBY

Self-Supervised Learning with Swin Transformers

**AdaMod** is a stochastic optimizer that restricts adaptive learning rates with adaptive and momental upper bounds. The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks.


The weight updates are performed as:


$$ g\_{t} = \nabla{f}\_{t}\left(\theta\_{t-1}\right) $$

$$ m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)g\_{t} $$

$$ v\_{t} = \beta\_{2}v\_{t-1} + \left(1-\beta\_{2}\right)g\_{t}^{2} $$

$$ \hat{m}\_{t} = m\_{t} / \left(1 - \beta^{t}\_{1}\right)$$

$$ \hat{v}\_{t} = v\_{t} / \left(1 - \beta^{t}\_{2}\right)$$

$$ \eta\_{t} = \alpha\_{t} / \left(\sqrt{\hat{v}\_{t}} + \epsilon\right) $$

$$ s\_{t} = \beta\_{3}s\_{t-1} + (1-\beta\_{3})\eta\_{t} $$

$$ \hat{\eta}\_{t} = \text{min}\left(\eta\_{t}, s\_{t}\right) $$

$$ \theta\_{t} = \theta\_{t-1} - \hat{\eta}\_{t}\hat{m}\_{t} $$

AdaMod

An Adaptive and Momental Bound Method for Stochastic Learning

**Cross-Covariance Attention**, or **XCA**, is an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) which operates along the feature dimension instead of the token dimension as in [conventional transformers](https://paperswithcode.com/methods/category/transformers).

Using the definitions of queries, keys and values from conventional attention, the cross-covariance attention function is defined as:

$$
\text { XC-Attention }(Q, K, V)=V \mathcal{A}_{\mathrm{XC}}(K, Q), \quad \mathcal{A}\_{\mathrm{XC}}(K, Q)=\operatorname{Softmax}\left(\hat{K}^{\top} \hat{Q} / \tau\right)
$$

where each output token embedding is a convex combination of the $d\_{v}$ features of its corresponding token embedding in $V$. The attention weights $\mathcal{A}$ are computed based on the cross-covariance matrix.

Cross-Covariance Attention

XCiT: Cross-Covariance Image Transformers

**VoiceFilter-Lite** is a single-channel source separation model that runs on the device to preserve only the speech signals from a target user, as part of a streaming speech recognition system. In this architecture, the voice filtering model operates as a frame-by-frame frontend signal processor to enhance the features consumed by the speech recognizer, without reconstructing audio signals from the features. The key contributions are (1) A system to perform speech separation directly on ASR input features; (2) An asymmetric loss function to penalize oversuppression during training, to make the model harmless under various acoustic environments, (3) An adaptive suppression strength mechanism to adapt to different noise conditions.

VoiceFilter-Lite

VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition

**GLOW** is a type of flow-based generative model that is based on an invertible $1 \times 1$ [convolution](https://paperswithcode.com/method/convolution). This builds on the flows introduced by [NICE](https://paperswithcode.com/method/nice) and [RealNVP](https://paperswithcode.com/method/realnvp). It consists of a series of steps of flow, combined in a multi-scale architecture; see the Figure to the right. Each step of flow consists of Act Normalization followed by an *invertible $1 \times 1$ convolution* followed by an [affine coupling](https://paperswithcode.com/method/affine-coupling) layer.

GLOW

Glow: Generative Flow with Invertible 1x1 Convolutions

In the OneR method, model input can be one of image, text or image+text, and CMC objective is combined with the traditional image-text contrastive (ITC) loss. Masked modeling is also carried out for all three input types (i.e., image, text and multi-modal). This framework employs no modality-specific architectural component except for the initial token embedding layer, making our model generic and modality-agnostic with minimal inductive bias.

OneR

Unifying Vision-Language Representation Space with Single-tower Transformer

Channel & spatial attention combines the advantages of channel attention and spatial attention. It adaptively selects both important objects and regions

Channel & Spatial attention

Residual Attention Network for Image Classification

**End-to-End Neural Diarization** is a neural network for speaker diarization in which a neural network directly outputs speaker diarization results given a multi-speaker recording. To realize such an end-to-end model, the speaker diarization problem is formulated as a multi-label classification problem and a permutation-free objective function is introduced to directly minimize diarization errors. The EEND method can explicitly handle speaker overlaps during training and inference. Just by feeding multi-speaker recordings with corresponding speaker segment labels, the model can be adapted to real conversations.

Viet-Anh on Software

AI Glossary

Tech Terminology

Short-Term Dense Concatenate