**Co-Correcting** is a noise-tolerant deep learning framework for medical image classification based on mutual learning and annotation correction. It consists of three modules: the dual-network architecture, the curriculum learning module, and the label correction module.

Co-Correcting

Co-Correcting: Noise-tolerant Medical Image Classification via mutual Label Correction

**Principle Components Analysis (PCA)** is an unsupervised method primary used for dimensionality reduction within machine learning.  PCA is calculated via a singular value decomposition (SVD) of the design matrix, or alternatively, by calculating the covariance matrix of the data and performing eigenvalue decomposition on the covariance matrix. The results of PCA provide a low-dimensional picture of the structure of the data and the leading (uncorrelated) latent factors determining variation in the data.

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis#/media/File:GaussianScatterPCA.svg)

**ESACL**, or **Enhanced Seq2Seq Autoencoder via Contrastive Learning**, is a denoising sequence-to-sequence (seq2seq) autoencoder via contrastive learning for abstractive text summarization. The model adopts a standard [Transformer](https://paperswithcode.com/method/transformer)-based architecture with a multilayer bi-directional encoder and an autoregressive decoder. To enhance its denoising ability, self-supervised contrastive learning is incorporated along with various sentence-level document augmentation.

ESACL

Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization

**Noisy Student Training** is a semi-supervised learning approach. It extends the idea of self-training
and distillation with the use of equal-or-larger student models and noise added to the student during learning. It has three main steps: 

1. train a teacher model on labeled images
2. use the teacher to generate pseudo labels on unlabeled images
3. train a student model on the combination of labeled images and pseudo labeled images. 

The algorithm is iterated a few times by treating the student as a teacher to relabel the unlabeled data and training a new student.

Noisy Student Training seeks to improve on self-training and distillation in two ways. First, it makes the student larger than, or at least equal to, the teacher so the student can better learn from a larger dataset. Second, it adds noise to the student so the noised student is forced to learn harder from the pseudo labels. To noise the student, it uses input noise such as [RandAugment](https://paperswithcode.com/method/randaugment) data augmentation, and model noise such as [dropout](https://paperswithcode.com/method/dropout) and [stochastic depth](https://paperswithcode.com/method/stochastic-depth) during training.

Noisy Student

Self-training with Noisy Student improves ImageNet classification

**Beta-VAE** is a type of variational autoencoder that seeks to discover disentangled latent factors. It modifies [VAEs](https://paperswithcode.com/method/vae) with an adjustable hyperparameter $\beta$ that balances latent channel capacity and independence constraints with reconstruction accuracy. The idea is to maximize the probability of generating the real data while keeping the distance between the real and estimated distributions small, under a threshold $\epsilon$. We can use the Kuhn-Tucker conditions to write this as a single equation:

$$ \mathcal{F}\left(\theta, \phi, \beta; \mathbf{x}, \mathbf{z}\right) = \mathbb{E}\_{q\_{\phi}\left(\mathbf{z}|\mathbf{x}\right)}\left[\log{p}\_{\theta}\left(\mathbf{x}\mid\mathbf{z}\right)\right] - \beta\left[D\_{KL}\left(\log{q}\_{\theta}\left(\mathbf{z}\mid\mathbf{x}\right)||p\left(\mathbf{z}\right)\right) - \epsilon\right]$$

where the KKT multiplier $\beta$ is the regularization coefficient that constrains the capacity of the latent channel $\mathbf{z}$ and puts implicit independence pressure on the learnt posterior due to the isotropic nature of the Gaussian prior $p\left(\mathbf{z}\right)$.

We write this again using the complementary slackness assumption to get the Beta-VAE formulation:

$$ \mathcal{F}\left(\theta, \phi, \beta; \mathbf{x}, \mathbf{z}\right) \geq  \mathcal{L}\left(\theta, \phi, \beta; \mathbf{x}, \mathbf{z}\right) = \mathbb{E}\_{q\_{\phi}\left(\mathbf{z}|\mathbf{x}\right)}\left[\log{p}\_{\theta}\left(\mathbf{x}\mid\mathbf{z}\right)\right] - \beta\{D}\_{KL}\left(\log{q}\_{\theta}\left(\mathbf{z}\mid\mathbf{x}\right)||p\left(\mathbf{z}\right)\right)$$

Beta-VAE

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

**ThunderNet** is a two-stage object detection model. The design of ThunderNet aims at the computationally expensive structures in state-of-the-art two-stage detectors. The backbone utilises a [ShuffleNetV2](https://paperswithcode.com/method/shufflenet-v2) inspired network called [SNet](https://paperswithcode.com/method/snet) designed for object detection. In the detection part, ThunderNet follows the detection head design in Light-Head [R-CNN](https://paperswithcode.com/method/r-cnn), and further compresses the [RPN](https://paperswithcode.com/method/rpn) and R-CNN subnet. To eliminate the performance degradation induced by small backbones and small feature maps, ThunderNet uses two new efficient architecture blocks, [Context Enhancement Module](https://paperswithcode.com/method/context-enhancement-module) (CEM) and [Spatial Attention Module](https://paperswithcode.com/method/spatial-attention-module) (SAM). CEM combines the feature maps from multiple scales to leverage local and global context information, while SAM uses the information learned in RPN to refine the feature distribution in RoI warping.

ThunderNet

ThunderNet: Towards Real-time Generic Object Detection

**Iterative Latent Variable Refinement**, or **ILVR**, is a method to guide the generative process in denoising diffusion probabilistic models (DDPMs) to generate high-quality images based on a given reference image. ILVR conditions the generation process in well-performing unconditional DDPM. Each transition in the generation process is refined utilizing a given reference image. By matching each latent variable, ILVR ensures the given condition in each transition thus enables sampling from a conditional distribution. Thus, ILVR generates high-quality images sharing desired semantics.

ILVR

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models

**Negative Face Recognition**, or **NFR**, is a face recognition approach that enhances the soft-biometric privacy on the template-level by representing face templates in a complementary (negative) domain. While ordinary templates characterize facial properties of an individual, negative templates describe facial properties that does not exist for this individual. This suppresses privacy-sensitive information from stored templates. Experiments are conducted on two publicly available datasets captured under controlled and uncontrolled scenarios on three privacy-sensitive attributes.

Unsupervised Enhancement of Soft-biometric Privacy with Negative Face Recognition

**Feature Fusion Module v2** is a feature fusion module from the [M2Det](https://paperswithcode.com/method/m2det) object detection model, and is crucial for constructing the final multi-level feature pyramid. They use [1x1 convolution](https://paperswithcode.com/method/1x1-convolution) layers to compress the channels of the input features and use a concatenation operation to aggregate these feature map. FFMv2 takes the base feature and the largest output feature map of the previous [Thinned U-Shape Module](https://paperswithcode.com/method/tum) (TUM) – these two are of the same scale – as input, and produces the fused feature for the next TUM.

FFMv2

M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network

A **Selective Kernel Convolution** is a [convolution](https://paperswithcode.com/method/convolution) that enables neurons to adaptively adjust their RF sizes among multiple kernels with different kernel sizes. Specifically, the SK convolution has three operators – Split, Fuse and Select. Multiple branches with different kernel sizes are fused using
[softmax](https://paperswithcode.com/method/softmax) attention that is guided by the information in these branches. Different attentions on these branches yield different sizes of the effective receptive fields of neurons in the fusion layer

Selective Kernel Convolution

Selective Kernel Networks

**CaiT**, or **Class-Attention in Image Transformers**, is a type of [vision transformer](https://paperswithcode.com/methods/category/vision-transformer) with several design alterations upon the original [ViT](https://paperswithcode.com/method/vision-transformer). First a new layer scaling approach called [LayerScale](https://paperswithcode.com/method/layerscale) is used, adding a learnable diagonal matrix on output of each residual block, initialized close to (but not at) 0, which improves the training dynamics. Secondly, [class-attention layers](https://paperswithcode.com/method/ca) are introduced to the architecture. This creates an architecture where the transformer layers involving [self-attention](https://paperswithcode.com/method/scaled) between patches are explicitly separated from class-attention layers -- that are devoted to extract the content of the processed patches into a single vector so that it can be fed to a linear classifier.

CaiT

Going deeper with Image Transformers

**PSPNet**, or **Pyramid Scene Parsing Network**, is a semantic segmentation model that utilises a pyramid parsing module that exploits global context information by different-region based context aggregation. The local and global clues together make the final prediction more reliable. We also propose an optimization

Given an input image, PSPNet use a pretrained CNN with the dilated network strategy to extract the feature map. The final feature map size is $1/8$ of the input image. On top of the map, we use the [pyramid pooling module](https://paperswithcode.com/method/pyramid-pooling-module) to gather context information. Using our 4-level pyramid, the pooling kernels cover the whole, half of, and small portions of the image. They are fused as the global prior.
Then we concatenate the prior with the original feature map in the final part of. It is followed by a [convolution](https://paperswithcode.com/method/convolution) layer to generate the final prediction map.

PSPNet

Pyramid Scene Parsing Network

**mBART** is a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the [BART objective](https://paperswithcode.com/method/bart). The input texts are noised by masking phrases and permuting sentences, and a single [Transformer model](https://paperswithcode.com/method/transformer) is learned to recover the texts. Different from other pre-training approaches for machine translation, mBART pre-trains a complete autoregressive [Seq2Seq](https://paperswithcode.com/method/seq2seq) model. mBART is trained once for all languages, providing a set of parameters that can be fine-tuned for any of the language pairs in both supervised and unsupervised settings, without any task-specific or language-specific modifications or initialization schemes.

mBART

Multilingual Denoising Pre-training for Neural Machine Translation

Lightweight or mobile neural networks used for real-time computer vision tasks contain fewer parameters than normal
networks, which lead to a constrained performance. In this work, we proposed a novel activation function named Tanh Exponential
Activation Function (TanhExp) which can improve the performance for these networks on image classification task significantly.
The definition of TanhExp is $f(x) = x tanh(e^x)$. We demonstrate the simplicity, efficiency, and robustness of TanhExp on various
datasets and network models and TanhExp outperforms its counterparts in both convergence speed and accuracy. Its behaviour
also remains stable even with noise added and dataset altered. We show that without increasing the size of the network, the
capacity of lightweight neural networks can be enhanced by TanhExp with only a few training epochs and no extra parameters
added.

TanhExp

TanhExp: A Smooth Activation Function with High Convergence Speed for Lightweight Neural Networks

**Auxiliary Batch Normalization** is a type of regularization used in adversarial training schemes. The idea is that adversarial examples should have a separate [batch normalization](https://paperswithcode.com/method/batch-normalization) components to the clean examples, as they have different underlying statistics.

Auxiliary Batch Normalization

Adversarial Examples Improve Image Recognition

Dynamic Time Warping (DTW) [1] is one of well-known distance measures between a pairwise of time series. The main idea of DTW is to compute the distance from the matching of similar elements between time series. It uses the dynamic programming technique to find the optimal temporal matching between elements of two time series.

For instance, similarities in walking could be detected using DTW, even if one person was walking faster than the other, or if there were accelerations and decelerations during the course of an observation. DTW has been applied to temporal sequences of video, audio, and graphics data — indeed, any data that can be turned into a linear sequence can be analyzed with DTW. A well known application has been automatic speech recognition, to cope with different speaking speeds. Other applications include speaker recognition and online signature recognition. It can also be used in partial shape matching application.

In general, DTW is a method that calculates an optimal match between two given sequences (e.g. time series) with certain restriction and rules:

1. Every index from the first sequence must be matched with one or more indices from the other sequence, and vice versa
2. The first index from the first sequence must be matched with the first index from the other sequence (but it does not have to be its only match)
3. The last index from the first sequence must be matched with the last index from the other sequence (but it does not have to be its only match)
4. The mapping of the indices from the first sequence to indices from the other sequence must be monotonically increasing, and vice versa, i.e. if j>i  are indices from the first sequence, then there must not be two indices l>k in the other sequence, such that index i is matched with index l and index j is matched with index k, and vice versa.

[1] Sakoe, Hiroaki, and Seibi Chiba. "Dynamic programming algorithm optimization for spoken word recognition." IEEE transactions on acoustics, speech, and signal processing 26, no. 1 (1978): 43-49.

**Fast Sample Re-Weighting**, or **FSR**, is a sample re-weighting strategy to tackle problems such as dataset biases, noisy labels and imbalanced classes. It leverages a dictionary (essentially an extra buffer) to monitor the training history reflected by the model updates during meta optimization periodically, and utilises a valuation function to discover meaningful samples from training data as the proxy of reward data. The unbiased dictionary keeps being updated and provides reward signals to optimize sample weights. Additionally, instead of maintaining model states for both model and sample weight updates separately, feature sharing is enabled for saving the computation cost used for maintaining respective states.

Fast Sample Re-Weighting

Learning Fast Sample Re-weighting Without Reward Data

**StyleGAN2** is a generative adversarial network that builds on [StyleGAN](https://paperswithcode.com/method/stylegan) with several improvements. First, [adaptive instance normalization](https://paperswithcode.com/method/adaptive-instance-normalization) is redesigned and replaced with a normalization technique called [weight demodulation](https://paperswithcode.com/method/weight-demodulation). Secondly, an improved training scheme upon progressively growing is introduced, which achieves the same goal - training starts by focusing on low-resolution images and then progressively shifts focus to higher and higher resolutions - without changing the network topology during training. Additionally, new types of regularization like lazy regularization and [path length regularization](https://paperswithcode.com/method/path-length-regularization) are proposed.

Viet-Anh on Software

AI Glossary

Tech Terminology

Co-Correcting

Principal Components Analysis

Enhanced Seq2Seq Autoencoder via Contrastive Learning

Noisy Student

Beta-VAE

ThunderNet

Iterative Latent Variable Refinement

Negative Face Recognition

Feature Fusion Module v2

Selective Kernel Convolution

Class-Attention in Image Transformers

PSPNet

mBART

Tanh Exponential Activation Function

Auxiliary Batch Normalization

Dynamic Time Warping

Fast Sample Re-Weighting

StyleGAN2

Viet-Anh on Software