**NAS-FPN** is a Feature Pyramid Network that is discovered via [Neural Architecture Search](https://paperswithcode.com/method/neural-architecture-search) in a novel scalable search space covering all cross-scale connections. The discovered architecture consists of a combination of top-down and bottom-up connections to fuse features across scales

NAS-FPN

NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection

**FastMoE ** is a distributed MoE training system based on PyTorch with common accelerators. The system provides a hierarchical interface for both flexible model design and adaption to different applications, such as [Transformer-XL](https://paperswithcode.com/method/transformer-xl) and Megatron-LM.

FastMoE

FastMoE: A Fast Mixture-of-Expert Training System

**TD3** builds on the [DDPG](https://paperswithcode.com/method/ddpg) algorithm for reinforcement learning, with a couple of modifications aimed at tackling overestimation bias with the value function. In particular, it utilises [clipped double Q-learning](https://paperswithcode.com/method/clipped-double-q-learning), delayed update of target and policy networks, and [target policy smoothing](https://paperswithcode.com/method/target-policy-smoothing) (which is similar to a [SARSA](https://paperswithcode.com/method/sarsa) based update; a safer update, as they provide higher value to actions resistant to perturbations).

Addressing Function Approximation Error in Actor-Critic Methods

**Neural Tangent Transfer**, or **NTT**, is a method for finding trainable sparse networks in a label-free manner. Specifically, NTT finds sparse networks whose training dynamics, as characterized by the neural tangent kernel, mimic those of dense networks in function space.

Neural Tangent Transfer

Finding trainable sparse networks through Neural Tangent Transfer

**Prescribed GANs** add noise to the output of a density network and optimize an entropy-regularized adversarial loss. The added noise renders tractable approximations of the predictive log-likelihood and stabilizes the training procedure. The entropy regularizer encourages PresGANs to capture all the modes of the data distribution. Fitting PresGANs involves computing the intractable gradients of the [entropy regularization](https://paperswithcode.com/method/entropy-regularization) term; PresGANs sidestep this intractability using
unbiased stochastic estimates.

PresGAN

Prescribed Generative Adversarial Networks

Spectral clustering has attracted increasing attention due to
the promising ability in dealing with nonlinearly separable datasets [15], [16]. In spectral clustering, the spectrum of the graph Laplacian is used to reveal the cluster structure. The spectral clustering algorithm mainly consists of two steps: 1) constructs the low dimensional embedded representation of the data based on the eigenvectors of the graph Laplacian, 2) applies k-means on the constructed low dimensional data to obtain the clustering result. Thus,

Spectral Clustering

A Tutorial on Spectral Clustering

**OMGD**, or **Online Multi-Granularity Distillation** is a framework for learning efficient [GANs](https://paperswithcode.com/methods/category/generative-adversarial-networks). The student generator is optimized in a discriminator-free and ground-truth-free setting. The scheme trains the teacher and student alternatively, promoting these two generators iteratively and progressively. The progressively optimized teacher generator helps to warm up the student and guide the optimization direction step by step.

Specifically, the student generator $G\_{S}$ only leverages the complementary teacher generators $G^{W}\_{T}$ and $G^{D}\_{T}$ for optimization and can be trained in the discriminator-free and ground-truth-free setting. This framework transfers different levels concepts from the intermediate layers and output layer to perform the knowledge distillation. The whole optimization is conducted on an online distillation scheme. Namely,  $G^{W}\_{T}$, $G^{D}\_{T}$ and $G\_{S}$ are optimized simultaneously and progressively.

OMGD

Online Multi-Granularity Distillation for GAN Compression

The Auditory Cortex ResNet, briefly AUCO ResNet, is proposed and tested. It is a deep neural network architecture especially designed for audio classification trained end-to-end. It is inspired by the architectural organization of rat's auditory cortex, containing also innovations 2 and 3. The network outperforms the state-of-the-art accuracies on a reference audio benchmark dataset without any kind of preprocessing, imbalanced data handling and, most importantly, any kind of data augmentation.

AUCO ResNet

AUCO ResNet: an end-to-end network for Covid-19 pre-screening from cough and breath

Gradual self-training is a method for semi-supervised domain adaptation. The goal is to adapt an initial classifier trained on a source domain given only unlabeled data that shifts gradually in distribution towards a target domain. 

This comes up for example in applications ranging from sensor networks and self-driving car perception modules to brain-machine interfaces, where machine learning systems must adapt to data distributions that evolve over time.

The gradual self-training algorithm begins with a classifier $w_0$ trained on labeled examples from the source domain (Figure a). For each successive domain $P_t$, the algorithm generates pseudolabels for unlabeled examples from that domain, and then trains a regularized supervised classifier on the pseudolabeled examples. The intuition, visualized in the Figure, is that after a single gradual shift, most examples are pseudolabeled correctly so self-training learns a good classifier on the shifted data, but the shift from the source to the target can be too large for self-training to correct.

Gradual Self-Training

Understanding Self-Training for Gradual Domain Adaptation

**Invertible Image Conversion Net**, or **IICNet**, is a generic framework for reversible image conversion tasks. Unlike previous encoder-decoder based methods, IICNet maintains a highly invertible structure based on invertible neural networks (INNs) to better preserve the information during conversion. It uses a relation module and a channel squeeze layer to improve the INN nonlinearity to extract cross-image relations and the network flexibility, respectively.

IICNet

IICNet: A Generic Framework for Reversible Image Conversion

An **RGCN**, or **Relational Graph Convolution Network**, is a an application of the [GCN framework](https://paperswithcode.com/method/gcn) to modeling relational data, specifically
to link prediction and entity classification tasks.

See [here](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/4_rgcn.html) for an in-depth explanation of RGCNs by DGL.

RGCN

Modeling Relational Data with Graph Convolutional Networks

MinCutPool is a trainable pooling operator for graphs that learns to map nodes into clusters.
The method is trained to approximate the minimum K-cut of the graph to ensure that the clusters are balanced, while also jointly optimizing the objective of the task at hand.

MinCutPool

Spectral Clustering with Graph Neural Networks for Graph Pooling

**Balanced L1 Loss** is a loss function used for the object detection task. Classification and localization problems are solved simultaneously under the guidance of a multi-task loss since
[Fast R-CNN](https://paperswithcode.com/method/fast-r-cnn), defined as:

$$ L\_{p,u,t\_{u},v} = L\_{cls}\left(p, u\right) + \lambda\left[u \geq 1\right]L\_{loc}\left(t^{u}, v\right) $$

$L\_{cls}$ and $L\_{loc}$ are objective functions corresponding to recognition and localization respectively. Predictions and targets in $L\_{cls}$ are denoted as $p$ and $u$. $t\_{u}$ is the corresponding regression results with class $u$. $v$ is the regression target. $\lambda$ is used for tuning the loss weight under multi-task learning. We call samples with a loss greater than or equal to 1.0 outliers. The other samples are called inliers.

A natural solution for balancing the involved tasks is to tune the loss weights of them. However, owing to the unbounded regression targets, directly raising the weight of localization loss will make the model more sensitive to outliers. These outliers, which can be regarded as hard samples, will produce excessively large gradients that are harmful to the training process. The inliers, which can be regarded as the easy samples, contribute little gradient to the overall gradients compared with the outliers. To be more specific, inliers only contribute 30% gradients average per sample compared with outliers. Considering these issues, the authors introduced the balanced L1 loss, which is denoted as $L\_{b}$.

Balanced L1 loss is derived from the conventional smooth L1 loss, in which an inflection point is set to separate inliers from outliners, and clip the large gradients produced by outliers with a maximum value of 1.0, as shown by the dashed lines in the Figure to the right. The key idea of balanced L1 loss is promoting the crucial regression gradients, i.e. gradients from inliers (accurate samples), to rebalance
the involved samples and tasks, thus achieving a more balanced training within classification, overall localization and accurate localization. Localization loss $L\_{loc}$ uses balanced L1 loss is defined as:

$$ L\_{loc} = \sum\_{i\in{x,y,w,h}}L\_{b}\left(t^{u}\_{i}-v\_{i}\right) $$

The Figure to the right shows that the balanced L1 loss increases the gradients of inliers under the control of a factor denoted as $\alpha$. A small $\alpha$ increases more gradient for inliers, but the gradients of outliers are not influenced. Besides, an overall promotion magnification controlled by γ is also brought in for tuning the upper bound of regression errors, which can help the objective function better balancing involved tasks. The two factors that control different aspects are mutually enhanced to reach a more balanced training.$b$ is used to ensure $L\_{b}\left(x = 1\right)$ has the same value for both formulations in the equation below.

By integrating the gradient formulation above, we can get the balanced L1 loss as:

$$ L\_{b}\left(x\right) = \frac{\alpha}{b}\left(b|x| + 1\right)ln\left(b|x| + 1\right) - \alpha|x| \text{ if } |x| < 1$$

$$ L\_{b}\left(x\right) = \gamma|x| + C \text{ otherwise } $$

in which the parameters $\gamma$, $\alpha$, and $b$ are constrained by $\alpha\text{ln}\left(b + 1\right) = \gamma$. The default parameters are set as $\alpha = 0.5$ and $\gamma = 1.5$

Balanced L1 Loss

Libra R-CNN: Towards Balanced Learning for Object Detection

**GoogLeNet** is a type of convolutional neural network based on the [Inception](https://paperswithcode.com/method/inception-module) architecture. It utilises Inception modules, which allow the network to choose between multiple convolutional filter sizes in each block. An Inception network stacks these modules on top of each other, with occasional max-pooling layers with stride 2 to halve the resolution of the grid.

GoogLeNet

Going Deeper with Convolutions

**Activation Normalization** is a type of normalization used for flow-based generative models; specifically it was introduced in the [GLOW](https://paperswithcode.com/method/glow) architecture. An ActNorm layer performs an affine transformation of the activations using a scale and bias parameter per channel, similar to [batch normalization](https://paperswithcode.com/method/batch-normalization). These parameters are initialized such that the post-actnorm activations per-channel have zero mean and unit variance given an initial minibatch of data. This is a form of data dependent initilization. After initialization, the scale and bias are treated as regular trainable parameters that are independent of the data.

Activation Normalization

Glow: Generative Flow with Invertible 1x1 Convolutions

**T5**, or **Text-to-Text Transfer Transformer**, is a [Transformer](https://paperswithcode.com/method/transformer) based architecture that uses a text-to-text approach. Every task – including translation, question answering, and classification – is cast as feeding the model text as input and training it to generate some target text. This allows for the use of the same model, loss function, hyperparameters, etc. across our diverse set of tasks. The changes compared to [BERT](https://paperswithcode.com/method/bert) include:

- adding a *causal* decoder to the bidirectional architecture.
- replacing the fill-in-the-blank cloze task with a mix of alternative pre-training tasks.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

ALP-GMM is is an algorithm that learns to generate a learning curriculum for black box reinforcement learning agents, whereby it sequentially samples parameters controlling a stochastic procedural generation of tasks or environments.

ALP-GMM

Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments

**PeleeNet** is a convolutional neural network  and object detection backbone that is a variation of [DenseNet](https://paperswithcode.com/method/densenet) with optimizations to meet a memory and computational budget. Unlike competing networks, it does not use depthwise convolutions and instead relies on regular convolutions.

Viet-Anh on Software

AI Glossary

Tech Terminology

NAS-FPN

FastMoE

Twin Delayed Deep Deterministic

Neural Tangent Transfer

Prescribed Generative Adversarial Network

Spectral Clustering

Online Multi-granularity Distillation

Auditory Cortex ResNet

Gradual Self-Training

IICNet

Relational Graph Convolution Network

MinCut Pooling

Balanced L1 Loss

GoogLeNet

Activation Normalization

T5

Absolute Learning Progress and Gaussian Mixture Models for Automatic Curriculum Learning

PeleeNet

Viet-Anh on Software