**Center Pooling** is a pooling technique for object detection that aims to capture richer and more recognizable visual patterns. The geometric centers of objects do not necessarily convey very recognizable visual patterns (e.g., the human head contains strong visual patterns, but the center keypoint is often in the middle of the human body). 

The detailed process of center pooling is as follows: the backbone outputs a feature map, and to determine if a pixel in the feature map is a center keypoint, we need to find the maximum value in its both horizontal and vertical directions and add them together. By doing this, center pooling helps the better detection of center keypoints.

Center Pooling

CenterNet: Keypoint Triplets for Object Detection

**NADAM**, or **Nesterov-accelerated Adaptive Moment Estimation**, combines [Adam](https://paperswithcode.com/method/adam) and [Nesterov Momentum](https://paperswithcode.com/method/nesterov-accelerated-gradient). The update rule is of the form:

$$ \theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{\hat{v}\_{t}}+\epsilon}\left(\beta\_{1}\hat{m}\_{t} + \frac{(1-\beta\_{t})g\_{t}}{1-\beta^{t}\_{1}}\right)$$

Image Source: [Incorporating Nesterov Momentum into Adam](http://cs229.stanford.edu/proj2015/054_report.pdf)

NADAM

**Hydra** is a multi-headed neural network for model distillation with a shared body network. The shared body network learns a joint feature representation that enables each head to capture the predictive behavior of each ensemble member.  Existing distillation methods often train a distillation network to imitate the prediction of a larger network. Hydra instead learns to distill the individual predictions of each ensemble member into separate light-weight head models while amortizing the computation through a shared heavy-weight body network. This retains the diversity of ensemble member predictions which is otherwise lost in knowledge distillation.

Hydra

Hydra: Preserving Ensemble Diversity for Model Distillation

**Variational Trace Distance Estimation**, or **VTDE**, is a variational algorithm for trace norm estimation that only involves one ancillary qubit. Notably, the cost function in VTDE gathers information from a single-qubit observable and thus could avoid the barren plateau issue with logarithmic depth parameterized circuits.

VTDE

Variational Quantum Algorithms for Trace Distance and Fidelity Estimation

**XLSR** is a multilingual speech recognition model built on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining. A shared quantization module over feature encoder representations produces multilingual quantized speech units whose embeddings are then used as targets for a [Transformer](https://paperswithcode.com/method/transformer) trained by contrastive learning. The model learns to share discrete tokens across languages, creating bridges across languages.

XLSR

Unsupervised Cross-lingual Representation Learning for Speech Recognition

**Universal Language Model Fine-tuning**, or **ULMFiT**, is an architecture and transfer learning method that can be applied to NLP tasks. It involves a 3-layer [AWD-LSTM](https://paperswithcode.com/method/awd-lstm) architecture for its representations. The training consists of three steps: 1) general language model pre-training on a Wikipedia-based text, 2) fine-tuning the language model on a target task, and 3) fine-tuning the classifier on the target task.

As different layers capture different types of information, they are fine-tuned to different extents using [discriminative fine-tuning](https://paperswithcode.com/method/discriminative-fine-tuning). Training is performed using [Slanted triangular learning rates](https://paperswithcode.com/method/slanted-triangular-learning-rates) (STLR), a learning rate scheduling strategy that first linearly increases the learning rate and then linearly decays it.

Fine-tuning the target classifier is achieved in ULMFiT using gradual unfreezing. Rather than fine-tuning all layers at once, which risks catastrophic forgetting, ULMFiT gradually unfreezes the model starting from the last layer (i.e., closest to the output) as this contains the least general knowledge. First the last layer is unfrozen and all unfrozen layers are fine-tuned for one epoch. Then the next group of frozen layers is unfrozen and fine-tuned and repeat, until all layers are fine-tuned until convergence at the last iteration.

ULMFiT

Universal Language Model Fine-tuning for Text Classification

**Experience Replay** is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t}, s\_{t+1}\right)$ in a data-set $D = e\_{1}, \cdots, e\_{N}$ , pooled over many episodes into a replay memory. We then usually sample the memory randomly for a minibatch of experience, and use this to learn off-policy, as with Deep Q-Networks. This tackles the problem of autocorrelation leading to unstable training, by making the problem more like a supervised learning problem.

Image Credit: [Hands-On Reinforcement Learning with Python, Sudharsan Ravichandiran](https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781788836524)

Experience Replay

**DeeBERT** is a method for accelerating [BERT](https://paperswithcode.com/method/bert) inference. It inserts extra classification layers (which are referred to as off-ramps) between each [transformer](https://paperswithcode.com/method/transformer) layer of BERT. All transformer layers and off-ramps are jointly fine-tuned on a given downstream dataset. At inference time, after a sample goes through a transformer layer, it is passed to the following off-ramp. If the off-ramp is confident of the prediction, the result is returned; otherwise, the sample is sent to the next transformer layer.

DeeBERT

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

**GBlock** is a type of [residual block](https://paperswithcode.com/method/residual-block) used in the [GAN-TTS](https://paperswithcode.com/method/gan-tts) text-to-speech architecture - it is a stack of two residual blocks. As the generator is producing raw audio (e.g. a 2s training clip corresponds
to a sequence of 48000 samples), dilated convolutions are used to ensure that the receptive field of $G$ is large enough to capture long-term dependencies. The four kernel size-3 convolutions in each GBlock have increasing dilation factors: 1, 2, 4, 8. Convolutions are preceded by Conditional Batch Normalisation, conditioned on the linear embeddings of the noise term $z \sim N\left(0, \mathbf{I}\_{128}\right)$ in the single-speaker case, or the concatenation of $z$ and a one-hot representation of the speaker ID in the multi-speaker case. The embeddings are different for
each BatchNorm instance. 

A GBlock contains two skip connections, the first of which in [GAN](https://paperswithcode.com/method/gan)-TTS performs upsampling if the output frequency is higher than the input, and it also contains a size-1 [convolution](https://paperswithcode.com/method/convolution)
if the number of output channels is different from the input.

GBlock

High Fidelity Speech Synthesis with Adversarial Networks

F2DNet, a novel two-stage object detection architecture which eliminates redundancy of classical two-stage detectors by replacing the region proposal network with focal detection network and
bounding box head with fast suppression head.

F2DNet

F2DNet: Fast Focal Detection Network for Pedestrian Detection

**DenseNAS-A** is a mobile convolutional neural network discovered through the [DenseNAS](https://paperswithcode.com/method/densenas) [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) method. The basic building block is MBConvs, or inverted bottleneck residuals, from the MobileNet architectures.

DenseNAS-A

Densely Connected Search Space for More Flexible Neural Architecture Search

An **LR-Net** is a type of non-convolutional neural network that utilises local relation layers instead of convolutions for image feature extraction. Otherwise, the architecture follows the same design as a [ResNet](https://paperswithcode.com/method/resnet).

LR-Net

Although significant effort has been applied to fact-checking, the prevalence of fake news over social media, which has profound impact on justice, public trust and our society, remains a serious problem. In this work, we focus on propagation-based fake news detection, as recent studies have demonstrated that fake news and real news spread differently online. Specifically, considering the capability of graph neural networks (GNNs) in dealing with non-Euclidean data, we use GNNs to differentiate between the propagation patterns of fake and real news on social media. In particular, we concentrate on two questions: (1) Without relying on any text information, e.g., tweet content, replies and user descriptions, how accurately can GNNs identify fake news? Machine learning models are known to be vulnerable to adversarial attacks, and avoiding the dependence on text-based features can make the model less susceptible to the manipulation of advanced fake news fabricators. (2) How to deal with new, unseen data? In other words, how does a GNN trained on a given dataset perform on a new and potentially vastly different dataset? If it achieves unsatisfactory performance, how do we solve the problem without re-training the model on the entire data from scratch? We study the above questions on two datasets with thousands of labelled news items, and our results show that: (1) GNNs can achieve comparable or superior performance without any text information to state-of-the-art methods. (2) GNNs trained on a given dataset may perform poorly on new, unseen data, and direct incremental training cannot solve the problem---this issue has not been addressed in the previous work that applies GNNs for fake news detection. In order to solve the problem, we propose a method that achieves balanced performance on both existing and new datasets, by using techniques from continual learning to train GNNs incrementally.

GNNCL

Graph Neural Networks with Continual Learning for Fake News Detection from Social Media

TLC convert the global operation to a local one so that it extract representations based on local spatial region of features as in training phase.

Improving Image Restoration by Revisiting Global Information Aggregation

**Weight Modulation** is an alternative to [adaptive instance normalization](https://paperswithcode.com/method/adaptive-instance-normalization) for use in generative adversarial networks, specifically it is introduced in [StyleGAN2](https://paperswithcode.com/method/stylegan2). The purpose of [instance normalization](https://paperswithcode.com/method/instance-normalization) is to remove the effect of $s$ - the scales of the features maps - from the statistics of the [convolution](https://paperswithcode.com/method/convolution)’s output feature maps. Weight modulation tries to achieve this goal more directly. Assuming that input activations are i.i.d. random variables with unit standard deviation. After modulation and convolution, the output activations have standard deviation of:

$$ \sigma\_{j} = \sqrt{{\sum\_{i,k}w\_{ijk}'}^{2}} $$

i.e., the outputs are scaled by the $L\_{2}$ norm of the corresponding weights. The subsequent normalization aims to restore the outputs back to unit standard deviation. This can be achieved if we scale (“demodulate”) each output feature map $j$ by $1/\sigma\_{j}$ . Alternatively, we can again bake this into the convolution weights:

$$ w''\_{ijk} = w'\_{ijk} / \sqrt{{\sum\_{i, k}w'\_{ijk}}^{2} + \epsilon} $$

where $\epsilon$ is a small constant to avoid numerical issues.

Weight Demodulation

Analyzing and Improving the Image Quality of StyleGAN

**Deep-MAC**, or **Deep Mask-heads Above CenterNet**, is a type of anchor-free instance segmentation model based on [CenterNet](https://paperswithcode.com/method/centernet).  The motivation for this new architecture is that boxes are much cheaper to annotate than masks, so the authors address the “partially supervised” instance segmentation problem, where all classes have bounding box annotations but only a subset of classes have mask annotations. 

For predicting bounding boxes, CenterNet outputs 3 tensors: (1) a class-specific [heatmap](https://paperswithcode.com/method/heatmap) which indicates the probability of the center of a bounding box being present at each location, (2) a class-agnostic 2-channel tensor indicating the height and width of the bounding box at each center pixel, and (3) since the output feature map is typically smaller than the image (stride 4 or 8), CenterNet also predicts an x and y direction offset to recover this discretization error at each center pixel.

For Deep-MAC, in parallel to the box-related prediction heads, we add a fourth pixel embedding branch $P$. For each bounding box
$b$, we crop a region $P\_{b}$ from $P$ corresponding to $b$ via [ROIAlign](https://paperswithcode.com/method/roi-align) which results in a 32 × 32 tensor. We then feed each $P\_{b}$ to a mask-head. The final prediction at the end is a class-agnostic 32 × 32 tensor which we pass through a sigmoid to get per-pixel probabilities. We train this mask-head via a per-pixel cross-entropy loss averaged over all pixels and instances. During post-processing, the predicted mask is re-aligned according to the predicted box and resized to the resolution of the image. 

In addition to this 32 × 32 cropped feature map, we add two inputs for improved stability of some mask-heads: (1) Instance embedding: an additional head is added to the backbone that predicts a per-pixel embedding. For each bounding box $b$ we extract its embedding from the center pixel. This embedding is tiled to a size of 32 × 32 and concatenated to the pixel embedding crop. This helps condition the mask-head on a particular instance and disambiguate it from others. (2) Coordinate Embedding: Inspired by [CoordConv](https://paperswithcode.com/method/coordconv), the authors add a 32 × 32 × 2 tensor holding normalized $\left(x, y\right)$ coordinates relative to the bounding box $b$.

Deep-MAC

The surprising impact of mask-head architecture on novel class segmentation

**RegNetX** is a convolutional network design space with simple, regular models with parameters: depth $d$, initial width $w\_{0} > 0$, and slope $w\_{a} > 0$, and generates a different block width $u\_{j}$ for each block $j < d$. The key restriction for the RegNet types of model is that there is a linear parameterisation of block widths (the design space only contains models with this linear structure):

$$ u\_{j} = w\_{0} + w\_{a}\cdot{j} $$

For **RegNetX** we have additional restrictions: we set $b = 1$ (the bottleneck ratio), $12 \leq d \leq 28$, and $w\_{m} \geq 2$ (the width multiplier).

RegNetX

Designing Network Design Spaces

**Clipped Double Q-learning** is a variant on [Double Q-learning](https://paperswithcode.com/method/double-q-learning) that upper-bounds the less biased Q estimate $Q\_{\theta\_{2}}$ by the biased estimate $Q\_{\theta\_{1}}$. This is equivalent to taking the minimum of the two estimates, resulting in the following target update:

$$ y\_{1} = r + \gamma\min\_{i=1,2}Q\_{\theta'\_{i}}\left(s', \pi\_{\phi\_{1}}\left(s'\right)\right) $$

The motivation for this extension is that vanilla double [Q-learning](https://paperswithcode.com/method/q-learning) is sometimes ineffective if the target and current networks are too similar, e.g. with a slow-changing policy in an actor-critic framework.

Viet-Anh on Software

AI Glossary

Tech Terminology

Center Pooling

NADAM

Hydra

Variational Trace Distance Estimation

XLSR

Universal Language Model Fine-tuning

Experience Replay

DeeBERT

GBlock

Fast Focal Detection Network

DenseNAS-A

LR-Net

Graph Neural Networks with Continual Learning

Test-time Local Converter

Weight Demodulation

Deep-MAC

RegNetX

Clipped Double Q-learning

Viet-Anh on Software