Documents often exhibit various forms of degradation, which make it hard to be read and substantially deteriorate the
performance of an OCR system. In this paper, we propose an effective end-to-end framework named Document Enhancement
Generative Adversarial Networks (DE-GAN) that uses the conditional GANs (cGANs) to restore severely degraded document images.
To the best of our knowledge, this practice has not been studied within the context of generative adversarial deep networks. We
demonstrate that, in different tasks (document clean up, binarization, deblurring and watermark removal), DE-GAN can produce an
enhanced version of the degraded document with a high quality. In addition, our approach provides consistent improvements compared to state-of-the-art methods over the widely used DIBCO 2013, DIBCO 2017 and H-DIBCO 2018 datasets, proving its ability to restore a degraded document image to its ideal condition. The obtained results on a wide variety of degradation reveal the flexibility of the proposed model to be exploited in other document enhancement problems.

DE-GAN

**WaveVAE** is a generative audio model that can be used as a vocoder in text-to-speech systems. It is a [VAE](https://paperswithcode.com/method/vae) based model that can be trained from scratch by jointly optimizing the encoder $q\_{\phi}\left(\mathbf{z}|\mathbf{x}, \mathbf{c}\right)$ and decoder $p\_{\theta}\left(\mathbf{x}|\mathbf{z}, \mathbf{c}\right)$, where $\mathbf{z}$ is latent variables and $\mathbf{c}$ is the mel spectrogram conditioner. 

The encoder of WaveVAE $q\_{\phi}\left(\mathbf{z}|\mathbf{x}\right)$ is parameterized by a Gaussian autoregressive [WaveNet](https://paperswithcode.com/method/wavenet) that maps the ground truth audio x into the same length latent representation $\mathbf{z}$. The decoder $p\_{\theta}\left(\mathbf{x}|\mathbf{z}\right)$ is parameterized by the one-step ahead predictions from an inverse autoregressive flow.

The training objective is the ELBO for the observed $\mathbf{x}$ in the VAE.

WaveVAE

Non-Autoregressive Neural Text-to-Speech

**Fawkes** is an image cloaking system that helps individuals inoculate their images against unauthorized facial recognition models. Fawkes achieves this by helping users add imperceptible pixel-level changes ("cloaks") to their own photos before releasing them. When used to train facial recognition models, these "cloaked" images produce functional models that consistently cause normal images of the user to be misidentified.

Fawkes

Fawkes: Protecting Privacy against Unauthorized Deep Learning Models

**VQ-VAE** is a type of variational autoencoder that uses vector quantisation to obtain a discrete latent representation. It differs from [VAEs](https://paperswithcode.com/method/vae) in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, ideas from vector quantisation (VQ) are incorporated. Using the VQ method allows the model to circumvent issues of posterior collapse - where the latents are ignored when they are paired with a powerful autoregressive decoder - typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes.

VQ-VAE

Neural Discrete Representation Learning

UNITER or UNiversal Image-TExt Representation model is a large-scale pre-trained model for joint multimodal embedding. It is pre-trained using four image-text datasets COCO, Visual Genome, Conceptual Captions, and SBU Captions. It can power heterogeneous downstream V+L tasks with joint multimodal embeddings. 
UNITER takes the visual regions of the image and textual tokens of the sentence as inputs. A faster R-CNN is used in Image Embedder to extract the visual features of each region and a Text Embedder is used to tokenize the input sentence into WordPieces.  

It proposes WRA via the Optimal Transport to provide more fine-grained alignment between word tokens and image regions that is effective in calculating the minimum cost of transporting the contextualized image embeddings to word embeddings and vice versa. 

Four pretraining tasks were designed for this model. They are Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). This model is different from the previous models because it uses conditional masking on pre-training tasks.

UNITER

UNITER: UNiversal Image-TExt Representation Learning

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Source: [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)

Knowledge Distillation

Distilling the Knowledge in a Neural Network

**UCNet** is a probabilistic framework for RGB-D Saliency Detection that employs uncertainty by learning from the data labelling process. It utilizes conditional variational autoencoders to model human annotation uncertainty and generate multiple saliency maps for each input image by sampling in the latent space.

UCNet

UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders

**OPT-IML** is a version of OPT fine-tuned on a large collection of 1500+ NLP tasks divided into various task categories.

OPT-IML

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

Building on the recent successes of distributed training of RL agents, R2D2 is an RL approach that trains a RNN-based RL agents from distributed prioritized experience replay. 
Using a single network architecture and fixed set of hyperparameters, Recurrent Replay Distributed DQN quadrupled the previous state of the art on Atari-57, and matches the state of the art on DMLab-30. 
It was the first agent to exceed human-level performance in 52 of the 57 Atari games.

R2D2

Recurrent Experience Replay in Distributed Reinforcement Learning

A novel low-resource intrinsic metric to evaluate word
embedding quality based on graph modularity.

Categorical Modularity

Evaluating Word Embeddings with Categorical Modularity

**Skim and Intensive Reading Model**, or **SIRM**, is a deep neural network for figuring out implied textual meaning. It consists of two main components, namely the skim reading component and intensive reading component. N-gram features are quickly extracted from the skim reading component, which is a combination of several convolutional neural networks, as skim (entire) information. An intensive reading component enables a hierarchical investigation for both local (sentence) and global (paragraph) representation, which encapsulates the current embedding and the contextual information with a dense connection.

SIRM

Read Beyond the Lines: Understanding the Implied Textual Meaning via a Skim and Intensive Reading Model

A capsule is an activation vector that basically executes on its inputs some complex internal
computations. Length of these activation vectors signifies the
probability of availability of a feature. Furthermore, the condition
of the recognized element is encoded as the direction in which
the vector is pointing. In traditional, CNN uses Max pooling for
invariance activities of neurons, which is nothing except a minor
change in input and the neurons of output signal will remains
same.

Capsule Network

Dynamic Routing Between Capsules

**Leaky Rectified Linear Unit**, or **Leaky ReLU**, is a type of activation function based on a [ReLU](https://paperswithcode.com/method/relu), but it has a small slope for negative values instead of a flat slope. The slope coefficient is determined before training, i.e. it is not learnt during training. This type of activation function is popular in tasks where we may suffer from sparse gradients, for example training generative adversarial networks.

Leaky ReLU

**Proximal Policy Optimization**, or **PPO**, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable performance of [TRPO](https://paperswithcode.com/method/trpo), while using only first-order optimization. 

Let $r\_{t}\left(\theta\right)$ denote the probability ratio $r\_{t}\left(\theta\right) = \frac{\pi\_{\theta}\left(a\_{t}\mid{s\_{t}}\right)}{\pi\_{\theta\_{old}}\left(a\_{t}\mid{s\_{t}}\right)}$, so $r\left(\theta\_{old}\right) = 1$. TRPO maximizes a “surrogate” objective:

$$ L^{\text{CPI}}\left({\theta}\right) = \hat{\mathbb{E}}\_{t}\left[\frac{\pi\_{\theta}\left(a\_{t}\mid{s\_{t}}\right)}{\pi\_{\theta\_{old}}\left(a\_{t}\mid{s\_{t}}\right)})\hat{A}\_{t}\right] = \hat{\mathbb{E}}\_{t}\left[r\_{t}\left(\theta\right)\hat{A}\_{t}\right] $$

Where $CPI$ refers to a conservative policy iteration. Without a constraint, maximization of $L^{CPI}$ would lead to an excessively large policy update; hence, we PPO modifies the objective, to penalize changes to the policy that move $r\_{t}\left(\theta\right)$ away from 1:

$$ J^{\text{CLIP}}\left({\theta}\right) = \hat{\mathbb{E}}\_{t}\left[\min\left(r\_{t}\left(\theta\right)\hat{A}\_{t}, \text{clip}\left(r\_{t}\left(\theta\right), 1-\epsilon, 1+\epsilon\right)\hat{A}\_{t}\right)\right] $$

where $\epsilon$ is a hyperparameter, say, $\epsilon = 0.2$. The motivation for this objective is as follows. The first term inside the min is $L^{CPI}$. The second term, $\text{clip}\left(r\_{t}\left(\theta\right), 1-\epsilon, 1+\epsilon\right)\hat{A}\_{t}$ modifies the surrogate
objective by clipping the probability ratio, which removes the incentive for moving $r\_{t}$ outside of the interval $\left[1 − \epsilon, 1 + \epsilon\right]$. Finally, we take the minimum of the clipped and unclipped objective, so the final objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse. 

One detail to note is that when we apply PPO for a network where we have shared parameters for actor and critic functions, we typically add to the objective function an error term on value estimation and an entropy term to encourage exploration.

Proximal Policy Optimization Algorithms

**ProxylessNet-CPU** is an image model learnt with the [ProxylessNAS](https://paperswithcode.com/method/proxylessnas) [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) algorithm that is optimized for CPU devices. It uses inverted residual blocks (MBConvs) from [MobileNetV2](https://paperswithcode.com/method/mobilenetv2) as its basic building block.

ProxylessNet-CPU

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

• Parallel Layers – We use a “parallel” formulation in each Transformer block (Wang & Komatsuzaki, 2021), rather than the standard “serialized” formulation. Specifically, the standard formulation can be written as:   
    y = x + MLP(LayerNorm(x + Attention(LayerNorm(x)))   

Whereas the parallel formulation can be written as:   
    y = x + MLP(LayerNorm(x)) + Attention(LayerNorm(x))   

The parallel formulation results in roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused. Ablation experiments showed a small quality degradation at 8B scale but no quality degradation at 62B scale, so we extrapolated that the effect of parallel layers should be quality neutral at the 540B scale.

Parallel Layers

PaLM: Scaling Language Modeling with Pathways

EncAttAgg introduced two attenders to tackle two problems: 1) We introduce a mutual attender layer to efficiently obtain the entity-pair-specific mention representations. 2) We introduce an integration attender to weight mention pairs of a target entity pair.

EncAttAgg

Compress an ensemble of models into a single one by averaging their weights (under certain pre-conditions).

Viet-Anh on Software

AI Glossary

Tech Terminology

DE-GAN: A Conditional Generative Adversarial Network for Document Enhancement

WaveVAE

Fawkes

VQ-VAE

UNiversal Image-TExt Representation Learning

Knowledge Distillation

UCNet

OPT-IML

Recurrent Replay Distributed DQN

Categorical Modularity

Skim and Intensive Reading Model

Capsule Network

Leaky ReLU

Proximal Policy Optimization

ProxylessNet-CPU

Parallel Layers

Encoder-Attender-Aggregator

Model Soups

Viet-Anh on Software