**Automatic Structured Variational Inference (ASVI)** is a fully automated method for constructing structured variational families, inspired by the closed-form update in conjugate Bayesian models. These convex-update families incorporate the forward pass of the input probabilistic program and can therefore capture complex statistical dependencies. Convex-update families have the same space and time complexity as the input probabilistic program and are therefore tractable for a very large family of models including both continuous and discrete variables.

ASVI

Automatic structured variational inference

DASPP is a deeper version of the [ASPP](https://paperswithcode.com/method/aspp) module (the latter from [DeepLabv3](https://paperswithcode.com/method/deeplabv3)) that adds standard 3 × 3 [convolution](https://paperswithcode.com/method/convolution) after 3 × 3 dilated convolutions to refine the features and also fusing the input and the output of the DASPP module via short [residual connection](https://paperswithcode.com/method/residual-connection). Also, the number of convolution filters of ASPP is reduced from 255 to 96 to gain computational performance.

DASPP

LiteSeg: A Novel Lightweight ConvNet for Semantic Segmentation

**CSPDenseNet-Elastic** is a convolutional neural network and object detection backbone where we apply the Cross Stage Partial Network (CSPNet) approach to [DenseNet-Elastic](https://paperswithcode.com/method/densenet-elastic). The CSPNet partitions the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network.

CSPDenseNet-Elastic

CSPNet: A New Backbone that can Enhance Learning Capability of CNN

**Channel-wise Cross Attention** is a module for semantic segmentation used in the [UCTransNet](https://paperswithcode.com/method/uctransnet) architecture. It is used to fuse features of inconsistent semantics between the Channel [Transformer](https://paperswithcode.com/method/transformer) and [U-Net](https://paperswithcode.com/method/u-net) decoder. It guides the channel and information filtration of the Transformer features and eliminates the ambiguity with the decoder features.

Mathematically, we take the $i$-th level Transformer output $\mathbf{O\_{i}} \in \mathbb{R}^{C×H×W}$ and i-th level decoder feature map $\mathbf{D\_{i}} \in \mathbb{R}^{C×H×W}$ as the inputs of Channel-wise Cross Attention. Spatial squeeze is performed by a [global average pooling](https://paperswithcode.com/method/global-average-pooling) (GAP) layer, producing vector $\mathcal{G}\left(\mathbf{X}\right) \in \mathbb{R}^{C×1×1}$ with its $k$th channel $\mathcal{G}\left(\mathbf{X}\right) = \frac{1}{H×W}\sum^{H}\_{i=1}\sum^{W}\_{j=1}\mathbf{X}^{k}\left(i, j\right)$. We use this operation to embed the global spatial information and then generate the attention mask:

$$ \mathbf{M}\_{i} = \mathbf{L}\_{1} \cdot \mathcal{G}\left(\mathbf{O\_{i}}\right) + \mathbf{L}\_{2} \cdot \mathcal{G}\left(\mathbf{D}\_{i}\right) $$

where $\mathbf{L}\_{1} \in \mathbb{R}^{C×C}$ and $\mathbf{L}\_{2} \in \mathbb{R}^{C×C}$ and being weights of two Linear layers and the [ReLU](https://paperswithcode.com/method/relu) operator $\delta\left(\cdot\right)$. This operation in the equation above encodes the channel-wise dependencies. Following [ECA-Net](https://paperswithcode.com/method/eca-net) which empirically showed avoiding dimensionality reduction is important for learning channel attention, the authors use a single [Linear layer](https://paperswithcode.com/method/linear-layer) and sigmoid function to build the channel attention map. The resultant vector is used to recalibrate or excite $\mathbf{O\_{i}}$ to $\mathbf{\bar{O}\_{i}} = \sigma\left(\mathbf{M\_{i}}\right) \cdot \mathbf{O\_{i}}$, where the activation $\sigma\left(\mathbf{M\_{i}}\right)$ indicates the importance of each channel. Finally, the masked $\mathbf{\bar{O}}\_{i}$ is concatenated with the up-sampled features of the $i$-th level decoder.

Channel-wise Cross Attention

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer

**FixRes** is an image scaling strategy that seeks to optimize classifier performance. It is motivated by the observation that data augmentations induce a significant discrepancy between the size of the objects seen by the classifier at train and test time: in fact, a lower train resolution improves the classification at test time! FixRes is a simple strategy to optimize the classifier performance, that employs different train and test resolutions. The calibrations are: (a) calibrating the object sizes by adjusting the crop size and (b) adjusting statistics before spatial pooling.

FixRes

Fixing the train-test resolution discrepancy

**MAML**, or **Model-Agnostic Meta-Learning**, is a model and task-agnostic algorithm for meta-learning that trains a model’s parameters such that a small number of gradient updates will lead to fast learning on a new task.

Consider a model represented by a parametrized function $f\_{\theta}$ with parameters $\theta$. When adapting to a new task $\mathcal{T}\_{i}$, the model’s parameters $\theta$ become $\theta'\_{i}$. With MAML, the updated parameter vector $\theta'\_{i}$ is computed using one or more gradient descent updates on task $\mathcal{T}\_{i}$. For example, when using one gradient update,

$$ \theta'\_{i} = \theta - \alpha\nabla\_{\theta}\mathcal{L}\_{\mathcal{T}\_{i}}\left(f\_{\theta}\right) $$

The step size $\alpha$ may be fixed as a hyperparameter or metalearned. The model parameters are trained by optimizing for the performance of $f\_{\theta'\_{i}}$ with respect to $\theta$ across tasks sampled from $p\left(\mathcal{T}\_{i}\right)$. More concretely the meta-objective is as follows:

$$ \min\_{\theta} \sum\_{\mathcal{T}\_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}\_{\mathcal{T\_{i}}}\left(f\_{\theta'\_{i}}\right) = \sum\_{\mathcal{T}\_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}\_{\mathcal{T\_{i}}}\left(f\_{\theta - \alpha\nabla\_{\theta}\mathcal{L}\_{\mathcal{T}\_{i}}\left(f\_{\theta}\right)}\right) $$

Note that the meta-optimization is performed over the model parameters $\theta$, whereas the objective is computed using the updated model parameters $\theta'$. In effect MAML aims to optimize the model parameters such that one or a small number of gradient steps on a new task will produce maximally effective behavior on that task. The meta-optimization across tasks is performed via stochastic gradient descent ([SGD](https://paperswithcode.com/method/sgd)), such that the model parameters $\theta$ are updated as follows:

$$ \theta \leftarrow \theta - \beta\nabla\_{\theta} \sum\_{\mathcal{T}\_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}\_{\mathcal{T\_{i}}}\left(f\_{\theta'\_{i}}\right)$$

where $\beta$ is the meta step size.

MAML

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

**VoVNetV2** is a convolutional neural network that improves upon [VoVNet](https://paperswithcode.com/method/vovnet) with two effective strategies: (1) [residual connection](https://paperswithcode.com/method/residual-connection) for alleviating the optimization problem of larger VoVNets and (2) effective Squeeze-Excitation (eSE) dealing with the channel information loss problem of the original squeeze-and-excitation module.

VoVNetV2

CenterMask : Real-Time Anchor-Free Instance Segmentation

A **Harmonic Network**, or **Harm-Net**, is a type of convolutional neural network that replaces convolutional layers with "harmonic blocks" that use [Discrete Cosine Transform](https://paperswithcode.com/method/discrete-cosine-transform) (DCT) filters. These blocks can be useful in  truncating high-frequency information (possible due to the redundancies in the spectral domain).

Harm-Net

Harmonic Convolutional Networks based on Discrete Cosine Transform

A **BiGAN**, or **Bidirectional GAN**, is a type of generative adversarial network where the generator  not only maps latent samples to generated data, but also has an inverse mapping from data to the latent representation. The motivation is to make a type of GAN that can learn rich representations for us in applications like unsupervised learning.

In addition to the generator $G$ from the standard [GAN](https://paperswithcode.com/method/gan) framework, BiGAN includes an encoder $E$ which maps data $\mathbf{x}$ to latent representations $\mathbf{z}$. The BiGAN discriminator $D$ discriminates not only in data space ($\mathbf{x}$ versus $G\left(\mathbf{z}\right)$), but jointly in data and latent space (tuples $\left(\mathbf{x}, E\left(\mathbf{x}\right)\right)$ versus $\left(G\left(z\right), z\right)$), where the latent component is either an encoder output $E\left(\mathbf{x}\right)$ or a generator input $\mathbf{z}$.

BiGAN

Adversarial Feature Learning

**Pixel2Style2Pixel**, or **pSp**, is an image-to-image translation framework that is based on a novel encoder that directly generates a series of style vectors which are fed into a pretrained [StyleGAN](https://paperswithcode.com/method/stylegan) generator, forming the extended $\mathcal{W+}$ latent space. Feature maps are first extracted using a standard feature pyramid over a [ResNet](https://paperswithcode.com/method/resnet) backbone. Then, for each of $18$ target styles, a small mapping network is trained to extract the learned styles from the corresponding feature map, where styles $(0-2)$ are generated from the small feature map, $(3-6)$ from the medium feature map, and $(7-18)$ from the largest feature map. The mapping network, map2style, is a small fully convolutional network, which gradually reduces spatial size using a set of 2-strided convolutions followed by [LeakyReLU](https://paperswithcode.com/method/leaky-relu) activations. Each generated 512 vector, is fed into [StyleGAN](https://paperswithcode.com/method/stylegan), starting from its matching affine transformation, $A$.

pixel2style2pixel

Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation

**L2M** is a learning algorithm that can work for most cross-domain distribution matching tasks. It automatically learns the cross-domain distribution matching without relying on hand-crafted priors on the matching loss. Instead, L2M reduces the inductive bias by using a meta-network to learn the distribution matching loss in a data-driven way.

Learning to Match Distributions for Domain Adaptation

The **Introspective Adversarial Network (IAN)** is a hybridization of [GANs](https://paperswithcode.com/method/gan) and [VAEs](https://paperswithcode.com/method/vae) that leverages the power of the adversarial objective while maintaining the VAE’s efficient inference mechanism. It uses the discriminator of the GAN, $D$, as a feature extractor for an inference subnetwork, $E$, which is implemented as a fully-connected layer on top of the final convolutional layer of the discriminator. We infer latent values $Z \sim E\left(X\right) = q\left(Z\mid{X}\right)$ for reconstruction and sample random values $Z \sim p\left(Z\right)$ from a standard normal for random image generation using the generator network, $G$.

Three distinct loss functions are used:

- $\mathcal{L}\_{img}$, the L1 pixel-wise reconstruction loss, which is preferred to the L2 reconstruction loss for its higher average gradient.
- $\mathcal{L\_{feature}}$, the feature-wise reconstruction loss, evaluated as the L2 difference between the original and reconstruction in the space of the hidden layers of the discriminator.
- $\mathcal{L}\_{adv}$, the ternary adversarial loss, a modification of the adversarial loss that forces the discriminator to label a sample as real, generated, or reconstructed (as opposed to a binary
real vs. generated label).

Including the VAE’s KL divergence between the inferred latents $E\left(X\right)$ and the prior $p\left(Z\right)$, the loss function for the generator and encoder network is thus:

$$\mathcal{L}\_{E, G} = \lambda\_{adv}\mathcal{L}\_{G\_{adv}} + \lambda\_{img}\mathcal{L}\_{img}  + \lambda\_{feature}\mathcal{L}\_{feature}  + D\_{KL}\left(E\left(X\right) || p\left(Z\right)\right) $$

Where the $\lambda$ terms weight the relative importance of each loss. We set $\lambda\_{img}$ to 3 and leave the other terms at 1. The discriminator is updated solely using the ternary adversarial loss. During each training step, the generator produces reconstructions $G\left(E\left(X\right)\right)$ (using the standard VAE reparameterization trick) from data $X$ and random samples $G\left(Z\right)$, while the discriminator observes $X$ as well as the reconstructions and random samples, and both networks are simultaneously updated.

Neural Photo Editing with Introspective Adversarial Networks

**Fisher-BRC** is an actor critic algorithm for offline reinforcement learning that encourages the learned policy to stay close to the data, namely parameterizing the critic as the $\log$-behavior-policy, which generated the offline dataset, plus a state-action value offset term, which can be learned using a neural network. Behavior regularization then corresponds to an appropriate regularizer on the offset term. A gradient penalty regularizer is used for the offset term, which is equivalent to Fisher divergence regularization, suggesting connections to the score matching and generative energy-based model literature.

Fisher-BRC

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

**DeepDrug** is a deep learning framework to overcome these shortcomings by using graph convolutional networks to learn the graphical representations of drugs and proteins such as molecular fingerprints and residual structures in order to boost the prediction accuracy.

DeepDrug

DeepDrug: A General Graph-Based Deep Learning Framework for Drug Relation Prediction

The **Convolutional vision Transformer (CvT)** is an architecture which incorporates convolutions into the [Transformer](https://paperswithcode.com/method/transformer). The CvT design introduces convolutions to two core sections of the ViT architecture.

First, the Transformers are partitioned into multiple stages that form a hierarchical structure of Transformers. The beginning of each stage consists of a convolutional token embedding that performs an overlapping [convolution](https://paperswithcode.com/method/convolution) operation with stride on a 2D-reshaped token map (i.e., reshaping flattened token sequences back to the spatial grid), followed by [layer normalization](https://paperswithcode.com/method/layer-normalization). This allows the model to not only capture local information, but also progressively decrease the sequence length while simultaneously increasing the dimension of token features across stages, achieving spatial downsampling while concurrently increasing the number of feature maps, as is performed in CNNs. 

Second, the linear projection prior to every self-attention block in the Transformer module is replaced with a proposed convolutional projection, which employs a s × s depth-wise separable convolution operation on an 2D-reshaped token map. This allows the model to further capture local spatial context and reduce semantic ambiguity in the attention mechanism. It also permits management of computational complexity, as the stride of convolution can be used to subsample the key and value matrices to improve efficiency by 4× or more, with minimal degradation of performance.

CvT: Introducing Convolutions to Vision Transformers

**ALBERT** is a [Transformer](https://paperswithcode.com/method/transformer) architecture based on [BERT](https://paperswithcode.com/method/bert) but with much fewer parameters. It achieves this through two parameter reduction techniques. The first is a factorized embeddings parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, the size of the hidden layers is separated from the size of vocabulary embedding. This makes it easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network. 

Additionally, ALBERT utilises a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed to address the ineffectiveness of the next sentence prediction (NSP) loss proposed in the original BERT.

ALBERT

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

**WaveGrad DBlocks** are used to downsample the temporal dimension of noisy waveform in [WaveGrad](https://paperswithcode.com/method/wavegrad). They are similar to UBlocks except that only one [residual block](https://paperswithcode.com/method/residual-block) is included. The dilation factors are 1, 2, 4 in the main branch. Orthogonal initialization is used.

WaveGrad DBlock

WaveGrad: Estimating Gradients for Waveform Generation

A **Stochastic Dueling Network**, or **SDN**, is an architecture for learning a value function $V$. The SDN learns both $V$ and $Q$ off-policy while maintaining consistency between the two estimates. At each time step it outputs a stochastic estimate of $Q$ and a deterministic estimate of $V$.

Viet-Anh on Software

AI Glossary

Tech Terminology

Automatic Structured Variational Inference

Deeper Atrous Spatial Pyramid Pooling

CSPDenseNet-Elastic

Channel-wise Cross Attention

FixRes

Model-Agnostic Meta-Learning

VoVNetV2

Harm-Net

Bidirectional GAN

pixel2style2pixel

Learning to Match

Introspective Adversarial Network

Fisher-BRC

DeepDrug

Convolutional Vision Transformer

ALBERT

WaveGrad DBlock

Stochastic Dueling Network

Viet-Anh on Software