A dual graph convolutional neural network jointly considers the two essential assumptions of semi-supervised learning: (1) local consistency and (2) global consistency. Accordingly, two convolutional neural networks are devised to embed the local-consistency-based and global-consistency-based knowledge, respectively.

Description  and image from: [Dual Graph Convolutional Networks for Graph-Based Semi-Supervised Classification](https://persagen.com/files/misc/zhuang2018dual.pdf)

DualGCN

**Gaussian Gated Linear Network**, or **G-GLN**, is a multi-variate extension to the recently proposed [GLN](https://paperswithcode.com/method/gln) family of deep neural networks by reformulating the GLN neuron as a gated product of Gaussians. This Gaussian Gated Linear Network (G-GLN) formulation exploits the fact that exponential family densities are closed under multiplication, a property that has seen much use in [Gaussian Process](https://paperswithcode.com/method/gaussian-process) and related literature. Similar to the Bernoulli GLN, every neuron in the G-GLN directly predicts the target distribution.  

Precisely, a G-GLN is a feed-forward network of data-dependent distributions. Each neuron calculates the sufficient statistics $\left(\mu, \sigma\_{2}\right)$ for its associated PDF using its active weights, given those emitted by neurons in the preceding layer. It consists of consists of $L+1$ layers indexed by $i \in\{0, \ldots, L\}$ with $K\_{i}$ neurons in each layer. The weight space for a neuron in layer $i$ is denoted by $\mathcal{W}\_{i}$; the subscript is needed since the dimension of the weight space depends on $K_{i-1}$. Each neuron/distribution is indexed by its position in the network when laid out on a grid; for example, $f\_{i k}$ refers to the family of PDFs defined by the $k$ th neuron in the $i$ th layer. Similarly, $c\_{i k}$ refers to the context function associated with each neuron in layers $i \geq 1$, and $\mu\_{i k}$ and $\sigma\_{i k}^{2}$ (or $\Sigma\_{i k}$ in the multivariate case) referring to the sufficient statistics for each Gaussian PDF.

There are two types of input to neurons in the network. The first is the side information, which can be thought of as the input features, and is used to determine the weights used by each neuron via half-space gating. The second is the input to the neuron, which is the PDFs output by the previous layer, or in the case of layer 0, some provided base models. To apply a G-GLN in a supervised learning setting, we need to map the sequence of input-label pairs $\left(x\_{t}, y\_{t}\right)$ for $t=1,2, \ldots$ onto a sequence of (side information, base Gaussian PDFs, label) triplets $\left(z\_{t},\left$f\_{0 i}\right$\_{i}, y\_{t}\right)$. The side information $z\_{t}$ is set to the (potentially normalized) input features $x\_{t}$. The Gaussian PDFs for layer 0 will generally include the necessary base Gaussian PDFs to span the target range, and optionally some base prediction PDFs that capture domain-specific knowledge.

G-GLN

Gaussian Gated Linear Networks

**UCTransNet** is an end-to-end deep learning network for semantic segmentation that takes [U-Net](https://paperswithcode.com/method/u-net) as the main structure of the network. The original skip connections of U-Net are replaced by CTrans consisting of two components: [Channel-wise Cross fusion Transformer](https://paperswithcode.com/method/channel-wise-cross-fusion-transformer) ([CCT](https://paperswithcode.com/method/cct)) and [Channel-wise Cross Attention](https://paperswithcode.com/method/channel-wise-cross-attention) (CCA) to guide the fused multi-Scale channel-wise information to effectively connect to the decoder features for eliminating the ambiguity.

UCTransNet

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer

**Powerpropagation** is a weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, it gives rise to weight updates exhibiting a “rich get richer” dynamic, leaving low-magnitude parameters largely unaffected by learning.In other words, parameters with larger magnitudes are allowed to adapt faster in order to represent the required features to solve the task, while smaller magnitude parameters are restricted, making it more likely that they will be irrelevant in representing the learned solution.  Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely.

Powerpropagation

Powerpropagation: A sparsity inducing weight reparameterisation

Triplet attention comprises of three branches each responsible for capturing crossdimension between the spatial dimensions and channel dimension of the input. Given an input tensor with shape (C × H × W), each branch is responsible for aggregating cross-dimensional interactive features between either the spatial dimension H or W and the channel dimension C.

Triplet Attention

Rotate to Attend: Convolutional Triplet Attention Module

** Sigmoid Linear Units**, or **SiLUs**, are activation functions for
neural networks. The activation of the SiLU is computed by the sigmoid function multiplied by its input, or $$ x\sigma(x).$$

See [Gaussian Error Linear Units](https://arxiv.org/abs/1606.08415) ([GELUs](https://paperswithcode.com/method/gelu)) where the SiLU was originally coined, and see [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://arxiv.org/abs/1702.03118) and [Swish: a Self-Gated Activation Function](https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later.

SiLU

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

**FreeAnchor** is an anchor supervision method for object detection. Many CNN-based object detectors assign anchors for ground-truth objects under the restriction of object-anchor Intersection-over-Unit (IoU). In contrast, FreeAnchor is a learning-to-match approach that breaks the IoU restriction, allowing objects to match anchors in a flexible manner. It updates hand-crafted anchor assignment to free anchor matching by formulating detector training as a maximum likelihood estimation (MLE) procedure. FreeAnchor targets at learning features which best explain a class of objects in terms of both classification and localization.

FreeAnchor

FreeAnchor: Learning to Match Anchors for Visual Object Detection

**Displaced Aggregation Unit** replaces classic [convolution](https://paperswithcode.com/method/convolution) layer in ConvNets with learnable positions of units.  This introduces explicit structure of hierarchical compositions and results in several benefits:

* fully adjustable and **learnable receptive fields** through spatially-adjustable filter units
* **reduced parameters** for spatial coverage
efficient inference
* **decupling** of the parameters from the receptive field sizes

More information can be found [here.](https://www.vicos.si/Research/DeepCompositionalNet)

DAU-ConvNet

Spatially-Adaptive Filter Units for Compact and Efficient Deep Neural Networks

The extremely low computational cost of lightweight CNNs constrains the depth and width of the networks, further decreasing their representational power. To address the above problem, Chen et al. proposed dynamic convolution, a novel operator design that increases  representational power with negligible additional computational cost and does not change the width or depth of the network in parallel with CondConv.

Dynamic convolution uses $K$ parallel convolution kernels of the same  size and input/output dimensions instead of one kernel per layer. Like SE blocks, it adopts a squeeze-and-excitation mechanism to generate the attention weights for the different convolution kernels. These kernels are then aggregated dynamically by weighted summation and applied to the input feature map $X$:
\begin{align}
    s & = \text{softmax} (W_{2} \delta (W_{1}\text{GAP}(X)))
\end{align}
\begin{align}
    \text{DyConv} &= \sum_{i=1}^{K} s_k \text{Conv}_k 
\end{align}
\begin{align}
    Y &= \text{DyConv}(X)
\end{align}
Here the convolutions are combined by summation of weights and biases of convolutional kernels. 

Compared to applying convolution to the feature map, the computational cost of squeeze-and-excitation and weighted summation is extremely low. Dynamic convolution thus provides an efficient operation to improve  representational power and can be easily used as a replacement for any convolution.

Dynamic Convolution

Dynamic Convolution: Attention over Convolution Kernels

ChebNet involves a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs.

Description from: [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://arxiv.org/pdf/1606.09375.pdf)

ChebNet

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Nyströmformer replaces the self-attention in [BERT](https://paperswithcode.com/method/bert)-small and BERT-base using the proposed Nyström approximation. This reduces self-attention complexity to $O(n)$ and allows the [Transformer](https://paperswithcode.com/method/transformer) to support longer sequences.

Nyströmformer

Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

The **Truncation Trick** is a latent sampling procedure for generative adversarial networks, where we sample $z$ from a truncated normal (where values which fall outside a range are resampled to fall inside that range). 
The original implementation was in [Megapixel Size Image Creation with GAN](https://paperswithcode.com/paper/megapixel-size-image-creation-using).
In [BigGAN](http://paperswithcode.com/method/biggan), the authors find this provides a boost to the Inception Score and FID.

Truncation Trick

Megapixel Size Image Creation using Generative Adversarial Networks

Using cubic splines to improve factorization machine accuracy with numerical features

FM with splines

Basis Function Encoding of Numerical Features in Factorization Machines for Improved Accuracy

**TSRUc**, or **Transformation-based Spatial Recurrent Unit c**, is a modification of a [ConvGRU](https://paperswithcode.com/method/cgru) used in the [TriVD-GAN](https://paperswithcode.com/method/trivd-gan) architecture for video generation.

Instead of computing the reset gate $r$ and resetting $h\_{t−1}$, the TSRUc computes the parameters of a transformation $\theta$, which we use to warp $h\_{t−1}$. The rest of our model is unchanged (with $\hat{h}\_{t-1}$ playing the role of $h'\_{t}$ in $c$’s update equation from ConvGRU. The TSRUc module is described by the following equations:

$$ \theta\_{h,x} = f\left(h\_{t−1}, x\_{t}\right) $$

$$ \hat{h}\_{t-1} = w\left(h\_{t-1}; \theta\_{h, x}\right) $$

$$ c = \rho\left(W\_{c} \star\_{n}\left[\hat{h}\_{t-1};x\_{t}\right] + b\_{c} \right) $$

$$ u = \sigma\left(W\_{u} \star\_{n}\left[h\_{t-1};x\_{t}\right] + b\_{u} \right) $$

$$ h\_{t} = u \odot h\_{t-1} + \left(1-u\right) \odot c $$

In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and [ReLU](https://paperswithcode.com/method/relu) functions respectively and the $\star\_{n}$ represents a [convolution](https://paperswithcode.com/method/convolution) with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUc

Transformation-based Adversarial Video Prediction on Large-Scale Data

A **Cyclical Learning Rate Policy** combines a linear learning rate decay with warm restarts.

Image: [ESPNetv2](https://paperswithcode.com/method/espnetv2)

Cyclical Learning Rate Policy

An **XCiT Layer** is the main building block of the [XCiT](https://paperswithcode.com/method/xcit) architecture which uses a [cross-covariance attention]() operator as its principal operation. The XCiT layer consists of three main blocks, each preceded by [LayerNorm](https://paperswithcode.com/method/layer-normalization) and followed by a [residual connection](https://paperswithcode.com/method/residual-connection): (i) the core [cross-covariance attention](https://paperswithcode.com/method/cross-covariance-attention) (XCA) operation, (ii) the [local patch interaction](https://paperswithcode.com/method/local-patch-interaction) (LPI) module, and (iii) a [feed-forward network](https://paperswithcode.com/method/feedforward-network) (FFN). By transposing the query-key interaction, the computational complexity of XCA is linear in the number of data elements N, rather than quadratic as in conventional self-attention.

XCiT Layer

XCiT: Cross-Covariance Image Transformers

**Adaptive Dropout** is a regularization technique that extends dropout by allowing the dropout probability to be different for different units. The intuition is that there may be hidden units that can individually make confident predictions for the presence or absence of an important feature or combination of features. [Dropout](https://paperswithcode.com/method/dropout) will ignore this confidence and drop the unit out 50% of the time. 

Denote the activity of unit $j$ in a deep neural network by $a\_{j}$ and assume that its inputs are {$a\_{i}: i < j$}. In dropout, $a\_{j}$ is randomly set to zero with probability 0.5. Let $m\_{j}$ be a binary variable that is used to mask, the activity $a\_{j}$, so that its value is:

$$ a\_{j} = m\_{j}g \left( \sum\_{i: i<j}w\_{j, i}a\_{i} \right)$$

where $w\_{j,i}$ is the weight from unit $i$ to unit $j$ and $g\left(·\right)$ is the activation function and $a\_{0} = 1$ accounts for biases. Whereas in standard dropout, $m\_{j}$ is Bernoulli with probability $0.5$, adaptive dropout uses adaptive dropout probabilities that depends on input activities:

$$ P\left(m\_{j} = 1\mid{\{a\_{i}: i < j\}}\right) = f \left( \sum\_{i: i<j}\pi{\_{j, i}a\_{i}} \right) $$

where $\pi\_{j, i}$ is the weight from unit $i$ to unit $j$ in the standout network or the adaptive dropout network; $f(·)$ is a sigmoidal function. Here 'standout' refers to a binary belief network is that is overlaid on a neural network as part of the overall regularization technique.

Adaptive Dropout

Adaptive dropout for training deep neural networks

**ACKTR**, or **Actor Critic with Kronecker-factored Trust Region**, is an actor-critic method for reinforcement learning that applies [trust region optimization](https://paperswithcode.com/method/trpo) using a recently proposed Kronecker-factored approximation to the curvature. The method extends the framework of natural policy gradient and optimizes both the actor and the critic using Kronecker-factored approximate
curvature (K-FAC) with trust region.

Viet-Anh on Software

AI Glossary

Tech Terminology

Dual Graph Convolutional Networks

Gaussian Gated Linear Network