Stochastic Steady-state Embedding (SSE) is an algorithm that can learn many steady-state algorithms over graphs. Different from graph neural network family models, SSE is trained stochastically which only requires 1-hop information, but can capture fixed point relationships efficiently and effectively.

Description and Image from: [Learning Steady-States of Iterative Algorithms over Graphs](https://proceedings.mlr.press/v80/dai18a.html)

Learning Steady-States of Iterative Algorithms over Graphs

Recurrent Event Network (RE-NET) is an autoregressive architecture for predicting future interactions. The occurrence of a fact (event) is modeled as a probability distribution conditioned on temporal sequences of past knowledge graphs. RE-NET employs a recurrent event encoder to encode past facts and uses a neighborhood aggregator to model the connection of facts at the same timestamp. Future facts can then be inferred in a sequential manner based on the two modules.

RE-NET

Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs

Deformable ConvNets do not learn an affine transformation. They divide convolution into two steps, firstly sampling features on a regular grid $ \mathcal{R} $ from the input feature map, then aggregating sampled features by weighted summation using a convolution kernel. The process can be written as:
\begin{align}
    Y(p_{0}) &= \sum_{p_i \in \mathcal{R}} w(p_{i}) X(p_{0} + p_{i})
\end{align}
\begin{align}
    \mathcal{R}  &= \{(-1,-1), (-1, 0), \dots, (1, 1)\}
\end{align}
The deformable convolution augments the sampling process by introducing a group of learnable offsets $\Delta p_{i}$ which can be generated by a lightweight CNN. Using the offsets $\Delta p_{i}$, the deformable convolution can be formulated as:
\begin{align}
    Y(p_{0}) &= \sum_{p_i \in \mathcal{R}} w(p_{i}) X(p_{0} + p_{i} + \Delta p_{i}). 
\end{align}
Through the above method, adaptive sampling is achieved.
However, $\Delta p_{i}$ is a floating point value
unsuited to grid sampling. 
To address this problem, bilinear interpolation is used. Deformable RoI pooling is also used, which greatly improves object detection. 

Deformable ConvNets adaptively select the important regions and enlarge the valid receptive field of convolutional neural networks; this is important in object detection and semantic segmentation tasks.

Deformable ConvNets

Deformable Convolutional Networks

The **ScaledSoftSign** is a modification of **[SoftSign](https://paperswithcode.com/method/softsign-activation)** activation function that has trainable parameters.

$$ScaledSoftSign(x) = \frac{\alpha x}{\beta + |x|}$$

ScaledSoftSign

Trainable Activations for Image Classification

Please enter a description about the method here

Regularisation of Neural Networks by Enforcing Lipschitz Continuity

**TDN**, or **Temporaral Difference Network**, is an action recognition model that aims to capture multi-scale temporal information. To fully capture temporal information over the entire video, the TDN is established with a two-level difference modeling paradigm. Specifically, for local motion modeling, temporal difference over consecutive frames is used to supply 2D CNNs with finer motion pattern, while for global motion modeling, temporal difference across segments is incorporated to capture long-range structure for motion feature excitation.

TDN: Temporal Difference Networks for Efficient Action Recognition

**DropConnect** generalizes [Dropout](https://paperswithcode.com/method/dropout) by randomly dropping the weights rather than the activations with probability $1-p$. DropConnect is similar to Dropout as it introduces dynamic sparsity within the model, but differs in that the sparsity is on the weights $W$, rather than the output vectors of a layer. In other words, the fully connected layer with DropConnect becomes a sparsely connected layer in which the connections are chosen at random during the training stage. Note that this is not equivalent to setting $W$ to be a fixed sparse matrix during training.

For a DropConnect layer, the output is given as:

$$ r = a \left(\left(M * W\right){v}\right)$$

Here $r$ is the output of a layer, $v$ is the input to a layer, $W$ are weight parameters, and $M$ is a binary matrix encoding the connection information where $M\_{ij} \sim \text{Bernoulli}\left(p\right)$. Each element of the mask $M$ is drawn independently for each example during training, essentially instantiating a different connectivity for each example seen. Additionally, the biases are also masked out during training.

DropConnect

Regularization of Neural Networks using DropConnect

**Asynchronous Interaction Aggregation**, or **AIA**, is a network that leverages different interactions to boost action detection. There are two key designs in it: one is the Interaction Aggregation structure (IA) adopting a uniform paradigm to model and integrate multiple types of interaction; the other is the Asynchronous Memory Update algorithm (AMU) that enables us to achieve better performance by modeling very long-term interaction dynamically.

Asynchronous Interaction Aggregation

Asynchronous Interaction Aggregation for Action Detection

**Fishr** is a learning scheme to enforce domain invariance in the space of the gradients of the loss function: specifically, it introduces a regularization term that matches the domain-level variances of gradients across training domains. Critically, the strategy exhibits close relations with the Fisher Information and the Hessian of the loss. Forcing domain-level gradient covariances to be similar during the learning procedure eventually aligns the domain-level loss landscapes locally around the final weights.

Fishr

Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization

In machine learning, a hyperparameter is a parameter whose value is used to control learning process, and HPO is the problem of choosing a set of optimal hyperparameters for a learning algorithm.

Algorithms for Hyper-Parameter Optimization

**rnnDrop** is a [dropout](https://paperswithcode.com/method/dropout) based regularization technique for [recurrent neural networks](https://paperswithcode.com/methods/category/recurrent-neural-networks). It amounts to using the same dropout mask at every timestep. It drops both the non-recurrent and recurrent connections. A simple figure to explain the idea is shown to the right. The figure shows an RNN being trained with rnnDrop for three frames $\left(t-1, t, t+1\right)$ on two different training sequences in the data (denoted as ‘sequence1’ and ‘sequence2’). The black circles denote the randomly omitted hidden nodes during training, and the dotted arrows stand for the model weights connected to those omitted nodes.

*From: RnnDrop: A Novel Dropout for RNNs in ASR by Moon et al*

rnnDrop

**Path Aggregation Network**, or **PANet**, aims to boost information flow in a proposal-based instance segmentation framework. Specifically, the feature hierarchy is enhanced with accurate localization signals in lower layers by [bottom-up path augmentation](https://paperswithcode.com/method/bottom-up-path-augmentation), which shortens the information path between lower layers and topmost feature. Additionally, [adaptive feature pooling](https://paperswithcode.com/method/adaptive-feature-pooling) is employed, which links feature grid and all feature levels to make useful information in each feature level propagate directly to following proposal subnetworks. A complementary branch capturing different views for each proposal is created to further improve mask prediction.

PANet

Path Aggregation Network for Instance Segmentation

Class activation maps could be used to interpret the prediction decision made by the convolutional neural network (CNN).

Image source: [Learning Deep Features for Discriminative Localization](https://paperswithcode.com/paper/learning-deep-features-for-discriminative)

Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks

**ClariNet** is an end-to-end text-to-speech architecture. Unlike previous TTS systems which use text-to-spectogram models with a separate waveform [synthesizer](https://paperswithcode.com/method/synthesizer) (vocoder), ClariNet is a text-to-wave architecture that is fully convolutional and can be trained from scratch. In ClariNet, the [WaveNet](https://paperswithcode.com/method/wavenet) module is conditioned on the hidden states instead of the mel-spectogram. The architecture is otherwise based on [Deep Voice 3](https://paperswithcode.com/method/deep-voice-3).

ClariNet

ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

**SNIPER** is a multi-scale training approach for instance-level recognition tasks like object detection and instance-level segmentation. Instead of processing all pixels in an image pyramid, SNIPER selectively processes context regions around the ground-truth objects (a.k.a chips). This can help to speed up multi-scale training as it operates on low-resolution chips. Due to its memory-efficient design, SNIPER can benefit from [Batch Normalization](https://paperswithcode.com/method/batch-normalization) during training and it makes larger batch-sizes possible for instance-level recognition tasks on a single GPU.

SNIPER

SNIPER: Efficient Multi-Scale Training

The **Invertible 1x1 Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) used in flow-based generative models that reverses the ordering of channels. The weight matrix is initialized as a random rotation matrix. The log-determinant of an invertible 1 × 1 convolution of a $h \times w \times c$ tensor $h$ with $c \times c$ weight matrix $\mathbf{W}$ is straightforward to compute:

$$ \log | \text{det}\left(\frac{d\text{conv2D}\left(\mathbf{h};\mathbf{W}\right)}{d\mathbf{h}}\right) | = h \cdot w \cdot \log | \text{det}\left(\mathbf{W}\right) | $$

Invertible 1x1 Convolution

Glow: Generative Flow with Invertible 1x1 Convolutions

**CRF-RNN** is a formulation of a [CRF](https://paperswithcode.com/method/crf) as a Recurrent Neural Network. Specifically it formulates mean-field approximate inference for the Conditional Random Fields with Gaussian pairwise potentials as Recurrent Neural Networks.

CRF-RNN

Conditional Random Fields as Recurrent Neural Networks

**VEGA** is an AutoML framework that is compatible and optimized for multiple hardware platforms. It integrates various modules of AutoML, including [Neural Architecture Search](https://paperswithcode.com/method/neural-architecture-search) (NAS), Hyperparameter Optimization (HPO), Auto Data Augmentation, Model Compression, and Fully Train. To support a variety of search algorithms and tasks, it involves a fine-grained search space and a description language to enable easy adaptation to different search algorithms and tasks.

Viet-Anh on Software

AI Glossary

Tech Terminology

Stochastic Steady-state Embedding

Recurrent Event Network