A **Zero-padded Shortcut Connection** is a type of [residual connection](https://paperswithcode.com/method/residual-connection) used in the [PyramidNet](https://paperswithcode.com/method/pyramidnet) architecture. For PyramidNets, identity mapping alone cannot be used for a shortcut because the feature map dimension differs among individual residual units. Therefore, only a zero-padded shortcut or projection shortcut can be used for all the residual units. However,  a projection shortcut can hamper information propagation and lead to optimization problems, especially for very deep networks. On the other hand, the zero-padded shortcut avoids the overfitting problem because no additional parameters exist.

Zero-padded Shortcut Connection

Deep Pyramidal Residual Networks

**ShuffleNet** is a convolutional neural network designed specially for mobile devices with very limited computing power. The architecture utilizes two new operations, pointwise group [convolution](https://paperswithcode.com/method/convolution) and [channel shuffle](https://paperswithcode.com/method/channel-shuffle), to reduce computation cost while maintaining accuracy.

ShuffleNet

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

**Phase Shuffle** is a technique for removing pitched noise artifacts that come from using transposed convolutions in audio generation models. Phase shuffle is an operation with hyperparameter $n$. It randomly perturbs the phase of each layer’s activations by −$n$ to $n$ samples before input to the next layer.

In the original application in [WaveGAN](https://paperswithcode.com/method/wavegan), the authors only apply phase shuffle to the discriminator, as the latent vector already provides the generator a mechanism to manipulate the phase
of a resultant waveform. Intuitively speaking, phase shuffle makes the discriminator’s job more challenging by requiring invariance to the phase of the input waveform.

Phase Shuffle

Adversarial Audio Synthesis

**Cross View Training**, or **CVT**, is a semi-supervised algorithm for training distributed word representations that makes use of unlabelled and labelled examples. 

CVT adds $k$ auxiliary prediction modules to the model, a Bi-[LSTM](https://paperswithcode.com/method/lstm) encoder, which are used when learning on unlabeled examples. A prediction module is usually a small neural network (e.g., a hidden layer followed by a [softmax](https://paperswithcode.com/method/softmax) layer). Each one takes as input an intermediate representation $h^j(x_i)$ produced by the model (e.g., the outputs of one of the LSTMs in a Bi-LSTM model). It outputs a distribution over labels $p\_{j}^{\theta}\left(y\mid{x\_{i}}\right)$.

Each $h^j$ is chosen such that it only uses a part of the input $x_i$; the particular choice can depend on the task and model architecture. The auxiliary prediction modules are only used during training; the test-time prediction come from the primary prediction module that produces $p_\theta$.

Cross-View Training

Semi-Supervised Sequence Modeling with Cross-View Training

**PRNet+** is a multi-task neural network for outdoor position recovery from measurement record (MR) data. PRNet+ develops a feature extraction module to learn common local-, short- and long-term spatio-temporal locality from heterogeneous MR samples, with a convolutional neural network (CNN), long short-term memory cells ([LSTM](https://paperswithcode.com/method/lstm)), and attention mechanisms. Specifically, PRNet+ 1) allows the various-length sequences of MR samples, such that the two components (CNN and LSTM) are able to capture spatial locality from the samples within each MR sequence, 2) exploits two attention mechanisms for the time-interval between neighbouring MR samples, together with the one between neighbouring MR sequences, to capture temporal locality, and 3) incorporates the detected transportation modes and predicted locations of heterogeneous MR data into a joint loss for better result.

PRNet+

Outdoor Position Recovery from HeterogeneousTelco Cellular Data

A general multimodal attention unit for any number of modalities. Graphical models inspire it, i.e., it infers several attention beliefs via aggregated interaction messages.

Factor Graph Attention

**3D Dynamic Scene Graph**, or **DSG**, is a representation that captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatio-temporal relations among nodes.

3D Dynamic Scene Graph

Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs

**MaxUp** is an adversarial data augmentation technique for improving the generalization performance of machine learning models. The idea is to generate a set of augmented data with some random perturbations or transforms, and minimize the maximum, or worst case loss over the augmented data.  By doing so, we implicitly introduce a smoothness or robustness regularization against the random perturbations, and hence improve the generation performance.  For example, in the case of Gaussian perturbation, MaxUp is asymptotically equivalent to using the gradient norm of the loss as a penalty to encourage smoothness.

MaxUp

MaxUp: A Simple Way to Improve Generalization of Neural Network Training

**Pointer Networks** tackle problems where input and output data are sequential data, but can't be solved by seq2seq type models because discrete categories of output elements depend on the variable input size (and are not decided in advance).

A Pointer Network learns the conditional  probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. They solve the problem of variable size output dictionaries using [additive attention](https://paperswithcode.com/method/additive-attention). But instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, Pointer Networks use attention as a pointer to select a member of the input sequence as the output. 

Pointer-Nets can be used to learn approximate solutions to challenging geometric problems such as finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem.

Pointer Network

Pointer Networks

To aggregate global spatial information,
an SE block applies global pooling to the feature map.
However, it ignores pixel-wise spatial information,
which is important in dense prediction tasks.
Therefore, Roy et al. proposed
spatial and channel SE blocks (scSE). 
Like BAM, spatial SE blocks are used, complementing SE blocks, 
to provide spatial attention weights to focus on important regions.

Given the input feature map $X$, two parallel modules, spatial SE and channel SE, are applied to feature maps to encode spatial and channel information respectively. The channel SE module is an ordinary SE block, while the spatial SE module adopts $1\times 1$ convolution for spatial squeezing. The outputs from the two modules are fused. The overall process can be written as
\begin{align}
    s_c & = \sigma (W_{2} \delta (W_{1}\text{GAP}(X)))
\end{align}
\begin{align}
    X_\text{chn} & = s_c  X 
\end{align}
\begin{align}
    s_s &= \sigma(\text{Conv}^{1\times 1}(X))
\end{align}
\begin{align}
    X_\text{spa} & = s_s  X
\end{align}
\begin{align}
    Y &= f(X_\text{spa},X_\text{chn})  
\end{align}

where $f$ denotes the fusion function, which can be  maximum, addition, multiplication or concatenation. 

The proposed scSE block combines channel and spatial attention to
enhance features as well as 
capturing pixel-wise spatial information.
Segmentation tasks are greatly benefited as a result.
The integration of an scSE block in F-CNNs makes a consistent improvement in semantic segmentation at negligible extra cost.

scSE

Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks

**Random Search** replaces the exhaustive enumeration of all combinations by selecting them randomly. This can be simply applied to the discrete setting described above, but also generalizes to continuous and mixed spaces. It can outperform Grid search, especially when only a small number of hyperparameters affects the final performance of the machine learning algorithm. In this case, the optimization problem is said to have a low intrinsic dimensionality. Random Search is also embarrassingly parallel, and additionally allows the inclusion of prior knowledge by specifying the distribution from which to sample.


Extracted from [Wikipedia](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Random_search)

Source [Paper](https://dl.acm.org/doi/10.5555/2188385.2188395)

Image Source: [BERGSTRA AND BENGIO](https://dl.acm.org/doi/pdf/10.5555/2188385.2188395)

Random Search

Please enter a description about the method here

Pose Contrastive Learning

Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment

**Euclidean Norm Regularization** is a regularization step used in [generative adversarial networks](https://paperswithcode.com/methods/category/generative-adversarial-networks), and is typically added to both the generator and discriminator losses:

$$ R\_{z} = w\_{r} \cdot ||\Delta{z}||^{2}\_{2} $$

where the scalar weight $w\_{r}$ is a parameter.

Image: [LOGAN](https://paperswithcode.com/method/logan)

Euclidean Norm Regularization

Deep Compressed Sensing

PatchMerger is a module for Vision Transformers that decreases the number of tokens/patches passed onto each individual transformer encoder block whilst maintaining performance and reducing compute. PatchMerger takes linearly transforms an input of shape N patches × D dimensions through a learnable weight matrix of shape M output patches × D. This generates M scores, in which a Softmax function is applied for each score. The resulting output has a shape of M × N, which is multiplied to the original input to get an output of shape M × D.

Mathematically, $$Y = \text{softmax}({W^T}{X^T})X$$

Image and formula from: Renggli, C., Pinto, A. S., Houlsby, N., Mustafa, B., Puigcerver, J., & Riquelme, C. (2022). Learning to Merge Tokens in Vision Transformers. arXiv preprint arXiv:2202.12015.

Patch Merger

Learning to Merge Tokens in Vision Transformers

**MixConv**, or **Mixed Depthwise Convolution**, is a type of [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) that naturally mixes up multiple kernel sizes in a single [convolution](https://paperswithcode.com/method/convolution). It is based on the insight that depthwise convolution applies a single kernel size to all channels, which MixConv overcomes by combining the benefits of multiple kernel sizes. It does this by partitioning channels into groups and applying a different kernel size to each group.

MixConv

MixConv: Mixed Depthwise Convolutional Kernels

A method to convert a Tree Ensemble model into a Rule list. This makes the AI model more transparent.

TE2Rules

TE2Rules: Explaining Tree Ensembles using Rules

**Meta Reward Learning (MeRL)** is a meta-learning method for the problem of learning from sparse and underspecified rewards. For example, an agent receives a complex input, such as a natural language instruction, and needs to generate a complex response, such as an action sequence, while only receiving binary success-failure feedback. The key insight of MeRL in dealing with underspecified rewards is that spurious trajectories and programs that achieve accidental success are detrimental to the agent's generalization performance. For example, an agent might be able to solve a specific instance of the maze problem above. However, if it learns to perform spurious actions during training, it is likely to fail when provided with unseen instructions. To mitigate this issue, MeRL optimizes a more refined auxiliary reward function, which can differentiate between accidental and purposeful success based on features of action trajectories. The auxiliary reward is optimized by maximizing the trained agent's performance on a hold-out validation set via meta learning.

MeRL

Learning to Generalize from Sparse and Underspecified Rewards

**TrOCR** is an end-to-end [Transformer](https://paperswithcode.com/methods/category/transformers)-based OCR model for text recognition with pre-trained CV and NLP models. It leverages the [Transformer](https://paperswithcode.com/method/transformer) architecture for both image understanding and wordpiece-level text generation. It first resizes the input text image into $384 × 384$ and then the image is split into a sequence of 16 patches which are used as the input to image Transformers.  Standard Transformer architecture with the [self-attention mechanism](https://paperswithcode.com/method/scaled) is leveraged on both encoder and decoder parts, where wordpiece units are generated as the recognized text from the input image.

Viet-Anh on Software

AI Glossary

Tech Terminology

Zero-padded Shortcut Connection

ShuffleNet

Phase Shuffle

Cross-View Training

PRNet+

Factor Graph Attention

3D Dynamic Scene Graph

MaxUp

Pointer Network

Spatial and Channel SE Blocks

Random Search

Self-Supervised Cross View Cross Subject Pose Contrastive Learning

Euclidean Norm Regularization

Patch Merger Module

Mixed Depthwise Convolution

Tree Ensemble to Rules

Meta Reward Learning

TrOCR

Viet-Anh on Software