**Step Decay** is a learning rate schedule that drops the learning rate by a factor every few epochs, where the number of epochs is a hyperparameter.

Image Credit: [Suki Lau](https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1)

Step Decay

**PermuteFormer** is a [Performer](https://paperswithcode.com/method/performer)-based model with relative position encoding that scales linearly on long sequences. PermuteFormer applies position-dependent transformation on queries and keys to encode positional information into the attention module. This transformation is carefully crafted so that the final output of self-attention is not affected by absolute positions of tokens.

Each token’s query / key feature is illustrated as a row of blocks in the figure, and its elements are marked with different colors. The position-aware permutation permutes elements of each token’s query / key feature along the head size dimension in each attention head. Depending on the token’s position, the permutation applied to query / key feature is different.

PermuteFormer

PermuteFormer: Efficient Relative Position Encoding for Long Sequences

**Cross-Covariance Image Transformers**, or **XCiT**, is a type of [vision transformer](https://paperswithcode.com/methods/category/vision-transformer) that aims to combine the accuracy of [conventional transformers](https://paperswithcode.com/methods/category/transformers) with the scalability of [convolutional architectures](https://paperswithcode.com/methods/category/convolutional-neural-networks). 

The [self-attention operation](https://paperswithcode.com/method/scaled) underlying transformers yields global interactions between all tokens, i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. The authors propose a “transposed” version of self-attention called [cross-covariance attention](https://paperswithcode.com/method/cross-covariance-attention) that operates across feature channels rather than tokens, where the interactions are based on the cross-covariances matrix between keys and queries.

XCiT

XCiT: Cross-Covariance Image Transformers

**CodeGen** is an autoregressive transformers with next-token prediction language modeling as the learning objective trained on a natural language corpus and programming language data curated from GitHub.

CodeGen

CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene understanding and visual odometry, which are key components in autonomous and robotic systems. 
Approaches based on the state of the art vision transformer architectures are extremely deep and complex not suitable for real-time inference operations on edge and autonomous systems equipped with low resources (i.e. robot indoor navigation and surveillance). This paper presents SPEED, a Separable Pyramidal pooling EncodEr-Decoder architecture designed to achieve real-time frequency performances on multiple hardware platforms. The proposed model is a fast-throughput deep architecture for MDE able to obtain depth estimations with high accuracy from low resolution images using minimum hardware resources (i.e. edge devices). Our encoder-decoder model exploits two depthwise separable pyramidal pooling layers, which allow to increase the inference frequency while reducing the overall computational complexity. The proposed method performs better than other fast-throughput architectures in terms of both accuracy and frame rates, achieving real-time performances over cloud CPU, TPU and the NVIDIA Jetson TX1 on two indoor benchmarks: the NYU Depth v2 and the DIML Kinect v2 datasets.

SPEED

**Distributed Any-Batch Mirror Descent** (DABMD) is based on distributed Mirror Descent but uses a fixed per-round computing time to limit the waiting by fast nodes to receive information updates from slow nodes. DABMD is characterized by varying minibatch sizes across nodes. It is applicable to a broader range of problems compared with existing distributed online optimization methods such as those based on dual averaging, and it accommodates time-varying network topology.

DABMD

**CSPResNeXt** is a convolutional neural network where we apply the Cross Stage Partial Network (CSPNet) approach to [ResNeXt](https://paperswithcode.com/method/resnext). The CSPNet partitions the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. The use of a split and merge strategy allows for more gradient flow through the network.

CSPResNeXt

CSPNet: A New Backbone that can Enhance Learning Capability of CNN

The **DELU** is a type of activation function that has trainable parameters, uses the complex linear and exponential functions in the positive dimension and uses the **[SiLU](https://paperswithcode.com/method/silu)** in the negative dimension.

$$DELU(x) = SiLU(x), x \leqslant 0$$
$$DELU(x) = (n + 0.5)x + |e^{-x} - 1|, x > 0$$

DELU

Trainable Activations for Image Classification

**Domain Adaptive Ensemble Learning**, or **DAEL**, is an architecture for domain adaptation. The model is composed of a CNN feature extractor shared across domains and multiple classifier heads each trained to specialize in a particular source domain. Each such classifier is an expert to its own domain and a non-expert to others. DAEL aims to learn these experts collaboratively so that when forming an ensemble, they can leverage complementary information from each other to be more effective for an unseen target domain. To this end, each source domain is used in turn as a pseudo-target-domain with its own expert providing supervisory signal to the ensemble of non-experts learned from the other sources. For unlabeled target data under the UDA setting where real expert does not exist, DAEL uses pseudo-label to supervise the ensemble learning.

DAEL

Domain Adaptive Ensemble Learning

Rational Activation Functions, ratio of polynomials as learnable functions

Rational Activation Function

Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

**DLA**, or **Deep Layer Aggregation**,  iteratively and hierarchically merges the feature hierarchy across layers in neural networks to make networks with better accuracy and fewer parameters. 

In iterative deep aggregation (IDA), aggregation begins at the shallowest, smallest scale and then iteratively merges deeper,
larger scales. In this way shallow features are refined as
they are propagated through different stages of aggregation.

In hierarchical deep aggregation (HDA), blocks and stages
in a tree are merged to preserve and combine feature channels. With
HDA shallower and deeper layers are combined to learn
richer combinations that span more of the feature hierarchy.
While IDA effectively combines stages, it is insufficient
for fusing the many blocks of a network, as it is still only
sequential.

Deep Layer Aggregation

A **Pyramidal Bottleneck Residual Unit** is a type of residual unit where the number of channels gradually increases as a function of the depth at which the layer occurs, which is similar to a pyramid structure of which the shape gradually widens from the top downwards. It also consists of a bottleneck using 1x1 convolutions. It was introduced as part of the [PyramidNet](https://paperswithcode.com/method/pyramidnet) architecture.

Pyramidal Bottleneck Residual Unit

mBERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

**Variational Dropout** is a regularization technique based on [dropout](https://paperswithcode.com/method/dropout), but uses a variational inference grounded approach. In Variational Dropout, we repeat the same dropout mask at each time step for both inputs, outputs, and recurrent layers (drop the same network units at each time step). This is in contrast to ordinary Dropout where different dropout masks are sampled at each time step for the inputs and outputs alone.

Variational Dropout

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

**PAUSE**, or **Positive and Annealed Unlabeled Sentence Embedding**, is an approach for learning sentence embeddings from a partially labeled dataset. It is based on a dual encoder schema that is widely adopted in supervised sentence embedding training. Each individual sample $\mathbf{x}$ contains a pair of hypothesis and premise sentences $(x\_{i},x^{\prime}_{i})$, each of which is fed into a pretrained encoder (e.g. [BERT](https://paperswithcode.com/method/bert)). As shown in Figure, the two encoders are identical during the training by sharing their weights.

PAUSE

PAUSE: Positive and Annealed Unlabeled Sentence Embedding

**FLAVR** is an architecture for video frame interpolation. It uses 3D space-time convolutions to enable end-to-end learning and inference for video frame interpolation. Overall, it consists of a [U-Net](https://paperswithcode.com/method/u-net) style architecture with 3D space-time convolutions and
deconvolutions (yellow blocks). Channel gating is used after all (de-)[convolution](https://paperswithcode.com/method/convolution) layers (blue blocks). The final prediction layer (the purple block) is implemented as a convolution layer to project the 3D feature maps into $(k−1)$ frame predictions. This design allows FLAVR to predict multiple frames in one inference forward pass.

FLAVR

FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation

**BigBiGAN** is a type of [BiGAN](https://paperswithcode.com/method/bigan) with a [BigGAN](https://paperswithcode.com/method/biggan) image generator. The authors initially used [ResNet](https://paperswithcode.com/method/resnet) as a baseline for the encoder $\mathcal{E}$ followed by a 4-layer MLP with skip connections, but they experimented with RevNets and found they outperformed with increased network width, so opted for this type of encoder for the final architecture.

BigBiGAN

Large Scale Adversarial Representation Learning

The **Lovasz-Softmax loss** is a loss function for multiclass semantic segmentation that incorporates the [softmax](https://paperswithcode.com/method/softmax) operation in the Lovasz extension. The Lovasz extension is a means by which we can achieve direct optimization of the mean intersection-over-union loss in neural networks.

Viet-Anh on Software

AI Glossary

Tech Terminology