The Legendre Memory Unit (LMU) is mathematically derived to orthogonalize
its continuous-time history – doing so by solving d coupled ordinary differential
equations (ODEs), whose phase space linearly maps onto sliding windows of
time via the Legendre polynomials up to degree d-1.  It is optimal for compressing temporal information.

See paper for equations (markdown isn't working).

Official github repo: [https://github.com/abr/lmu](https://github.com/abr/lmu)

Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks

**AutoSmart** is AutoML framework for temporal relational data. The framework includes automatic data processing, table merging, feature engineering, and model tuning, integrated with a time&memory control unit.

AutoSmart

AutoSmart: An Efficient and Automatic Machine Learning framework for Temporal Relational Data

**Attribute2Font** is a model that automatically creates fonts by synthesizing visually pleasing glyph images according to user-specified attributes and their corresponding values. Specifically, Attribute2Font is trained to perform font style transfer between any two fonts conditioned on their attribute values. After training, the model can generate glyph images in accordance with an arbitrary set of font attribute values. A unit named Attribute Attention Module is designed to make those generated glyph images better embody the prominent font attributes. A semi-supervised learning scheme is also introduced to exploit a large number of unlabeled fonts

Attribute2Font

Attribute2Font: Creating Fonts You Want From Attributes

**Colorization** is a self-supervision approach that relies on colorization as the pretext task in order to learn image representations.

Colorization

Colorful Image Colorization

A **Multiplicative RNN (mRNN)** is a type of recurrent neural network with multiplicative connections. In a standard RNN, the current input $x\_{t}$ is first transformed via the visible-to-hidden weight matrix $W\_{hx}$ and then contributes additively to the input for the current hidden state. An mRNN allows the current input (a character in the original example) to affect the hidden state dynamics by determining the entire hidden-to-hidden matrix (which defines the non-linear dynamics) in addition to providing an additive bias.

To achieve this goal, the authors modify the RNN so that its hidden-to-hidden weight matrix is a (learned) function of the current input $x\_{t}$:

$$ h\_{t} = \tanh\left(W\_{hx}x\_{t} + W\_{hh}^{\left(x\_{y}\right)}h\_{t-1} + b\_{h}\right)$$

$$ o\_{t} = W\_{oh}h\_{t} + b\_{o} $$

This is the same as the equations for a standard RNN, except that $W\_{hh}$ is replaced with $W^{(xt)}\_{hh}$. allowing each input (character) to specify a different hidden-to-hidden weight matrix.

mRNN

**UL2** is a unified framework for pretraining models that are universally effective across datasets and setups. UL2 uses Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes.

UL2: Unifying Language Learning Paradigms

**GCNII** is an extension of a [Graph Convolution Networks](https://www.paperswithcode.com/method/gcn) with two new techniques, initial residual and identify mapping, to tackle the problem of oversmoothing -- where stacking more layers and adding non-linearity tends to degrade performance. At each layer, initial residual constructs a skip connection from the input layer, while identity mapping adds an identity matrix to the weight matrix.

GCNII

Simple and Deep Graph Convolutional Networks

**Geomancer** is a nonparametric algorithm for symmetry-based disentangling of data manifolds. It learns a set of subspaces to assign to each point in the dataset, where each subspace is the tangent space of one disentangled submanifold. This means that geomancer can be used to disentangle manifolds for which there may not be a global axis-aligned coordinate system.

GEOMANCER

Disentangling by Subspace Diffusion

ARM-Net is an adaptive relation modeling network tailored for structured data, and a lightweight framework ARMOR based on ARM-Net for relational data analytics. The key idea is to model feature interactions with cross features selectively and dynamically, by first transforming the input features into exponential space, and then determining the interaction order and interaction weights adaptively for each cross feature. The authors propose a novel sparse attention mechanism to dynamically generate the interaction weights given the input tuple, so that we can explicitly model cross features of arbitrary orders with noisy features filtered selectively. Then during model inference, ARM-Net can specify the cross features being used for each prediction for higher accuracy and better interpretability.

ARM-Net

ARM-Net: Adaptive Relation Modeling Network for Structured Data

We propose a deep network architecture for the pansharpening problem called PanNet. We incorporate domain-specific knowledge to design our PanNet architecture by focusing on the two aims of the pan-sharpening problem: spectral and spatial preservation. For spectral preservation, we add up-sampled multispectral images to the network output, which directly propagates the spectral information to the reconstructed image. To preserve the spatial structure, we train our network parameters in the high-pass filtering domain rather than the image domain. We show that the trained network generalizes well to images from different satellites without needing retraining. Experiments show significant improvement over state-of-the-art methods visually and in terms of standard quality metrics.

PanNet

PanNet: A Deep Network Architecture for Pan-Sharpening

GBO is a novel metaheuristic optimization algorithm. The GBO, inspired by the gradient-based Newton’s method, uses two main operators: gradient search rule (GSR) and local escaping operator (LEO) and a set of vectors to explore the search space. The GSR employs the gradient-based method to enhance the exploration tendency and accelerate the convergence rate to achieve better positions in the search space. The LEO enables the proposed GBO to escape from local optima. The performance of the new algorithm was evaluated in two phases. 28 mathematical test functions were first used to evaluate various characteristics of the GBO, and then six engineering problems were optimized by the GBO. In the first phase, the GBO was compared with five existing optimization algorithms, indicating that the GBO yielded very promising results due to its enhanced capabilities of exploration, exploitation, convergence, and effective avoidance of local optima. The second phase also demonstrated the superior performance of the GBO in solving complex real-world engineering problems. 

* The source codes of GBO are publicly available at https://imanahmadianfar.com/codes/.

**ScaleNet**, or a **Scale Aggregation Network**, is a type of convolutional neural network which learns a neuron allocation for aggregating multi-scale information in different building blocks of a deep network. The most informative output neurons in each block are preserved while others are discarded, and thus neurons for multiple scales are competitively and adaptively allocated. The scale aggregation (SA) block concatenates feature maps at a wide range of scales. Feature maps for each scale are generated by a stack of downsampling, [convolution](https://paperswithcode.com/method/convolution) and upsampling operations.

ScaleNet

Data-Driven Neuron Allocation for Scale Aggregation Networks

**BytePS** is a distributed training method for deep neural networks. BytePS handles cases with varying number of CPU machines and makes traditional all-reduce and PS as two special cases of its framework. To further accelerate DNN training, BytePS proposes Summation Service and splits a DNN optimizer into two parts: gradient summation and parameter update. It keeps the CPU-friendly part, gradient summation, in CPUs, and moves parameter update, which is more computation heavy, to GPUs.

BytePS

**ASFF**, or **Adaptively Spatial Feature Fusion**, is a method for pyramidal feature fusion. It learns the way to spatially filter conflictive information to suppress inconsistency across different feature scales, thus improving the scale-invariance of features. 

ASFF enables the network to directly learn how to spatially filter features at other levels so that only useful information is kept for combination. For the features at a certain level, features of other levels are first integrated and resized into the same resolution and then trained to find the optimal fusion. At each spatial location, features at different levels are fused adaptively, *i.e.*, some features may be filter out as they carry contradictory information at this location and some may dominate with more discriminative clues. ASFF offers several advantages: (1) as the operation of searching the optimal fusion is differential, it can be conveniently learned in back-propagation; (2) it is agnostic to the backbone model and it is applied to single-shot detectors that have a feature pyramid structure; and (3) its implementation is simple and the increased computational cost is marginal.

Let $\mathbf{x}_{ij}^{n\rightarrow l}$ denote the feature vector at the position $(i,j)$ on the feature maps resized from level $n$ to level $l$. Following a feature resizing stage, we fuse the features at the corresponding level $l$ as follows:

$$
\mathbf{y}\_{ij}^l = \alpha^l_{ij} \cdot \mathbf{x}\_{ij}^{1\rightarrow l} + \beta^l_{ij} \cdot \mathbf{x}\_{ij}^{2\rightarrow l} +\gamma^l\_{ij} \cdot \mathbf{x}\_{ij}^{3\rightarrow l},
$$

where $\mathbf{y}\_{ij}^l$ implies the $(i,j)$-th vector of the output feature maps $\mathbf{y}^l$ among channels. $\alpha^l\_{ij}$, $\beta^l\_{ij}$ and $\gamma^l\_{ij}$ refer to the spatial importance weights for the feature maps at three different levels to level $l$, which are adaptively learned by the network. Note that $\alpha^l\_{ij}$, $\beta^l\_{ij}$ and $\gamma^l\_{ij}$ can be simple scalar variables, which are shared across all the channels. Inspired by acnet, we force $\alpha^l\_{ij}+\beta^l\_{ij}+\gamma^l\_{ij}=1$ and $\alpha^l\_{ij},\beta^l\_{ij},\gamma^l\_{ij} \in [0,1]$, and 

$$
	\alpha^l_{ij} = \frac{e^{\lambda^l\_{\alpha\_{ij}}}}{e^{\lambda^l\_{\alpha_{ij}}} + e^{\lambda^l\_{\beta_{ij}
		}} + e^{\lambda^l\_{\gamma_{ij}}}}.
$$

Here $\alpha^l\_{ij}$, $\beta^l\_{ij}$ and $\gamma^l\_{ij}$ are defined by using the [softmax](https://paperswithcode.com/method/softmax) function with $\lambda^l\_{\alpha_{ij}}$, $\lambda^l\_{\beta_{ij}}$ and $\lambda^l\_{\gamma_{ij}}$ as control parameters respectively. We use $1\times1$ [convolution](https://paperswithcode.com/method/convolution) layers to compute the weight scalar maps $\mathbf{\lambda}^l_\alpha$, $\mathbf{\lambda}^l\_\beta$ and $\mathbf{\lambda}^l\_\gamma$ from $\mathbf{x}^{1\rightarrow l}$, $\mathbf{x}^{2\rightarrow l}$ and $\mathbf{x}^{3\rightarrow l}$ respectively, and they can thus be learned through standard back-propagation.

With this method, the features at all the levels are adaptively aggregated at each scale. The outputs are used for object detection following the same pipeline of [YOLOv3](https://paperswithcode.com/method/yolov3).

ASFF

Learning Spatial Fusion for Single-Shot Object Detection

**CPVT**, or **Conditional Position Encoding Vision Transformer**, is a type of [vision transformer](https://paperswithcode.com/methods/category/vision-transformer) which utilizes [conditional positional encoding](https://paperswithcode.com/method/conditional-positional-encoding). Other than the new encodings, it follows the same architecture of [ViT](https://paperswithcode.com/method/vision-transformer) and [DeiT](https://paperswithcode.com/method/deit).

CPVT

Conditional Positional Encodings for Vision Transformers

**NoisyNet-Dueling** is a modification of a [Dueling Network](https://paperswithcode.com/method/dueling-network) that utilises noisy linear layers for exploration instead of $\epsilon$-greedy exploration as in the original Dueling formulation.

NoisyNet-Dueling

Noisy Networks for Exploration

**GShard** is a intra-layer parallel distributed method. It consists of set of simple APIs for annotations, and a compiler extension in XLA for automatic parallelization.

GShard

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

**TAPAS** is a weakly supervised question answering model that reasons over tables without generating logical forms. TAPAS predicts a minimal program by selecting a subset of the table cells and a possible aggregation operation to be executed on top of them. Consequently, TAPAS can learn operations from natural language, without the need to specify them in some formalism. This is implemented by extending [BERT](https://paperswithcode.com/method/bert)’s architecture with additional embeddings that capture tabular structure, and with two classification layers for selecting cells and predicting a corresponding aggregation operator.

Viet-Anh on Software

AI Glossary

Tech Terminology

Legendre Memory Unit

AutoSmart

Attribute2Font

Colorization

Multiplicative RNN

UL2

GCNII

Geometric Manifold Component Estimator

ARM-Net

Pansharpening Network

Gradient-based optimization

ScaleNet

BytePS

Adaptively Spatial Feature Fusion

Conditional Position Encoding Vision Transformer

NoisyNet-Dueling

GShard

TAPAS

Viet-Anh on Software