**Fast Minimum-Norm Attack**, or **FNM**, is a type of adversarial attack that works with different $\ell_{p}$-norm perturbation models ($p=0,1,2,\infty$), is robust to hyperparameter choices, does not require adversarial starting points, and converges within few lightweight steps. It works by iteratively finding the sample misclassified with maximum confidence within an $\ell_{p}$-norm constraint of size $\epsilon$, while adapting $\epsilon$ to minimize the distance of the current sample to the decision boundary.

Fast Minimum-Norm Attack

Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraints

**Mesh-TensorFlow** is a language for specifying a general class of distributed tensor computations. Where data-parallelism can be viewed as splitting tensors and operations along the "batch" dimension, in Mesh-TensorFlow, the user can specify any tensor dimensions to be split across any dimensions of a multi-dimensional mesh of processors. A MeshTensorFlow graph compiles into a SPMD program consisting of parallel operations coupled with collective communication primitives such as Allreduce.

Mesh-TensorFlow

Mesh-TensorFlow: Deep Learning for Supercomputers

**Soft Split and Soft Composition** are video frame based operations used in the [FuseFormer](https://paperswithcode.com/method/fuseformer) architecture, specifically the [FuseFormer blocks](https://paperswithcode.com/method/fuseformer-block). We softly split each frame into overlapped patches and then softly composite them back, by using an unfold and fold operator with patch size $k$ being greater than patch stride $s$. When compositing patches back to its original spatial shape, we add up feature values at each overlapping spatial location of neighboring patches.

Soft Split and Soft Composition

FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting

**ResNeXt-Elastic** is a convolutional neural network that is a modification of a [ResNeXt](https://paperswithcode.com/method/resnext) with elastic blocks (extra upsampling and downsampling).

ResNeXt-Elastic

ELASTIC: Improving CNNs with Dynamic Scaling Policies

**Fixed Factorized Attention** is a factorized attention pattern where specific cells summarize previous locations and propagate that information to all future cells. It was proposed as part of the [Sparse Transformer](https://paperswithcode.com/method/sparse-transformer) architecture.


A self-attention layer maps a matrix of input embeddings $X$ to an output matrix and is parameterized by a connectivity pattern $S = \text{set}\left(S\_{1}, \dots, S\_{n}\right)$, where $S\_{i}$ denotes the set of indices of the input vectors to which the $i$th output vector attends. The output vector is a weighted sum of transformations of the input vectors:

$$ \text{Attend}\left(X, S\right) = \left(a\left(\mathbf{x}\_{i}, S\_{i}\right)\right)\_{i\in\text{set}\left(1,\dots,n\right)}$$

$$ a\left(\mathbf{x}\_{i}, S\_{i}\right) = \text{softmax}\left(\frac{\left(W\_{q}\mathbf{x}\_{i}\right)K^{T}\_{S\_{i}}}{\sqrt{d}}\right)V\_{S\_{i}} $$

$$ K\_{Si} = \left(W\_{k}\mathbf{x}\_{j}\right)\_{j\in{S\_{i}}} $$

$$ V\_{Si} = \left(W\_{v}\mathbf{x}\_{j}\right)\_{j\in{S\_{i}}} $$

Here $W\_{q}$, $W\_{k}$, and $W\_{v}$ represent the weight matrices which transform a given $x\_{i}$ into a query, key, or value, and $d$ is the inner dimension of the queries and keys. The output at each position is a sum of the values weighted by the scaled dot-product similarity of the keys and queries.

Full self-attention for autoregressive models defines $S\_{i} = \text{set}\left(j : j \leq i\right)$, allowing every element to attend to all previous positions and its own position.

Factorized self-attention instead has $p$ separate attention heads, where the $m$th head defines a subset of the indices $A\_{i}^{(m)} ⊂ \text{set}\left(j : j \leq i\right)$ and lets $S\_{i} = A\_{i}^{(m)}$. The goal with the Sparse [Transformer](https://paperswithcode.com/method/transformer) was to find efficient choices for the subset $A$.

Formally for Fixed Factorized Attention, $A^{(1)}\_{i} = ${$j : \left(\lfloor{j/l\rfloor}=\lfloor{i/l\rfloor}\right)$}, where the brackets denote the floor operation, and $A^{(2)}\_{i} = ${$j : j \mod l \in ${$t, t+1, \ldots, l$}}, where $t=l-c$ and $c$ is a hyperparameter. The $i$-th output vector of the attention head attends to all input vectors either from $A^{(1)}\_{i}$ or $A^{(2)}\_{i}$. This pattern can be visualized in the figure to the right.

If the stride is 128 and $c = 8$, then all future positions greater than 128 can attend to positions 120-128, all positions greater than 256 can attend to 248-256, and so forth. 

A fixed-attention pattern with $c = 1$ limits the expressivity of the network significantly, as many representations in the network are only used for one block whereas a small number of locations are used by all blocks. The authors found choosing $c \in ${$8, 16, 32$} for typical values of $l \in
{128, 256}$ performs well, although this increases the computational cost of this method by $c$ in comparison to the [strided attention](https://paperswithcode.com/method/strided-attention).

Additionally, the authors found that when using multiple heads, having them attend to distinct subblocks of length $c$ within the block of size $l$ was preferable to having them attend to the same subblock.

Fixed Factorized Attention

Generating Long Sequences with Sparse Transformers

**Minibatch Discrimination** is a discriminative technique for generative adversarial networks where we discriminate between whole minibatches of samples rather than between individual samples. This is intended to avoid collapse of the generator.

Minibatch Discrimination

Improved Techniques for Training GANs

**GloVe Embeddings** are a type of word embedding that encode the co-occurrence probability ratio between two words as vector differences. GloVe uses a weighted least squares objective $J$ that minimizes the difference between the dot product of the vectors of two words and the logarithm of their number of co-occurrences:

$$ J=\sum\_{i, j=1}^{V}f\left(𝑋\_{i j}\right)(w^{T}\_{i}\tilde{w}_{j} + b\_{i} + \tilde{b}\_{j} - \log{𝑋}\_{ij})^{2} $$

where $w\_{i}$ and $b\_{i}$ are the word vector and bias respectively of word $i$, $\tilde{w}_{j}$ and $b\_{j}$ are the context word vector and bias respectively of word $j$, $X\_{ij}$ is the number of times word $i$ occurs in the context of word $j$, and $f$ is a weighting function that assigns lower weights to rare and frequent co-occurrences.

GloVe

GloVe: Global Vectors for Word Representation

**COCO-FUNIT** is few-shot image translation model which computes the style embedding of the example images conditioned on the input image and a new module called the constant style bias. It builds on top of [FUNIT](https://arxiv.org/abs/1905.01723) by identifying the content loss problem and then addressing it with a novel content-conditioned style encoder architecture.

The FUNIT method suffers from the content loss problem—the translation result is not well-aligned with the input image. While a direct theoretical analysis is likely elusive, we conduct an empirical study, aiming at identify the cause of the content loss problem. In analyses, the authors show that the FUNIT style encoder produces very different style codes using different crops -- suggesting the style code contains other information about the style image such as the object pose.

To make the style embedding more robust to small variations in the style image, a new style encoder architecture, the Content-Conditioned style encoder (COCO), is introduced. The most distinctive feature of this new encoder is the conditioning in the content image as illustrated in the top-right of the Figure. Unlike the style encoder in FUNIT, COCO takes both content and style image as input. With this content-conditioning scheme, a direct feedback path is created during learning to let the content image influence how the style code is computed. It also helps reduce the direct influence of the style image to the extract style code.

COCO-FUNIT

COCO-FUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder

CARLA is an open-source simulator for autonomous driving research. CARLA has been developed from the ground up to support development, training, and validation of autonomous urban driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely. 

Source: [Dosovitskiy et al.](https://arxiv.org/pdf/1711.03938v1.pdf)

Image source: [Dosovitskiy et al.](https://arxiv.org/pdf/1711.03938v1.pdf)

CARLA

CARLA: An Open Urban Driving Simulator

**Softplus** is an activation function $f\left(x\right) = \log\left(1+\exp\left(x\right)\right)$. It can be viewed as a smooth version of [ReLU](https://paperswithcode.com/method/relu).

Softplus

**CBNet** is a backbone architecture that consists of multiple identical backbones (specially called Assistant Backbones and Lead Backbone) and composite connections between neighbor backbones. From left to right, the output of each stage in an Assistant Backbone, namely higher-level
features, flows to the parallel stage of the succeeding backbone as part of inputs through composite connections. Finally, the feature maps of the last backbone named Lead
Backbone are used for object detection. The features extracted by CBNet for object detection fuse the high-level and low-level features of multiple backbones, hence improve the detection performance.

CBNet

CBNet: A Novel Composite Backbone Network Architecture for Object Detection

**Self-Supervised Temporal Domain Adaptation (SSTDA)** is a method for action segmentation with self-supervised temporal domain adaptation. It contains two self-supervised auxiliary tasks (binary and sequential domain prediction) to jointly align cross-domain feature spaces embedded with local and global temporal dynamics.

SSTDA

Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation

A **Deep Boltzmann Machine (DBM)** is a three-layer generative model. It is similar to a [Deep Belief Network](https://paperswithcode.com/method/deep-belief-network), but instead allows bidirectional connections in the bottom layers. Its energy function is  as an extension of the energy function of the RBM:

$$ E\left(v, h\right) = -\sum^{i}\_{i}v\_{i}b\_{i} - \sum^{N}\_{n=1}\sum_{k}h\_{n,k}b\_{n,k}-\sum\_{i, k}v\_{i}w\_{ik}h\_{k} - \sum^{N-1}\_{n=1}\sum\_{k,l}h\_{n,k}w\_{n, k, l}h\_{n+1, l}$$

for a DBM with $N$ hidden layers.

Source: [On the Origin of Deep Learning](https://arxiv.org/pdf/1702.07800.pdf)

Deep Boltzmann Machine

**Co-Scale Conv-Attentional Image Transformer** (CoaT) is a [Transformer](https://paperswithcode.com/method/transformer)-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers' encoder branches at individual scales, while allowing representations learned at different scales to effectively communicate with each other. Second, the conv-attentional mechanism is designed by realizing a relative position embedding formulation in the factorized attention module with an efficient [convolution](https://paperswithcode.com/method/convolution)-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities.

CoaT

Co-Scale Conv-Attentional Image Transformers

SRM combines style transfer with an attention mechanism. Its main contribution is style pooling which utilizes both mean and standard deviation of the input features to improve its capability to capture global information. It also adopts a lightweight channel-wise fully-connected (CFC) layer, in place of the original fully-connected layer, to reduce the computational requirements.
Given an input feature map $X \in \mathbb{R}^{C \times H \times W}$, SRM first collects global information by using style pooling ($\text{SP}(\cdot)$) which combines global average pooling and global standard deviation pooling. 
Then a channel-wise fully connected ($\text{CFC}(\cdot)$) layer (i.e. fully connected per channel), batch normalization $\text{BN}$ and sigmoid function $\sigma$ are used  to provide the attention vector. Finally,   as in an SE block, the input features are multiplied by the attention vector. Overall, an SRM can be written as:
\begin{align}
    s = F_\text{srm}(X, \theta) & = \sigma (\text{BN}(\text{CFC}(\text{SP}(X))))
\end{align}
\begin{align}
    Y & = s  X
\end{align}
The SRM block improves both squeeze and excitation modules, yet can be added after each residual unit like an SE block.

SRM: A Style-Based Recalibration Module for Convolutional Neural Networks

Object Dropout is a technique that perturbs object features in an image for [noisy student](https://paperswithcode.com/method/noisy-student) training. It performs at par with standard data augmentation techniques while being significantly faster than the latter to implement.

Object Dropout

Perturb, Predict & Paraphrase: Semi-Supervised Learning using Noisy Student for Image Captioning

**Linear Regression** is a method for modelling a relationship between a dependent variable and independent variables. These models can be fit with numerous approaches. The most common is *least squares*, where we minimize the mean square error between the predicted values $\hat{y} = \textbf{X}\hat{\beta}$ and actual values $y$: $\left(y-\textbf{X}\beta\right)^{2}$.

We can also define the problem in probabilistic terms as a generalized linear model (GLM) where the pdf is a Gaussian distribution, and then perform maximum likelihood estimation to estimate $\hat{\beta}$.

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Linear_regression)

Linear Regression

**Stand-Alone Self Attention** (SASA) replaces all instances of spatial [convolution](https://paperswithcode.com/method/convolution) with a form of self-attention applied to [ResNet](https://paperswithcode.com/method/resnet) producing a fully, stand-alone self-attentional model.

Viet-Anh on Software

AI Glossary

Tech Terminology

Fast Minimum-Norm Attack

Mesh-TensorFlow

Soft Split and Soft Composition

ResNeXt-Elastic

Fixed Factorized Attention

Minibatch Discrimination

GloVe Embeddings

COCO-FUNIT

CARLA: An Open Urban Driving Simulator

Softplus

Composite Backbone Network

Self-Supervised Temporal Domain Adaptation

Deep Boltzmann Machine

Co-Scale Conv-attentional Image Transformer

style-based recalibration module

Object Dropout

Linear Regression

Stand-Alone Self Attention

Viet-Anh on Software