**Crossbow** is a single-server multi-GPU system for training deep learning models that enables users to freely choose their preferred batch size—however small—while scaling to multiple GPUs. Crossbow uses many parallel model replicas and avoids reduced statistical efficiency through a new synchronous training method. [SMA](https://paperswithcode.com/method/slime-mould-algorithm-sma), a synchronous variant of model averaging, is used in which replicas independently explore the solution space with gradient descent, but adjust their search synchronously based on the trajectory of a globally-consistent average model.

Crossbow

CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers

**Nesterov Accelerated Gradient** is a momentum-based [SGD](https://paperswithcode.com/method/sgd) optimizer that "looks ahead" to where the parameters will be to calculate the gradient **ex post** rather than **ex ante**:

$$ v\_{t} = \gamma{v}\_{t-1} + \eta\nabla\_{\theta}J\left(\theta-\gamma{v\_{t-1}}\right) $$
$$\theta\_{t} = \theta\_{t-1} + v\_{t}$$

Like SGD with momentum $\gamma$ is usually set to $0.9$.

The intuition is that the [standard momentum](https://paperswithcode.com/method/sgd-with-momentum) method first computes the gradient at the current location and then takes a big jump in the direction of the updated accumulated gradient. In contrast Nesterov momentum first makes a big jump in the direction of the previous accumulated gradient and then measures the gradient where it ends up and makes a correction. The idea being that it is better to correct a mistake after you have made it. 

Image Source: [Geoff Hinton lecture notes](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)

Nesterov Accelerated Gradient

A **Dutch Eligibility Trace** is a type of [eligibility trace](https://paperswithcode.com/method/eligibility-trace) where the trace increments grow less quickly than the accumulative eligibility trace (helping avoid large variance updates). For the memory vector $\textbf{e}\_{t} \in \mathbb{R}^{b} \geq \textbf{0}$:

$$\mathbf{e\_{0}} = \textbf{0}$$

$$\textbf{e}\_{t} = \gamma\lambda\textbf{e}\_{t-1} + \left(1-\alpha\gamma\lambda\textbf{e}\_{t-1}^{T}\phi\_{t}\right)\phi\_{t}$$

Dutch Eligibility Trace

**PCA Whitening** is a processing step for image based data that makes input less redundant. Adjacent pixel or feature values can be highly correlated, and whitening through the use of [PCA](https://paperswithcode.com/method/pca) reduces this degree of correlation.

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis#/media/File:GaussianScatterPCA.svg)

PCA Whitening

**CV-MIM**, or **Contrastive Cross-View Mutual Information Maximization**, is a representation learning method to disentangle pose-dependent as well as view-dependent factors from 2D human poses. The method trains a network using cross-view mutual information maximization, which maximizes mutual information of the same pose performed from different viewpoints in a contrastive learning manner. It further utilizes two regularization terms to ensure disentanglement and smoothness of the learned representations.

CV-MIM

Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization

**Spatially Separable Self-Attention**, or **SSSA**, is an [attention module](https://paperswithcode.com/methods/category/attention-modules) used in the [Twins-SVT](https://paperswithcode.com/method/twins-svt) architecture that aims to reduce the computational complexity of [vision transformers](https://paperswithcode.com/methods/category/vision-transformer) for dense prediction tasks (given high-resolution inputs). SSSA is composed of [locally-grouped self-attention](https://paperswithcode.com/method/locally-grouped-self-attention) (LSA) and [global sub-sampled attention](https://paperswithcode.com/method/global-sub-sampled-attention) (GSA).

Formally, spatially separable self-attention (SSSA) can be written as:

$$
\hat{\mathbf{z}}\_{i j}^{l}=\text { LSA }\left(\text { LayerNorm }\left(\mathbf{z}\_{i j}^{l-1}\right)\right)+\mathbf{z}\_{i j}^{l-1} $$

$$\mathbf{z}\_{i j}^{l}=\mathrm{FFN}\left(\operatorname{LayerNorm}\left(\hat{\mathbf{z}}\_{i j}^{l}\right)\right)+\hat{\mathbf{z}}\_{i j}^{l} $$

$$ \hat{\mathbf{z}}^{l+1}=\text { GSA }\left(\text { LayerNorm }\left(\mathbf{z}^{l}\right)\right)+\mathbf{z}^{l} $$

$$ \mathbf{z}^{l+1}=\text { FFN }\left(\text { LayerNorm }\left(\hat{\mathbf{z}}^{l+1}\right)\right)+\hat{\mathbf{z}}^{l+1}$$

$$i \in\{1,2, \ldots ., m\}, j \in\{1,2, \ldots ., n\}
$$

where LSA means locally-grouped self-attention within a sub-window; GSA is the global sub-sampled attention by interacting with the representative keys (generated by the sub-sampling functions) from each sub-window $\hat{\mathbf{z}}\_{i j} \in \mathcal{R}^{k\_{1} \times k\_{2} \times C} .$ Both LSA and GSA have multiple heads as in the standard self-attention.

Spatially Separable Self-Attention

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

**Sparse Switchable Normalization (SSN)** is a variant on [Switchable Normalization](https://paperswithcode.com/method/switchable-normalization) where the importance ratios are constrained to be sparse. Unlike $\ell_1$ and $\ell_0$ constraints that impose difficulties in optimization, the constrained optimization problem is turned into feed-forward computation through [SparseMax](https://paperswithcode.com/method/sparsemax), which is a sparse version of [softmax](https://paperswithcode.com/method/softmax).

Sparse Switchable Normalization

SSN: Learning Sparse Switchable Normalization via SparsestMax

A **Support Vector Machine**, or **SVM**, is a non-parametric supervised learning model. For non-linear classification and regression, they utilise the kernel trick to map inputs to high-dimensional feature spaces. SVMs construct a hyper-plane or set of hyper-planes in a high or infinite dimensional space, which can be used for classification, regression or other tasks. Intuitively, a good separation is achieved by the hyper-plane that has the largest distance to the nearest training data points of any class (so-called functional margin), since in general the larger the margin the lower the generalization error of the classifier. The figure to the right shows the decision function for a linearly separable problem, with three samples on the margin boundaries, called “support vectors”. 

Source: [scikit-learn](https://scikit-learn.org/stable/modules/svm.html)

Mixture model network (MoNet) is a general framework allowing to design convolutional deep architectures on non-Euclidean domains such as graphs and manifolds.

Image and description from: [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://arxiv.org/pdf/1611.08402.pdf)

MoNet

Geometric deep learning on graphs and manifolds using mixture model CNNs

**NesT** stacks canonical transformer layers to conduct local self-attention on every image block independently, and then "nests" them hierarchically. Coupling of processed information between spatially adjacent blocks is achieved through a proposed block aggregation between every two hierarchies. The overall hierarchical structure can be determined by two key hyper-parameters: patch size $S × S$ and number of block hierarchies $T_d$. All blocks inside each hierarchy share one set of parameters. Given input of image, each image is linearly projected to an embedding. All embeddings are partitioned to blocks and flattened to generate final input. Each transformer layers is composed of a multi-head self attention (MSA) layer followed by a feed-forward fully-connected network (FFN) with skip-connection and Layer normalization. Positional embeddings are added to encode spatial information before feeding into the block. Lastly, a nested hierarchy with block aggregation is built -- every four spatially connected blocks are merged into one.

NesT

Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding

An **Associative LSTM** combines an [LSTM](https://paperswithcode.com/method/lstm) with ideas from Holographic Reduced Representations (HRRs) to enable key-value storage of data. HRRs use a “binding” operator to implement key-value
binding between two vectors (the key and its associated content). They natively implement associative arrays; as a byproduct, they can also easily implement stacks, queues, or lists.

Associative LSTM

Associative Long Short-Term Memory

**Group Normalization** is a normalization layer that divides channels into groups and normalizes the features within each group. GN does not exploit the batch dimension, and its computation is independent of batch sizes. In the case where the group size is 1, it is equivalent to [Instance Normalization](https://paperswithcode.com/method/instance-normalization).

As motivation for the method, many classical features like SIFT and HOG had *group-wise* features and involved *group-wise normalization*. For example, a HOG vector is the outcome of several spatial cells where each cell is represented by a normalized orientation histogram.

Formally, Group Normalization is defined as:

$$ \mu\_{i} = \frac{1}{m}\sum\_{k\in\mathcal{S}\_{i}}x\_{k} $$

$$ \sigma^{2}\_{i} = \frac{1}{m}\sum\_{k\in\mathcal{S}\_{i}}\left(x\_{k}-\mu\_{i}\right)^{2} $$

$$ \hat{x}\_{i} = \frac{x\_{i} - \mu\_{i}}{\sqrt{\sigma^{2}\_{i}+\epsilon}} $$

Here $x$ is the feature computed by a layer, and $i$ is an index. Formally, a Group Norm layer computes $\mu$ and $\sigma$ in a set $\mathcal{S}\_{i}$ defined as: $\mathcal{S}\_{i} = ${$k \mid k\_{N} = i\_{N} ,\lfloor\frac{k\_{C}}{C/G}\rfloor = \lfloor\frac{I\_{C}}{C/G}\rfloor $}.

Here $G$ is the number of groups, which is a pre-defined hyper-parameter ($G = 32$ by default). $C/G$ is the number of channels per group. $\lfloor$ is the floor operation, and the final term means that the indexes $i$ and $k$ are in the same group of channels, assuming each group of channels are stored in a sequential order along the $C$ axis.

Group Normalization

**Ape-X** is a distributed architecture for deep reinforcement learning. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared [experience replay](https://paperswithcode.com/method/experience-replay) memory; the learner replays samples of experience and updates the neural network. The architecture relies on [prioritized experience replay](https://paperswithcode.com/method/prioritized-experience-replay) to focus only on the most significant data generated by the actors.

In contrast to Gorila, Ape-X uses a shared, centralized replay memory, and instead of sampling
uniformly, it prioritizes, to sample the most useful data more often. All communications are batched with the centralized replay, increasing the efficiency and throughput at the cost of some latency. 
And by learning off-policy, Ape-X has the ability to combine data from many distributed actors, by giving the different actors different exploration policies, broadening the diversity of the experience they jointly encounter.

Ape-X

Distributed Prioritized Experience Replay

**Zero Redundancy Optimizer (ZeRO)** is a sharded data parallel method for distributed training. ZeRODP removes the memory state redundancies across data-parallel processes by partitioning the model states instead of replicating them, and it retains the compute/communication efficiency by retaining the computational granularity and communication volume of DP using a dynamic communication schedule during training.

ZeRO

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

**ARShoe** is a multi-branch network for pose estimation and segmentation tackling the "try-on" problem for augmented reality shoes. Consisting of an encoder and a decoder, the multi-branch network is trained to predict keypoints [heatmap](https://paperswithcode.com/method/heatmap) (heatmap), [PAFs](https://paperswithcode.com/method/pafs) heatmap (pafmap), and segmentation results (segmap) simultaneously. Post processes are then performed for a smooth and realistic virtual try-on.

ARShoe

ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones

**Single-Shot Multi-Object Tracker** or **SMOT**, is a tracking framework that converts any single-shot detector (SSD) model into an online multiple object tracker, which emphasizes simultaneously detecting and tracking of the object paths. Contrary to the existing tracking by detection approaches which suffer from errors made by the object detectors, SMOT adopts the recently proposed scheme of tracking by re-detection.

The proposed SMOT consists of two stages. The first stage generates temporally consecutive tracklets by exploring the temporal and spatial correlations from previous frame. The second stage performs online linking of the tracklets to generate a face track for each person (better view in color).

SMOT

SMOT: Single-Shot Multi Object Tracking

The Segment Anything Model (SAM) shows remarkable segmentation ability with sparse prompts like points. However, manual prompt is not always feasible, as it may not be accessible in real-world application. In this work, we aim to eliminate the need for manual prompt.The key idea is to employ Cross-modal Chains of Thought Prompting (CCTP) to reason visual prompts using the semantic information given by a generic text prompt. We introduce a test-time adaptation per-instance mechanism called Generalizable SAM (GenSAM) to automatically generate and optimize visual prompts the generic task prompt.  CCTP maps a single generic text prompt onto image-specific consensus foreground and background heatmaps using vision-language models, acquiring reliable visual prompts. Moreover, to test-time adapt the visual prompts, we further propose Progressive Mask Generation (PMG) to iteratively reweight the input image, guiding the model to focus on the targets in a coarse-to-fine manner.Crucially, all network parameters are fixed, avoiding the need for additional training.Experiments demonstrate the superiority of GenSAM. Experiments on three benchmarks demonstrate that GenSAM outperforms point supervision approaches and achieves comparable results to scribble supervision ones, solely relying on general task descriptions as prompts.

GenSAM

Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects

In the setting of multi-target regression, base boosting permits us to incorporate prior knowledge into the learning mechanism of gradient boosting (or Newton boosting, etc.). Namely, from the vantage of statistics, base boosting is a way of building the following additive expansion in a set of elementary basis functions:
\begin{equation}
h_{j}(X ; \{ \alpha_{j}, \theta_{j} \}) = X_{j} + \sum_{k=1}^{K_{j}} \alpha_{j,k} b(X ; \theta_{j,k}),
\end{equation}
where 
$X$ is an example from the domain $\mathcal{X},$
$\{\alpha_{j}, \theta_{j}\} = \{\alpha_{j,1},\dots, \alpha_{j,K_{j}},\theta_{j,1},\dots,\theta_{j,K_{j}}\}$ collects the expansion coefficients and parameter sets,
$X_{j}$ is the image of $X$ under the $j$th coordinate function (a prediction from a user-specified model),
$K_{j}$ is the number of basis functions in the linear sum,
$b(X; \theta_{j,k})$ is a real-valued function of the example $X,$ characterized by a parameter set $\theta_{j,k}.$

The aforementioned additive expansion differs from the 
[standard  additive expansion](https://projecteuclid.org/download/pdf_1/euclid.aos/1013203451):
\begin{equation}
h_{j}(X ; \{ \alpha_{j}, \theta_{j}\}) = \alpha_{j, 0} + \sum_{k=1}^{K_{j}} \alpha_{j,k} b(X ; \theta_{j,k}),
\end{equation}
as it replaces the constant offset value $\alpha_{j, 0}$ with a prediction from a user-specified model. In essence, this modification permits us to incorporate prior knowledge into the for loop of gradient boosting, as the for loop proceeds to build the linear sum by computing residuals that depend upon predictions from the user-specified model instead of the optimal constant model: $\mbox{argmin} \sum_{i=1}^{m_{train}} \ell_{j}(Y_{j}^{(i)}, c),$ where $m_{train}$ denotes the number of training examples, $\ell_{j}$ denotes a single-target loss function, and $c \in \mathbb{R}$ denotes a real number, e.g, $\mbox{argmin} \sum_{i=1}^{m_{train}} (Y_{j}^{(i)} - c)^{2} = \frac{\sum_{i=1}^{m_{train}} Y_{j}^{(i)}}{m_{train}}.$

Viet-Anh on Software

AI Glossary

Tech Terminology

Crossbow

Nesterov Accelerated Gradient

Dutch Eligibility Trace

PCA Whitening

Contrastive Cross-View Mutual Information Maximization

Spatially Separable Self-Attention

Sparse Switchable Normalization

Support Vector Machine

Mixture model network