XGPT is a method of cross-modal generative pre-training for image captioning designed to pre-train text-to-image caption generators through three novel generation tasks, including image-conditioned masked language modeling (IMLM), image-conditioned denoising autoencoding (IDA), and text-conditioned image feature generation (TIGF). The pre-trained XGPT can be fine-tuned without any task-specific architecture modifications and build strong image captioning models.

XGPT

XGPT: Cross-modal Generative Pre-Training for Image Captioning

**Serf**, or **Log-Softplus ERror activation Function**, is a type of activation function which is self-regularized and nonmonotonic in nature. It belongs to the [Swish](https://paperswithcode.com/method/swish) family of functions. Serf is defined as:

$$f\left(x\right) = x\text{erf}\left(\ln\left(1 + e^{x}\right)\right)$$

Serf

SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function

A **Pyramid Pooling Module** is a module for semantic segmentation which acts as an effective global contextual prior. The motivation is that the problem of using a convolutional network like a [ResNet](https://paperswithcode.com/method/resnet) is that, while the receptive field is already larger than the input image, the empirical receptive field is much smaller than the theoretical one especially on high-level layers. This makes many networks not sufficiently incorporate the momentous global scenery prior. 

The PPM is an effective global prior representation that addresses this problem. It contains information with different scales and varying among different sub-regions. Using our 4-level pyramid, the pooling kernels cover the whole, half of, and small portions of the image. They are fused as the global prior. Then we concatenate the prior with the original feature map in the final part.

Pyramid Pooling Module

Pyramid Scene Parsing Network

A wavelet **scattering transform** computes a translation invariant representation, which is stable to deformation, using a deep [convolution](https://paperswithcode.com/method/convolution) network architecture. It computes non-linear invariants with modulus and averaging pooling functions. It helps to eliminate the image variability due to translation and is stable to deformations. 

Image source: [Bruna and Mallat](https://arxiv.org/pdf/1203.1513v2.pdf)

ScatNet

Invariant Scattering Convolution Networks

**SqueezeNet** is a convolutional neural network that employs design strategies to reduce the number of parameters, notably with the use of fire modules that "squeeze" parameters using 1x1 convolutions.

SqueezeNet

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

**GHM-R** is a loss function designed to balance the gradient flow for bounding box refinement. The GHM first performs statistics on the number of examples with similar attributes w.r.t their gradient density and then attaches a harmonizing parameter to the gradient of each example according to the density. The modification of gradient can be equivalently implemented by reformulating the loss function. Embedding the GHM into the bounding box regression branch is denoted as GHM-R loss.

GHM-R

Gradient Harmonized Single-stage Detector

Neural message-passing algorithms for semi-supervised classification on graphs have recently achieved great success. However, for classifying a node these methods only consider nodes that are a few propagation steps away and the size of this utilized neighbourhood is hard to extend. This paper uses the relationship between graph convolutional networks (GCN) and PageRank to derive an improved propagation scheme based on personalized PageRank. We utilize this propagation procedure to construct a simple model, personalized propagation of neural predictions (PPNP), and its fast approximation, APPNP. Our model's training time is on par or faster and its number of parameters is on par or lower than previous models. It leverages a large, adjustable neighbourhood for classification and can be easily combined with any neural network. We show that this model outperforms several recently proposed methods for semi-supervised classification in the most thorough study done so far for GCN-like models.

APPNP

Predict then Propagate: Graph Neural Networks meet Personalized PageRank

**Stochastic Gradient Descent** is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using all available data. That is for weights $w$ and a loss function $L$ we have:

$$ w\_{t+1} = w\_{t} - \eta\hat{\nabla}\_{w}{L(w\_{t})} $$

Where $\eta$ is a learning rate. SGD reduces redundancy compared to batch gradient descent - which recomputes gradients for similar examples before each parameter update - so it is usually much faster.

(Image Source: [here](http://rasbt.github.io/mlxtend/user_guide/general_concepts/gradient-optimization/))

**Global-Local Attention** is a type of attention mechanism used in the [ETC](https://paperswithcode.com/method/etc) architecture. ETC receives two separate input sequences: the global input $x^{g} = (x^{g}\_{1}, \dots, x^{g}\_{n\_{g}})$ and the long input $x^{l} = (x^{l}\_{1}, \dots x^{l}\_{n\_{l}})$. Typically, the long input contains the input a [standard Transformer](https://paperswithcode.com/method/transformer) would receive, while the global input contains a much smaller number of auxiliary tokens ($n\_{g}  \ll n\_{l}$). Attention is then split into four separate pieces: global-to-global (g2g), global-tolong (g2l), long-to-global (l2g), and long-to-long (l2l). Attention in the l2l piece (the most computationally expensive piece) is restricted to a fixed radius $r \ll n\_{l}$. To compensate for this limited attention span, the tokens in the global input have unrestricted attention, and thus long input tokens can transfer information to each other through global input tokens. Accordingly, g2g, g2l, and l2g pieces of attention are unrestricted.

Global-Local Attention

ETC: Encoding Long and Structured Inputs in Transformers

ArXiving Before Submission Helps Everyone

**$\epsilon$-Greedy Exploration** is an exploration strategy in reinforcement learning that takes an exploratory action with probability $\epsilon$ and a greedy action with probability $1-\epsilon$. It tackles the exploration-exploitation tradeoff with reinforcement learning algorithms: the desire to explore the state space with the desire to seek an optimal policy. Despite its simplicity, it is still commonly used as an behaviour policy $\pi$ in several state-of-the-art reinforcement learning models.

Image Credit: [Robin van Embden](https://cran.r-project.org/web/packages/contextual/vignettes/sutton_barto.html)

Epsilon Greedy Exploration

**Cross-Scale Non-Local Attention**, or **CS-NL**,  is a non-local attention module for image super-resolution deep networks. It learns to mine long-range dependencies between LR features to larger-scale HR patches within the same feature map. Specifically, suppose we are conducting an s-scale super-resolution with the module, given a feature map $X$ of spatial size $(W, H)$, we first bilinearly downsample it to $Y$ with scale $s$, and match the $p\times p$ patches in $X$ with the downsampled $p \times p$ candidates in $Y$ to obtain the [softmax](https://paperswithcode.com/method/softmax) matching score. Finally, we conduct deconvolution.on the score by weighted adding the patches of size $\left(sp, sp\right)$ extracted from $X$. The obtained $Z$ of size $(sW, sH)$ will be $s$ times super-resolved than $X$.

Cross-Scale Non-Local Attention

Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining

**Audiovisual SlowFast Network**, or **AVSlowFast**, is an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are integrated with a Faster Audio pathway to model vision and sound in a unified representation. Audio and visual features are fused at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, [DropPathway](https://paperswithcode.com/method/droppathway) is used, which randomly drops the Audio pathway during training as an effective regularization technique. Inspired by prior studies in neuroscience, hierarchical audiovisual synchronization is performed to learn joint audiovisual features.

AVSlowFast

Audiovisual SlowFast Networks for Video Recognition

**Hybrid Firefly and Particle Swarm Optimization (HFPSO)** is a metaheuristic optimization algorithm that combines strong points of firefly and particle swarm optimization. HFPSO tries to determine the start of the local search process properly by checking the previous global best fitness values.

[Click Here for the Paper](https://www.sciencedirect.com/science/article/abs/pii/S156849461830084X)

[Codes (MATLAB)](https://www.mathworks.com/matlabcentral/fileexchange/67768-a-hybrid-firefly-and-particle-swarm-optimization-hfpso)

HFPSO

**I-BERT** is a quantized version of [BERT](https://paperswithcode.com/method/bert) that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer only approximation methods for nonlinear operations, e.g., [GELU](https://paperswithcode.com/method/gelu), [Softmax](https://paperswithcode.com/method/softmax), and [Layer Normalization](https://paperswithcode.com/method/layer-normalization), it performs an end-to-end integer-only [BERT](https://paperswithcode.com/method/bert) inference without any floating point calculation.

In particular, GELU and Softmax are approximated with lightweight second-order polynomials, which can be evaluated with integer-only arithmetic. For LayerNorm, integer-only computation is performed by leveraging a known algorithm for integer calculation of
square root.

I-BERT

I-BERT: Integer-only BERT Quantization

**DropPathway** randomly drops an audio pathway during training as a regularization technique for audiovisual recognition models.  Specifically, at each training iteration, we drop the Audio pathway altogether with probability $P\_{d}$. This way, we slow down the learning of the Audio pathway and make its learning dynamics more compatible with its visual counterpart. When dropping the audio pathway, we sum zero tensors with the visual pathways.

Note that DropPathway is different from simply setting different learning rates for the audio/visual pathways in that it 1) ensures the audio pathway has fewer parameter updates, 2) hinders the visual pathway to 'shortcut' training by memorizing audio information, and 3) provides extra regularization as different audio clips are dropped in each epoch.

DropPathway

**Hi-LANDER** is a hierarchical [graph neural network](https://paperswithcode.com/methods/category/graph-models) (GNN) model that learns how to cluster a set of images into an unknown number of identities using an image annotated with labels belonging to a disjoint set of identities. The hierarchical GNN uses an approach to merge connected components predicted at each level of the hierarchy to form a new graph at the next level. Unlike fully unsupervised hierarchical clustering, the choice of grouping and complexity criteria stems naturally from supervision in the training set.

Hi-LANDER

Learning Hierarchical Graph Neural Networks for Image Clustering

A **Groupwise Point Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) where we apply a [point convolution](https://paperswithcode.com/method/pointwise-convolution) groupwise (using different set of convolution filter groups).

Image Credit: [Chi-Feng Wang](https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728)

AI Glossary

XGPT

Serf

Pyramid Pooling Module

Scattering Transform

SqueezeNet

Gradient Harmonizing Mechanism R

Approximation of Personalized Propagation of Neural Predictions

Stochastic Gradient Descent

Global-Local Attention

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Epsilon Greedy Exploration

Cross-Scale Non-Local Attention

Audiovisual SlowFast Network

Hybrid Firefly and Particle Swarm Optimization

I-BERT

DropPathway

Hi-LANDER

Groupwise Point Convolution