In this paper, we aim to obtain improved attention for a visual question answering (VQA) task. It is challenging to provide supervision for attention. An observation we make is that visual explanations as obtained through class activation mappings (specifically Grad-[CAM](https://paperswithcode.com/method/cam)) that are meant to explain the performance of various networks could form a means of supervision. However, as the distributions of attention maps and that of Grad-CAMs differ, it would not be suitable to directly use these as a form of supervision. Rather, we propose the use of a discriminator that aims to distinguish samples of visual explanation and attention maps. The use of adversarial training of the attention regions as a two-player game between attention and explanation serves to bring the distributions of attention maps and visual explanations closer. Significantly, we observe that providing such a means of supervision also results in attention maps that are more closely related to human attention resulting in a substantial improvement over baseline stacked attention network (SAN) models. It also results in a good improvement in rank correlation metric on the VQA task. This method can also be combined with recent MCB based methods and results in consistent improvement. We also provide comparisons with other means for learning distributions such as based on Correlation Alignment (Coral), Maximum Mean Discrepancy (MMD) and Mean Square Error (MSE) losses and observe that the adversarial loss outperforms the other forms of learning the attention maps. Visualization of the results also confirms our hypothesis that attention maps improve using this form of supervision.

Explanation vs Attention

**Instance Normalization** (also known as contrast normalization) is a normalization layer where:

$$
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - \mu_{ti})^2.
$$

This prevents instance-specific mean and covariance shift simplifying the learning process. Intuitively, the normalization process allows to remove instance-specific contrast information from the content image in a task like image stylization, which simplifies generation.

Instance Normalization

Instance Normalization: The Missing Ingredient for Fast Stylization

**Depthwise Convolution** is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D [convolution](https://paperswithcode.com/method/convolution) performed over multiple input channels, the filter is as deep as the input and lets us freely mix channels to generate each element in the output. In contrast, depthwise convolutions keep each channel separate. To summarize the steps, we:

1. Split the input and filter into channels.
2. We convolve each input with the respective filter.
3. We stack the convolved outputs together.

Image Credit: [Chi-Feng Wang](https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728)

Depthwise Convolution

**Faster R-CNN** is an object detection model that improves on [Fast R-CNN](https://paperswithcode.com/method/fast-r-cnn) by utilising a region proposal network ([RPN](https://paperswithcode.com/method/rpn)) with the CNN model. The RPN shares full-image convolutional features with the detection network, enabling nearly cost-free region proposals. It is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by [Fast R-CNN](https://paperswithcode.com/method/fast-r-cnn) for detection. RPN and Fast [R-CNN](https://paperswithcode.com/method/r-cnn) are merged into a single network by sharing their convolutional features: the RPN component tells the unified network where to look.

As a whole, Faster R-CNN consists of two modules. The first module is a deep fully convolutional network that proposes regions, and the second module is the Fast R-CNN detector that uses the proposed regions.

Faster R-CNN

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

**BlendMask** is an [instance segmentation framework](https://paperswithcode.com/methods/category/instance-segmentation-models) built on top of the[ FCOS](https://paperswithcode.com/method/fcos) object detector. The bottom module uses either backbone or [FPN](https://paperswithcode.com/method/fpn) features to predict a set of bases. A single [convolution](https://paperswithcode.com/methods/category/convolutions) layer is added on top of the detection towers to produce attention masks along with each bounding box prediction. For each predicted instance, the [blender](https://paperswithcode.com/method/blender) crops the bases with its bounding box and linearly combine them according the learned attention maps. Note that the Bottom Module can take features either from ‘C’, or ‘P’ as the input.

BlendMask

BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation

In a **Replacing Eligibility Trace**, each time the state is revisited, the trace is reset to $1$ regardless of the presence of a prior trace.. For the memory vector $\textbf{e}\_{t} \in \mathbb{R}^{b} \geq \textbf{0}$:

$$\mathbf{e\_{0}} = \textbf{0}$$

$$\textbf{e}\_{t} = \gamma\lambda{e}\_{t-1}\left(s\right) \text{ if } s \neq s\_{t}$$

$$\textbf{e}\_{t} = 1 \text{ if } s = s\_{t}$$

They can be seen as crude approximation to dutch traces, which have largely superseded them as they perform better than replacing traces and have a clearer theoretical basis. Accumulating traces remain of interest for nonlinear function approximations where dutch traces are not available.

Source: Sutton and Barto, Reinforcement Learning, 2nd Edition

Replacing Eligibility Trace

**mt5** is a multilingual variant of [T5](https://paperswithcode.com/method/t5) that was pre-trained on a new Common Crawl-based dataset covering $101$ languages.

mT5: A massively multilingual pre-trained text-to-text transformer

**k-Sparse Autoencoders** are autoencoders with linear activation function, where in hidden layers only the $k$ highest activities are kept. This achieves exact sparsity in the hidden representation. Backpropagation only goes through the the top $k$ activated units. This can be achieved with a [ReLU](https://paperswithcode.com/method/relu) layer with an adjustable threshold.

k-Sparse Autoencoder

k-Sparse Autoencoders

**Inception-v4** is a convolutional neural network architecture that builds on previous iterations of the Inception family by simplifying the architecture and using more inception modules than [Inception-v3](https://paperswithcode.com/method/inception-v3).

Inception-v4

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

An **Elastic ResNeXt Block** is a modification of the [ResNeXt Block](https://paperswithcode.com/method/resnext-block) that adds downsamplings and upsamplings in parallel branches at each layer. It is called "elastic" because each layer in the network is flexible in terms of choosing the best scale by a soft policy.

Elastic ResNeXt Block

ELASTIC: Improving CNNs with Dynamic Scaling Policies

**DALL·E 2** is a generative text-to-image model made up of two main components: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding.

DALL·E 2

Hierarchical Text-Conditional Image Generation with CLIP Latents

**A2C**, or **Advantage Actor Critic**, is a synchronous version of the [A3C](https://paperswithcode.com/method/a3c) policy gradient method. As an alternative to the asynchronous implementation of A3C, A2C is a synchronous, deterministic implementation that waits for each actor to finish its segment of experience before updating, averaging over all of the actors. This more effectively uses GPUs due to larger batch sizes.

Image Credit: [OpenAI Baselines](https://openai.com/blog/baselines-acktr-a2c/)

Asynchronous Methods for Deep Reinforcement Learning

**CoVe**, or **Contextualized Word Vectors**, uses a deep [LSTM](https://paperswithcode.com/method/lstm) encoder from an attentional sequence-to-sequence model trained for machine translation to contextualize word vectors. $\text{CoVe}$ word embeddings are therefore a function of the entire input sequence. These word embeddings can then be used in downstream tasks by concatenating them with $\text{GloVe}$ embeddings:

$$ v = \left[\text{GloVe}\left(x\right), \text{CoVe}\left(x\right)\right]$$

and then feeding these in as features for the task-specific models.

CoVe

Learned in Translation: Contextualized Word Vectors

**Scaled dot-product attention** is an attention mechanism where the dot products are scaled down by $\sqrt{d_k}$. Formally we have a query $Q$, a key $K$ and a value $V$ and calculate the attention as:

$$ {\text{Attention}}(Q, K, V) = \text{softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V $$

If we assume that $q$ and $k$ are $d_k$-dimensional vectors whose components are independent random variables with mean $0$ and variance $1$, then their dot product, $q \cdot k = \sum_{i=1}^{d_k} u_iv_i$, has mean $0$ and variance $d_k$.  Since we would prefer these values to have variance $1$, we divide by $\sqrt{d_k}$.

Scaled Dot-Product Attention

Attention Is All You Need

**Multi-DConv-Head Attention**, or **MDHA**, is a type of [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention) that utilizes [depthwise convolutions](https://paperswithcode.com/method/depthwise-convolution) after the multi-head projections. It is used in the [Primer](https://paperswithcode.com/method/primer) [Transformer](https://paperswithcode.com/method/transformer) architecture.

Specifically, 3x1 depthwise convolutions are added after each of the multi-head projections for query $Q$, key $K$ and value $V$ in self-attention. These depthwise convolutions are performed over the spatial dimension of each dense projection’s output. Interestingly, this ordering of pointwise followed by depthwise convolution is the reverse of typical [separable convolution](https://paperswithcode.com/method/depthwise-separable-convolution), which the authors find to be less effective. They also find that wider depthwise convolution and [standard convolution](https://paperswithcode.com/method/convolution) not only do not improve performance, but in several cases hurt it. 

MDHA is similar to [Convolutional Attention](https://paperswithcode.com/method/cvt), which uses [separable convolution](https://paperswithcode.com/method/depthwise-separable-convolution) instead of depthwise convolution and does not apply convolution operations per attention head as in MDHA.

Multi-DConv-Head Attention

Primer: Searching for Efficient Transformers for Language Modeling

**GridMask** is a data augmentation method that randomly removes some pixels of an input image. Unlike other methods, the region that the algorithm removes is neither a continuous region nor random pixels in dropout. Instead, the algorithm removes a region with disconnected pixel sets, as shown in the Figure.

We express the setting as

$$
\tilde{\mathbf{x}}=\mathbf{x} \times M
$$

where $\mathbf{x} \in R^{H \times W \times C}$ represents the input image, $M \in$ $\{0,1\}^{H \times W}$ is the binary mask that stores pixels to be removed, and $\tilde{\mathbf{x}} \in R^{H \times W \times C}$ is the result produced by the algorithm. For the binary mask $M$, if $M_{i, j}=1$ we keep pixel $(i, j)$ in the input image; otherwise we remove it. GridMask is applied after the image normalization operation.

The shape of $M$ looks like a grid, as shown in the Figure . Four numbers $\left(r, d, \delta_{x}, \delta_{y}\right)$ are used to represent a unique $M$. Every mask is formed by tiling the units. $r$ is the ratio of the shorter gray edge in a unit. $d$ is the length of one unit. $\delta\_{x}$ and $\delta\_{y}$ are the distances between the first intact unit and boundary of the image.

GridMask

GridMask Data Augmentation

Contextual Graph Markov Model (CGMM) is an approach combining ideas from generative models and neural networks for the processing of graph data. It founds on a constructive methodology to build a deep architecture comprising layers of probabilistic models that learn to encode the structured information in an incremental fashion. Context is diffused in an efficient and scalable way across the graph vertexes and edges. The resulting graph encoding is used in combination with discriminative models to address structure classification benchmarks.

Description and image from: [Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing](https://arxiv.org/pdf/1805.10636.pdf)

CGMM

Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing

An **EESP Unit**, or  Extremely Efficient Spatial Pyramid of Depth-wise Dilated Separable Convolutions, is an image model block designed for edge devices. It was proposed as part of the [ESPNetv2](https://paperswithcode.com/method/espnetv2) CNN architecture. 

This building block is based on a reduce-split-transform-merge strategy. The EESP unit first projects the high-dimensional input feature maps into low-dimensional space using groupwise pointwise convolutions and then learns the representations in parallel using depthwise dilated separable convolutions with different dilation rates. Different dilation rates in each branch allow the EESP unit to learn the representations from a large effective receptive field. To remove the gridding artifacts caused by dilated convolutions, the EESP fuses the feature maps using [hierarchical feature fusion](https://paperswithcode.com/method/hierarchical-feature-fusion) (HFF).

AI Glossary

Explanation vs Attention: A Two-Player Game to Obtain Attention for VQA