In a given dataset for semantic image segmentation, the number of samples per class should be the same, so that no classifier would be biased towards the majority class (here included the background). It is very difficult, if not impossible, to achieve a perfect balance between the several classes of objects of a dataset. Considering that the segmentation of the objects  is accomplished at the pixel level, the number of pixels for each class must be taken into account. As a matter of fact, in image semantic segmentation, 
different classes and the background may have quite different
sizes. Therefore, the image segmentation problem is naturally unbalanced. The IPBI is based on the concept of entropy, a common measure used in many fields of science. In a general sense, it measures the amount of disorder of a system. For the sake of semantic image segmentation, the ideal dataset should have the same number of instances per class, as well as the same number of pixels in all classes. Similar reasoning can be done considering the number of pixels of all samples in a class, so that we can obtain the
pixels balance measure for the dataset. Overall, IPBI evaluates the balance of pixels and number of instances of an image semantic segmentation dataset and, so, it is usefull to compare different datasets.

IPBI

EPYNET: Efficient Pyramidal Network for Clothing Segmentation

**FCPose** is a fully convolutional multi-person [pose estimation framework](https://paperswithcode.com/methods/category/pose-estimation-models) using dynamic instance-aware convolutions. Different from existing methods, which often require ROI (Region of Interest) operations and/or grouping post-processing, FCPose eliminates the ROIs and grouping pre-processing with dynamic instance aware keypoint estimation heads. The dynamic keypoint heads are conditioned on each instance (person), and can encode the instance concept in the dynamically-generated weights of their filters. 

Overall, FCPose is built upon the one-stage object detector [FCOS](https://paperswithcode.com/method/fcos). The controller that generates the weights of the keypoint heads is attached to the FCOS heads. The weights $\theta\_{i}$ generated by the controller is used to fulfill the keypoint head $f$ for the instance $i$. Moreover, a keypoint refinement module is introduced to predict the offsets from each location of the heatmaps to the ground-truth keypoints. Finally, the coordinates derived from the predicted heatmaps are refined by the offsets predicted by the keypoint refinement module, resulting in the final keypoint results. "Rel. coord." is a map of the relative coordinates from all the locations of the feature maps $F$ to the location where the weights are generated. The relative coordinate map is concatenated to $F$ as the input to the keypoint head.

FCPose

FCPose: Fully Convolutional Multi-Person Pose Estimation with Dynamic Instance-Aware Convolutions

**MixNet** is a type of convolutional neural network discovered via AutoML that utilises MixConvs instead of regular depthwise convolutions.

MixNet

MixConv: Mixed Depthwise Convolutional Kernels

**Conditional Batch Normalization (CBN)** is a class-conditional variant of [batch normalization](https://paperswithcode.com/method/batch-normalization). The key idea is to predict the $\gamma$ and $\beta$ of the batch normalization from an embedding - e.g. a language embedding in VQA. CBN enables the linguistic embedding to manipulate entire feature maps by scaling them up or down, negating them, or shutting them off. CBN has also been used in [GANs](https://paperswithcode.com/methods/category/generative-adversarial-networks) to allow class information to affect the batch normalization parameters.

Consider a single convolutional layer with batch normalization module $\text{BN}\left(F\_{i,c,h,w}|\gamma\_{c}, \beta\_{c}\right)$ for which pretrained scalars $\gamma\_{c}$ and $\beta\_{c}$ are available. We would like to directly predict these affine scaling parameters from, e.g., a language embedding $\mathbf{e\_{q}}$. When starting the training procedure, these parameters must be close to the pretrained values to recover the original [ResNet](https://paperswithcode.com/method/resnet) model as a poor initialization could significantly deteriorate performance. Unfortunately, it is difficult to initialize a network to output the pretrained $\gamma$ and $\beta$. For these reasons, the authors propose to predict a change $\delta\beta\_{c}$ and $\delta\gamma\_{c}$ on the frozen original scalars, for which it is straightforward to initialize a neural network to produce an output with zero-mean and small variance.

The authors use a one-hidden-layer MLP to predict these deltas from a question embedding $\mathbf{e\_{q}}$ for all feature maps within the layer:

$$\Delta\beta = \text{MLP}\left(\mathbf{e\_{q}}\right)$$

$$\Delta\gamma = \text{MLP}\left(\mathbf{e\_{q}}\right)$$

So, given a feature map with $C$ channels, these MLPs output a vector of size $C$. We then add these predictions to the $\beta$ and $\gamma$ parameters:

$$ \hat{\beta}\_{c} = \beta\_{c} + \Delta\beta\_{c} $$

$$ \hat{\gamma}\_{c} = \gamma\_{c} + \Delta\gamma\_{c} $$

Finally, these updated $\hat{β}$ and $\hat{\gamma}$ are used as parameters for the batch normalization: $\text{BN}\left(F\_{i,c,h,w}|\hat{\gamma\_{c}}, \hat{\beta\_{c}}\right)$. The authors freeze all ResNet parameters, including $\gamma$ and $\beta$, during training. A ResNet consists of
four stages of computation, each subdivided in several residual blocks. In each block, the authors apply CBN to the three convolutional layers.

Conditional Batch Normalization

Modulating early visual processing by language

**GradientDICE** is a density ratio learning method for estimating the density ratio between the state distribution of the target policy and the sampling distribution in off-policy reinforcement learning. It optimizes a different objective from [GenDICE](https://arxiv.org/abs/2002.09072) by using the Perron-Frobenius theorem and eliminating GenDICE’s use of divergence, such that nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.

GradientDICE

GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values

**Temporal Pyramid Network**, or **TPN**, is a pyramid level module for action recognition at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks in a plug-and-play manner. The source of features and the fusion of features form a feature hierarchy for the backbone so that it can capture action instances at various tempos. In the TPN, a Backbone Network is used to extract multiple level features, a Spatial Semantic Modulation spatially downsamples features to align semantics, a Temporal Rate Modulation temporally downsamples features to adjust relative tempo among levels, Information Flow aggregates features in various directions to enhance and enrich level-wise representations and Final Prediction rescales and concatenates all levels of pyramid along channel dimension.

Temporal Pyramid Network for Action Recognition

**Rectified Adam**, or **RAdam**, is a variant of the [Adam](https://paperswithcode.com/method/adam) stochastic optimizer that introduces a term to rectify the variance of the adaptive learning rate. It seeks to tackle the bad convergence problem suffered by Adam. The authors argue that the root cause of this behaviour is that the adaptive learning rate has undesirably large variance in the early stage of model training, due to the limited amount of training samples being used. Thus, to reduce such variance, it is better to use smaller learning rates in the first few epochs of training - which justifies the warmup heuristic. This heuristic motivates RAdam which rectifies the variance problem:

$$g\_{t} = \nabla\_{\theta}f\_{t}\left(\theta\_{t-1}\right) $$

$$v\_{t} = 1/\beta\_{2}v\_{t-1} + \left(1-\beta\_{2}\right)g^{2}\_{t} $$

$$m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)g\_{t} $$

$$ \hat{m\_{t}} = m\_{t} / \left(1-\beta^{t}\_{1}\right) $$

$$ \rho\_{t} = \rho\_{\infty} - 2t\beta^{t}\_{2}/\left(1-\beta^{t}\_{2}\right) $$

$$\rho_{\infty} = \frac{2}{1-\beta_2} - 1$$ 

If the variance is tractable - $\rho\_{t} > 4$ then:

...the adaptive learning rate is computed as:

$$ l\_{t} = \sqrt{\left(1-\beta^{t}\_{2}\right)/v\_{t}}$$

...the variance rectification term is calculated as:

$$ r\_{t} = \sqrt{\frac{(\rho\_{t}-4)(\rho\_{t}-2)\rho\_{\infty}}{(\rho\_{\infty}-4)(\rho\_{\infty}-2)\rho\_{t}}}$$

...and we update parameters with adaptive momentum:

$$ \theta\_{t} = \theta\_{t-1} - \alpha\_{t}r\_{t}\hat{m}\_{t}l\_{t} $$

If the variance isn't tractable we update instead with:

$$ \theta\_{t} = \theta\_{t-1} - \alpha\_{t}\hat{m}\_{t} $$

RAdam

On the Variance of the Adaptive Learning Rate and Beyond

InterBERT aims to model interaction between information flows pertaining to different modalities. This new architecture builds multi-modal interaction and preserves the independence of single modal representation. InterBERT is built with an image embedding layer, a text embedding layer, a single-stream interaction module, and a two stream extraction module. The model is pre-trained with three tasks: 1) masked segment modeling, 2) masked region modeling, and 3) image-text matching.

InterBERT

InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining

Combines learned time-frequency representation with a masker architecture based on 1D [dilated convolution](https://paperswithcode.com/method/dilated-convolution).

ConvTasNet

Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation

Represent and associate with a composite of primitive fields.

Composite Fields

PifPaf: Composite Fields for Human Pose Estimation

**Spatial-Reduction Attention**, or **SRA**, is a [multi-head attention](https://paperswithcode.com/method/multi-head-attention) module used in the [Pyramid Vision Transformer](https://paperswithcode.com/method/pvt) architecture which reduces the spatial scale of the key $K$ and value $V$ before the attention operation. This reduces the computational/memory overhead. Details of the SRA in the stage $i$ can be formulated as follows:

$$
\text{SRA}(Q, K, V)=\text { Concat }\left(\operatorname{head}\_{0}, \ldots \text { head }\_{N\_{i}}\right) W^{O} $$

$$\text{ head}\_{j}=\text { Attention }\left(Q W\_{j}^{Q}, \operatorname{SR}(K) W\_{j}^{K}, \operatorname{SR}(V) W\_{j}^{V}\right)
$$

where Concat $(\cdot)$ is the concatenation operation. $W\_{j}^{Q} \in \mathbb{R}^{C\_{i} \times d\_{\text {head }}}$, $W\_{j}^{K} \in \mathbb{R}^{C\_{i} \times d\_{\text {head }}}$, $W\_{j}^{V} \in \mathbb{R}^{C\_{i} \times d\_{\text {head }}}$, and $W^{O} \in \mathbb{R}^{C\_{i} \times C\_{i}}$ are linear projection parameters. $N\_{i}$ is the head number of the attention layer in Stage $i$. Therefore, the dimension of each head (i.e. $\left.d\_{\text {head }}\right)$ is equal to $\frac{C\_{i}}{N\_{i}} . \text{SR}(\cdot)$ is the operation for reducing the spatial dimension of the input sequence ($K$ or $V$ ), which is written as:

$$
\text{SR}(\mathbf{x})=\text{Norm}\left(\operatorname{Reshape}\left(\mathbf{x}, R\_{i}\right) W^{S}\right)
$$

Here, $\mathbf{x} \in \mathbb{R}^{\left(H\_{i} W\_{i}\right) \times C\_{i}}$ represents a input sequence, and $R\_{i}$ denotes the reduction ratio of the attention layers in Stage $i .$ Reshape $\left(\mathbf{x}, R\_{i}\right)$ is an operation of reshaping the input sequence $\mathbf{x}$ to a sequence of size $\frac{H\_{i} W\_{i}}{R\_{i}^{2}} \times\left(R\_{i}^{2} C\_{i}\right)$. $W\_{S} \in \mathbb{R}^{\left(R\_{i}^{2} C\_{i}\right) \times C\_{i}}$ is a linear projection that reduces the dimension of the input sequence to $C\_{i}$. $\text{Norm}(\cdot)$ refers to layer normalization.

Spatial-Reduction Attention

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

**Xavier Initialization**, or **Glorot Initialization**, is an initialization scheme for neural networks. Biases are initialized be 0 and the weights $W\_{ij}$ at each layer are initialized as:

$$ W\_{ij} \sim U\left[-\frac{\sqrt{6}}{\sqrt{fan_{in} + fan_{out}}}, \frac{\sqrt{6}}{\sqrt{fan_{in} + fan_{out}}}\right] $$

Where $U$ is a uniform distribution and $fan_{in}$ is the size of the previous layer (number of columns in $W$) and $fan_{out}$ is the size of the current layer.

Xavier Initialization

A **Region Proposal Network**, or **RPN**, is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals. RPN and algorithms like [Fast R-CNN](https://paperswithcode.com/method/fast-r-cnn) can be merged into a single network by sharing their convolutional features - using the recently popular terminology of neural networks with attention mechanisms, the RPN component tells the unified network where to look.

RPNs are designed to efficiently predict region proposals with a wide range of scales and aspect ratios. RPNs use anchor boxes that serve as references at multiple scales and aspect ratios. The scheme can be thought of as a pyramid of regression references, which avoids enumerating images or filters of multiple scales or aspect ratios.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

**Fast Voxel Query** is a module used in the [Voxel Transformer](https://paperswithcode.com/method/votr) 3D object detection model implementation of self-attention, specifically Local and Dilated Attention. For each querying index $v\_{i}$, an attending voxel index $v\_{j}$ is determined by Local and Dilated Attention. Then we can lookup the non-empty index $j$ in the hash table with hashed $v\_{j}$ as the key. Finally, the non-empty index $j$ is used to gather the attending feature $f\_{j}$ from $\mathcal{F}$ for [multi-head attention](https://paperswithcode.com/method/multi-head-attention).

Fast Voxel Query

Voxel Transformer for 3D Object Detection

**Fast R-CNN** is an object detection model that improves in its predecessor [R-CNN](https://paperswithcode.com/method/r-cnn) in a number of ways. Instead of extracting CNN features independently for each region of interest, Fast R-CNN aggregates them into a single forward pass over the image; i.e. regions of interest from the same image share computation and memory in the forward and backward passes.

Fast R-CNN

**SpreadsheetCoder** is a neural network architecture for spreadsheet formula prediction. It is a [BERT](https://paperswithcode.com/method/bert)-based model architecture to represent the tabular context in both row-based and column-based formats. A [BERT](https://paperswithcode.com/method/bert) encoder computes an embedding vector for each input token, incorporating the contextual information from nearby rows and columns. The BERT encoder is initialized from the weights pre-trained on English text corpora, which is beneficial for encoding table headers. To handle cell references, a two-stage decoding process is used inspired by sketch learning for program synthesis. The decoder first generates a formula sketch, which does not include concrete cell references, and then predicts the corresponding cell ranges to generate the complete formula

SpreadsheetCoder

SpreadsheetCoder: Formula Prediction from Semi-structured Context

**OA-Mix** is a general and effective data augmentation method for single-domain generalization in object detection. It increases image diversity while preserving important semantic features with
multi-level transformations and object-aware mixing.

OA-Mix

Object-Aware Domain Generalization for Object Detection

An **Inverted Residual Block**, sometimes called an **MBConv Block**, is a type of residual block used for image models that uses an inverted structure for efficiency reasons. It was originally proposed for the [MobileNetV2](https://paperswithcode.com/method/mobilenetv2) CNN architecture. It has since been reused for several mobile-optimized CNNs.

A traditional [Residual Block](https://paperswithcode.com/method/residual-block) has a wide -> narrow -> wide structure with the number of channels. The input has a high number of channels, which are compressed with a [1x1 convolution](https://paperswithcode.com/method/1x1-convolution). The number of channels is then increased again with a 1x1 [convolution](https://paperswithcode.com/method/convolution) so input and output can be added. 

In contrast, an Inverted Residual Block follows a narrow -> wide -> narrow approach, hence the inversion. We first widen with a 1x1 convolution, then use a 3x3 [depthwise convolution](https://paperswithcode.com/method/depthwise-convolution) (which greatly reduces the number of parameters), then we use a 1x1 convolution to reduce the number of channels so input and output can be added.

Viet-Anh on Software

AI Glossary

Tech Terminology

Instances-Pixels Balance Index