BCI MI framework to classifiy brain signals using a multimodal decission making phase, with an addtional differentiation of the signal.

Motor-Imagery-Based Brain Computer Interface using Signal Derivation and Aggregation Functions

**Seesaw Loss** is a loss function for long-tailed instance segmentation. It dynamically re-balances the gradients of positive and negative samples on a tail class with two complementary factors: mitigation factor and compensation factor. The mitigation factor reduces punishments to tail categories w.r.t the ratio of cumulative training instances between different categories. Meanwhile, the compensation factor increases the penalty of misclassified instances to avoid false positives of tail categories. The synergy of the two factors enables Seesaw Loss to mitigate the overwhelming punishments on tail classes as well as compensate for the risk of misclassification caused by diminished penalties.

$$ L\_{seesaw}\left(\mathbf{x}\right) = - \sum^{C}\_{i=1}y\_{i}\log\left(\hat{\sigma}\_{i}\right) $$

$$ \text{with } \hat{\sigma\_{i}} = \frac{e^{z\_{i}}}{- \sum^{C}\_{j\neq{1}}\mathcal{S}\_{ij}e^{z\_{j}}+e^{z\_{i}} } $$

Here $\mathcal{S}\_{ij}$ works as a tunable balancing factor between different classes. By a careful design of $\mathcal{S}\_{ij}$, Seesaw loss adjusts the punishments on class j from positive samples of class $i$. Seesaw loss determines $\mathcal{S}\_{ij}$ by a mitigation factor and a compensation factor, as:

$$ \mathcal{S}\_{ij} =\mathcal{M}\_{ij} · \mathcal{C}\_{ij}  $$

The mitigation factor $\mathcal{M}\_{ij}$ decreases the penalty on tail class $j$ according to a ratio of instance numbers between tail class $j$ and head class $i$. The compensation factor $\mathcal{C}\_{ij}$ increases the penalty on class $j$ whenever an instance of class $i$ is misclassified to class $j$.

Seesaw Loss

Seesaw Loss for Long-Tailed Instance Segmentation

**Orthogonal Regularization** is a regularization technique for convolutional neural networks, introduced with generative modelling as the task in mind. Orthogonality is argued to be a desirable quality in ConvNet filters, partially because multiplication by an orthogonal matrix leaves the norm of the original matrix unchanged. This property is valuable in deep or recurrent networks, where repeated matrix multiplication can result in signals vanishing or exploding. To try to maintain orthogonality throughout training, Orthogonal Regularization encourages weights to be orthogonal by pushing them towards the nearest orthogonal manifold. The objective function is augmented with the cost:

$$ \mathcal{L}\_{ortho} = \sum\left(|WW^{T} − I|\right) $$

Where $\sum$ indicates a sum across all filter banks, $W$ is a filter bank, and $I$ is the identity matrix

Orthogonal Regularization

Neural Photo Editing with Introspective Adversarial Networks

**WaveNet** is an audio generative model based on the [PixelCNN](https://paperswithcode.com/method/pixelcnn) architecture. In order to deal with long-range temporal dependencies needed for raw audio generation, architectures are developed based on dilated causal convolutions, which exhibit very large receptive fields.

The joint probability of a waveform $\vec{x} = \{ x_1, \dots, x_T \}$ is factorised as a product of conditional probabilities as follows:

$$p\left(\vec{x}\right) = \prod_{t=1}^{T} p\left(x_t \mid x_1, \dots ,x_{t-1}\right)$$

Each audio sample $x_t$ is therefore conditioned on the samples at all previous timesteps.

WaveNet

WaveNet: A Generative Model for Raw Audio

Lung cancer classification in screening computed tomography (CT) scans is one of the most crucial tasks for early detection of this disease. Many lives can be saved if we are able to accurately classify malignant/cancerous lung nodules. Consequently, several deep learning based models have been proposed recently to classify lung nodules as malignant or benign. Nevertheless, the large variation in the size and heterogeneous appearance of the nodules makes this task an extremely challenging one. We propose a new Progressive Growing Channel Attentive Non-Local (ProCAN) network for lung nodule classification. The proposed method addresses this challenge from three different aspects. First, we enrich the Non-Local network by adding channel-wise attention capability to it. Second, we apply Curriculum Learning principles, whereby we first train our model on easy examples before hard ones. Third, as the classification task gets harder during the Curriculum learning, our model is progressively grown to increase its capability of handling the task at hand. We examined our proposed method on two different public datasets and compared its performance with state-of-the-art methods in the literature. The results show that the ProCAN model outperforms state-of-the-art methods and achieves an AUC of 98.05% and an accuracy of 95.28% on the LIDC-IDRI dataset. Moreover, we conducted extensive ablation studies to analyze the contribution and effects of each new component of our proposed method.

ProCAN

ProCAN: Progressive Growing Channel Attentive Non-Local Network for Lung Nodule Classification

**ReGLU** is an activation function which is a variant of [GLU](https://paperswithcode.com/method/glu). The definition is as follows:

$$ \text{ReGLU}\left(x, W, V, b, c\right) = \max\left(0, xW + b\right) \otimes \left(xV + c\right) $$

ReGLU

GLU Variants Improve Transformer

**Structurally Regularized Deep Clustering**, or **SRDC**, is a deep network based discriminative clustering method for domain adaptation that minimizes the KL divergence between predictive label distribution of the network and an introduced auxiliary one. Replacing the auxiliary distribution with that formed by ground-truth labels of source data implements the structural source regularization via a simple strategy of joint network training.

SRDC

Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering

**Twins-SVT** is a type of [vision transformer](https://paperswithcode.com/methods/category/vision-transformer) which utilizes a [spatially separable attention mechanism](https://paperswithcode.com/method/spatially-separable-self-attention) (SSAM) which is composed of two types of attention operations—(i) locally-grouped self-attention (LSA), and (ii) global sub-sampled attention (GSA), where LSA captures the fine-grained and short-distance information and GSA deals with the long-distance and global information. On top of this, it utilizes [conditional position encodings](https://paperswithcode.com/method/conditional-positional-encoding) as well as the architectural design of the [Pyramid Vision Transformer](https://paperswithcode.com/method/pvt).

Twins-SVT

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

**MiVOS** is a video object segmentation model which decouples interaction-to-mask and mask propagation. By decoupling interaction from propagation, MiVOS is versatile and not limited by the type of interactions. It uses three modules: Interaction-to-Mask, Propagation and Difference-Aware Fusion. Trained separately, the interaction module converts user interactions to an object mask, which is then temporally propagated by our propagation module using a novel top-filtering strategy in reading the space-time memory. To effectively take the user's intent into account, a novel difference-aware module is proposed to learn how to properly fuse the masks before and after each interaction, which are aligned with the target frames by employing the space-time memory.

MiVOS

Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion

**Voxel R-CNN** is a voxel-based two stage framework for 3D object detection. It consists of a 3D backbone network, a 2D bird-eye-view (BEV) Region Proposal Network and a detect head. Voxel RoI Pooling is devised to extract RoI features directly from raw features for further refinement. 

End-to-end, the point clouds are first divided into regular voxels and fed into the 3D backbone network for feature extraction. Then, the 3D feature volumes are converted into BEV representation, on which the 2D backbone and [RPN](https://paperswithcode.com/method/rpn) are applied for region proposal generation. Subsequently, [Voxel RoI Pooling](https://paperswithcode.com/method/voxel-roi-pooling) directly extracts RoI features from the 3D feature volumes. Finally the RoI features are exploited in the detect head for further box refinement.

Voxel R-CNN

Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection

Multimodal alignment between language and vision is the fundamental topic in current vision-language model research. Contrastive Captioners (CoCa), as a representative method, integrates Contrastive Language-Image Pretraining (CLIP) and Image Caption (IC) into a unified framework, resulting in impressive results. CLIP imposes a bidirectional constraints on global representation of entire images and sentences. Although IC conducts an unidirectional image-to-text generation on local representation, it lacks any constraint on local text-to-image reconstruction, which limits the ability to understand images at a fine-grained level when aligned with texts. To achieve multimodal alignment from both global and local perspectives, this paper proposes Symmetrizing Contrastive Captioners (SyCoCa), which introduces bidirectional interactions on images and texts across the global and local representation levels. Specifically, we expand a Text-Guided Masked Image Modeling (TG-MIM) head based on ITC and IC heads. The improved SyCoCa can further leverage textual cues to reconstruct contextual images and visual cues to predict textual contents. When implementing bidirectional local interactions, the local contents of images tend to be cluttered or unrelated to their textual descriptions. Thus, we employ an attentive masking strategy to select effective image patches for interaction. Extensive experiments on five vision-language tasks, including image-text retrieval, image-captioning, visual question answering, and zero-shot/finetuned image classification, validate the effectiveness of our proposed method.

SyCoCa

SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment

**PipeTransformer** is a method for automated elastic pipelining for efficient distributed training of [Transformer](https://paperswithcode.com/method/transformer) models. In PipeTransformer, an adaptive on the fly freeze algorithm is used that can identify and freeze some layers gradually during training, as well as an elastic pipelining system that can dynamically allocate resources to train the remaining active layers. More specifically, PipeTransformer automatically excludes frozen layers from the pipeline, packs active layers into fewer GPUs, and forks more replicas to increase data-parallel width.

PipeTransformer

PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers

A novel built-in attention mechanism, that is complementary to all other prior attention mechanisms (e.g. squeeze and excitation, transformers) that are external (i.e., not built-in - please read paper for more details)

Weight excitation

Weight Excitation: Built-in Attention Mechanisms in Convolutional Neural Networks

**MushroomRL** is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. The architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. MushroomRL comes with a strongly modular architecture that makes it easy to understand how each component is structured and how it interacts with other ones; moreover it provides an exhaustive list of RL methodologies, such as:

MushroomRL

MushroomRL: Simplifying Reinforcement Learning Research

**XLM** is a [Transformer](https://paperswithcode.com/method/transformer) based architecture that is pre-trained using one of three language modelling objectives:

1. Causal Language Modeling - models the probability of a word given the previous words in a sentence.
2. Masked Language Modeling - the masked language modeling objective of [BERT](https://paperswithcode.com/method/bert).
3. Translation Language Modeling - a (new) translation language modeling objective for improving cross-lingual pre-training.

The authors find that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models.

Cross-lingual Language Model Pretraining

Please enter a description about the method here

Library

0/1 Deep Neural Networks via Block Coordinate Descent

**DBlock** is a residual based block used in the discriminator of the [GAN-TTS](https://paperswithcode.com/method/gan-tts) architecture. They are similar to the [GBlocks](https://paperswithcode.com/method/gblock) used in the generator, but without batch normalisation.

DBlock

High Fidelity Speech Synthesis with Adversarial Networks

A **Data-Efficient Image Transformer** is a type of [Vision Transformer](https://paperswithcode.com/method/vision-transformer) for image classification tasks. The model is trained using a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention.

AI Glossary

Enhanced-Multimodal Fuzzy Framework

Seesaw Loss

Orthogonal Regularization

WaveNet

Progressive Growing Channel Attentive Non-Local Network

ReGLU

Structurally Regularized Deep Clustering

Twins-SVT

Modular Interactive VOS

Voxel R-CNN

Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment

PipeTransformer

Weight excitation

MushroomRL

XLM

Lib

DBlock

Data-efficient Image Transformer