**Spiking Neural Networks** (**SNNs**)  are a class of artificial neural networks inspired by the structure and functioning of the brain's neural networks. Unlike traditional artificial neural networks that operate based on continuous firing rates, SNNs simulate the behavior of individual neurons through discrete spikes or action potentials. These spikes are triggered when the neuron's membrane potential reaches a certain threshold, and they propagate through the network, communicating information and triggering subsequent neuron activations. This spike-based communication allows SNNs to capture the temporal dynamics of information processing and exhibit asynchronous, event-driven behavior, making them well-suited for tasks such as temporal pattern recognition, event detection, and real-time processing. SNNs have gained attention due to their potential in efficiently processing and encoding information, offering advantages in energy efficiency, robustness, and compatibility with neuromorphic hardware architectures.

Self-Normalizing Neural Networks

**CTAL** is a pre-training framework for strong audio-and-language representations with a [Transformer](https://paperswithcode.com/method/transformer), which aims to learn the intra-modality and inter-modalities connections between audio and language through two proxy tasks on a large amount of audio- and-language pairs: masked language modeling and masked cross-modal acoustic modeling. The pre-trained model is a Transformer for Audio and Language, i.e., CTAL, which consists of two modules, a language stream encoding module which adapts word as input element, and a text-referred audio stream encoder module which accepts both frame-level Mel-spectrograms and token-level output embeddings from the language stream

CTAL

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

**Animatable Reconstruction of Clothed Humans** is an end-to-end framework for accurate reconstruction of animation-ready 3D clothed humans from a monocular image. ARCH is a learned pose-aware model that produces detailed 3D rigged full-body human avatars from a single unconstrained RGB image. A Semantic Space and a Semantic Deformation Field are created using a parametric 3D body estimator. They allow the transformation of 2D/3D clothed humans into a canonical space, reducing ambiguities in geometry caused by pose variations and occlusions in training data. Detailed surface geometry and appearance are learned using an implicit function representation with spatial local features.

ARCH

ARCH: Animatable Reconstruction of Clothed Humans

**ALAE**, or **Adversarial Latent Autoencoder**, is a type of autoencoder that attempts to overcome some of the limitations of[ generative adversarial networks](https://paperswithcode.com/paper/generative-adversarial-networks). The architecture allows the latent distribution to be learned from data to address entanglement (A). The output data distribution is learned with an adversarial strategy (B). Thus, we retain the generative properties of GANs, as well as the ability to build on the recent advances in this area. For instance, we can include independent sources of stochasticity, which have proven essential for generating image details, or can leverage recent improvements on GAN loss functions, regularization, and hyperparameters tuning. Finally, to implement (A) and (B), AE reciprocity is imposed in the latent space (C). Therefore, we can avoid using reconstruction losses based on simple $\mathcal{l}\){2}$ norm that operates in data space, where they are often suboptimal, like for the image space. Since it works on the latent space, rather than autoencoding the data space, the approach is named Adversarial Latent Autoencoder (ALAE).

ALAE

Adversarial Latent Autoencoders

**RotatE** is a method for generating graph embeddings which is able to model and infer various relation patterns including: symmetry/antisymmetry, inversion, and composition. Specifically, the RotatE model defines each relation as a rotation from the source entity to the target entity in the complex vector space. The RotatE model is trained using a [self-adversarial negative sampling](https://paperswithcode.com/method/self-adversarial-negative-sampling) technique.

RotatE

RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

**Review-guided Answer Helpfulness Prediction** (RAHP) is a textual inference model for identifying helpful answers in e-commerce. It not only considers the interactions between QA pairs, but also investigates the opinion coherence between the answer and crowds' opinions reflected in the reviews, which is another important factor to identify helpful answers.

RAHP

Review-guided Helpful Answer Identification in E-commerce

A **Dense Block** is a module used in convolutional neural networks that connects *all layers* (with matching feature-map sizes) directly with each other. It was originally proposed as part of the [DenseNet](https://paperswithcode.com/method/densenet) architecture. To preserve the feed-forward nature, each layer obtains additional inputs from all preceding layers and passes on its own feature-maps to all subsequent layers. In contrast to [ResNets](https://paperswithcode.com/method/resnet), we never combine features through summation before they are passed into a layer; instead, we combine features by concatenating them. Hence, the $\ell^{th}$ layer has $\ell$ inputs, consisting of the feature-maps of all preceding convolutional blocks. Its own feature-maps are passed on to all $L-\ell$ subsequent layers. This introduces $\frac{L(L+1)}{2}$  connections in an $L$-layer network, instead of just $L$, as in traditional architectures: "dense connectivity".

Dense Block

Densely Connected Convolutional Networks

**Lookahead** is a type of stochastic optimizer that iteratively updates two sets of weights: "fast" and "slow". Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of *fast weights* generated by another optimizer.



**Algorithm 1** Lookahead Optimizer

**Require** Initial parameters $\phi_0$, objective function $L$ 

**Require** Synchronization period $k$, slow weights step size $\alpha$, optimizer $A$

&nbsp;&nbsp;  **for** $t=1, 2, \dots$

&nbsp;&nbsp;&nbsp;&nbsp; Synchronize parameters $\theta_{t,0} \gets \phi_{t-1}$

&nbsp;&nbsp;&nbsp;&nbsp; **for** $i=1, 2, \dots, k$

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sample minibatch of data $d \sim \mathcal{D}$

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\theta_{t,i} \gets \theta_{t,i-1} + A(L, \theta_{t,i-1}, d)$

&nbsp;&nbsp;&nbsp;&nbsp; **endfor**

&nbsp;&nbsp;&nbsp;&nbsp; Perform outer update $\phi_t \gets \phi_{t-1} + \alpha (\theta_{t,k} - \phi_{t-1})$

&nbsp;&nbsp; **endfor**

&nbsp;&nbsp; **return** parameters $\phi$

Lookahead

Lookahead Optimizer: k steps forward, 1 step back

**Grid R-CNN** is an object detection framework, where the traditional regression
formulation is replaced by a grid point guided localization mechanism.

Grid R-CNN divides the object bounding box region into grids and employs a fully convolutional network ([FCN](https://paperswithcode.com/method/fcn)) to predict the locations of grid points. Owing to the position sensitive property of fully convolutional architecture, Grid R-CNN maintains the explicit spatial information and grid points locations can be obtained in pixel level. When a certain number of grid points at specified location are known, the corresponding bounding box is definitely determined. Guided by the grid points, Grid R-CNN can determine more accurate object bounding box than regression method which lacks the guidance of explicit spatial information.

Grid R-CNN

**EsViT** proposes two techniques for developing efficient self-supervised vision transformers for visual representation leaning: a multi-stage architecture with sparse self-attention and a new pre-training task of region matching. The multi-stage architecture reduces modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. The new pretraining task allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations.

EsViT

Efficient Self-supervised Vision Transformers for Representation Learning

The **Re-Attention Module** is an attention layer used in the [DeepViT](https://paperswithcode.com/method/deepvit) architecture which mixes the attention map with a learnable matrix before multiplying with the values. The motivation is to re-generate the attention maps to increase their diversity at different layers with negligible computation and memory cost. The authors note that traditional self-attention fails to learn effective concepts for representation learning in deeper layers of ViT -- attention maps become more similar and less diverse in deeper layers (attention collapse) - and this hinders the model from getting expected performance gain. Re-attention is implemented by:

$$
\operatorname{Re}-\operatorname{Attention}(Q, K, V)=\operatorname{Norm}\left(\Theta^{\top}\left(\operatorname{Softmax}\left(\frac{Q K^{\top}}{\sqrt{d}}\right)\right)\right) V
$$

where transformation matrix $\Theta$ is multiplied to the self-attention map $\textbf{A}$ along the head dimension.

Re-Attention Module

DeepViT: Towards Deeper Vision Transformer

CrossTransformers is a Transformer-based neural network architecture which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled images, and then infer class membership by computing distances between spatially-corresponding features.

CrossTransformers

CrossTransformers: spatially-aware few-shot transfer

**In-Place Activated Batch Normalization**, or **InPlace-ABN**, substitutes the conventionally used succession of [BatchNorm](https://paperswithcode.com/method/batch-normalization) + Activation layers with a single plugin layer, hence avoiding invasive framework surgery while providing straightforward applicability for existing deep learning frameworks. It approximately halves the memory requirements during training of modern deep learning models.

InPlace-ABN

In-Place Activated BatchNorm for Memory-Optimized Training of DNNs

_**Independent component analysis** (ICA) is a statistical and computational technique for revealing hidden factors that underlie sets of random variables, measurements, or signals._

_ICA defines a generative model for the observed multivariate data, which is typically given as a large database of samples. In the model, the data variables are assumed to be linear mixtures of some unknown latent variables, and the mixing system is also unknown. The latent variables are assumed nongaussian and mutually independent, and they are called the independent components of the observed data. These independent components, also called sources or factors, can be found by ICA._

_ICA is superficially related to principal component analysis and factor analysis. ICA is a much more powerful technique, however, capable of finding the underlying factors or sources when these classic methods fail completely._


Extracted from (https://www.cs.helsinki.fi/u/ahyvarin/whatisica.shtml)

**Source papers**:

[Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture](https://doi.org/10.1016/0165-1684(91)90079-X)

[Independent component analysis, A new concept?](https://doi.org/10.1016/0165-1684(94)90029-9)

[Independent component analysis: algorithms and applications](https://doi.org/10.1016/S0893-6080(00)00026-5)

**BART** is a [denoising autoencoder](https://paperswithcode.com/method/denoising-autoencoder) for pretraining sequence-to-sequence models. It is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard [Transformer](https://paperswithcode.com/method/transformer)-based neural machine translation architecture. It uses a standard seq2seq/NMT architecture with a bidirectional encoder (like [BERT](https://paperswithcode.com/method/bert)) and a left-to-right decoder (like [GPT](https://paperswithcode.com/method/gpt)). This means the encoder's attention mask is fully visible, like BERT, and the decoder's attention mask is causal, like [GPT2](https://paperswithcode.com/method/gpt-2).

BART

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

**Confidence Calibration with an Auxiliary Class**, or **CCAC**, is a post-hoc confidence calibration method for DNN classifiers on OOD datasets. The key feature of CCAC is an auxiliary class in the calibration model which separates mis-classified samples from correctly classified ones, thus effectively mitigating the target DNN’s being confidently wrong. It also reduces the number of free parameters in CCAC to reduce free parameters and facilitate transfer to a new unseen dataset.

CCAC

Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets

A **1 x 1 Convolution** is a [convolution](https://paperswithcode.com/method/convolution) with some special properties in that it can be used for dimensionality reduction, efficient low dimensional embeddings, and applying non-linearity after convolutions. It maps an input pixel with all its channels to an output pixel which can be squeezed to a desired output depth. It can be viewed as an [MLP](https://paperswithcode.com/method/feedforward-network) looking at a particular pixel location.

Image Credit: [http://deeplearning.ai](http://deeplearning.ai)

1x1 Convolution

Network In Network

**RandomRotate** is a type of image data augmentation where we randomly rotate the image by a degree.

Viet-Anh on Software

AI Glossary

Tech Terminology

Spiking Neural Networks

CTAL

Animatable Reconstruction of Clothed Humans

Adversarial Latent Autoencoder

RotatE

Review-guided Answer Helpfulness Prediction

Dense Block

Lookahead

Grid R-CNN

EsViT

Re-Attention Module

CrossTransformers

In-Place Activated Batch Normalization

Independent Component Analysis

BART

Confidence Calibration with an Auxiliary Class)

1x1 Convolution

RandomRotate

Viet-Anh on Software