Pathology Language and Image Pre-Training (PLIP) is a vision-and-language foundation model created by fine-tuning CLIP on pathology images.

PLIP

Leveraging medical Twitter to build a visual–language foundation model for pathology AI

**Semantic reasoning network**, or **SRN**, is an end-to-end trainable framework for scene text recognition that consists of four parts: backbone network, parallel [visual attention](https://paperswithcode.com/method/visual-attention) module (PVAM), global semantic reasoning module (GSRM), and visual-semantic fusion decoder (VSFD). Given an input image, the backbone network is first used to extract 2D features $V$. Then, the PVAM is used to generate $N$ aligned 1-D features $G$, where each feature corresponds to a character in the text and captures the aligned visual information. These $N$ 1-D features $G$ are then fed into a GSRM to capture the semantic information $S$. Finally, the aligned visual features $G$ and the semantic information $S$ are fused by the VSFD to predict $N$ characters. For text string shorter than $N$, ’EOS’ are padded.

Semantic Reasoning Network

Towards Accurate Scene Text Recognition with Semantic Reasoning Networks

**SMITH**, or **Siamese Multi-depth Transformer-based Hierarchical Encoder**, is a [Transformer](https://paperswithcode.com/methods/category/transformers)-based model for document representation learning and matching. It contains several design choices to adapt [self-attention models](https://paperswithcode.com/methods/category/attention-modules) for long text inputs. For the model pre-training, a masked sentence block language modeling task is used in addition to the original masked word language model task used in [BERT](https://paperswithcode.com/method/bert), to capture sentence block relations within a document. Given a sequence of sentence block representation, the document level Transformers learn the contextual representation for each sentence block and the final document representation.

SMITH

Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

**AccoMontage** is a model for accompaniment arrangement, a type of music generation task involving intertwined constraints of melody, harmony, texture, and music structure. AccoMontage generates piano accompaniments for folk/pop songs based on a lead sheet (i.e. a melody with chord progression). It first retrieves phrase montages from a database while recombining them structurally using dynamic programming. Second, chords of the retrieved phrases are manipulated to match the lead sheet via style transfer. Lastly, the system offers controls over the generation process. In contrast to pure deep learning approaches, AccoMontage uses a hybrid pathway, in which rule-based optimization and deep learning are both leveraged.

AccoMontage

AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer

**SPADE**, or **Spatially-Adaptive Normalization** is a conditional normalization method for semantic image synthesis. Similar to [Batch Normalization](https://www.paperswithcode.com/method/batch-normalization), the activation is normalized in the channel-wise manner and then modulated with learned scale and bias. In the SPADE, the mask is first projected onto an embedding space and then convolved to produce the modulation parameters $\gamma$ and $\beta .$ Unlike prior conditional normalization methods, $\gamma$ and $\mathbf{\beta}$ are not vectors, but tensors with spatial dimensions. The produced $\gamma$ and $\mathbf{\beta}$ are multiplied and added to the normalized activation element-wise.

SPADE

Semantic Image Synthesis with Spatially-Adaptive Normalization

**DIoU-NMS** is a type of non-maximum suppression where we use Distance IoU rather than regular DIoU, in which the overlap area and the distance between two central points of bounding boxes are simultaneously considered when suppressing redundant boxes.

In original NMS, the IoU metric is used to suppress the redundant detection boxes, where the overlap area is the unique factor, often yielding false suppression for the cases with occlusion. With DIoU-NMS, we not only consider the overlap area but also central point distance between two boxes.

DIoU-NMS

Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression

The **Compressive Transformer** is an extension to the [Transformer](https://paperswithcode.com/method/transformer) which maps past hidden activations (memories) to a smaller set of compressed representations (compressed memories). The Compressive Transformer uses the same attention mechanism over its set of memories and compressed memories, learning to query both its short-term granular memory and longer-term coarse memory. It builds on the ideas of [Transformer-XL](https://paperswithcode.com/method/transformer-xl) which maintains a memory of past activations at each layer to preserve a longer history of context. The Transformer-XL discards past activations when they become sufficiently old (controlled by the size of the memory). The key principle of the Compressive Transformer is to compress these old memories, instead of discarding them, and store them in an additional [compressed memory](https://paperswithcode.com/method/compressed-memory).

At each time step $t$, we discard the oldest compressed memories (FIFO) and then the oldest $n$ states from ordinary memory are compressed and shifted to the new slot in compressed memory. During training, the compressive memory component is optimized separately from the main language model (separate training loop).

Compressive Transformer

Compressive Transformers for Long-Range Sequence Modelling

An **All-Attention Layer** is an attention module and layer for transformers that merges the self-attention and feedforward sublayers into a single unified attention layer. As opposed to the two-step mechanism of the [Transformer](https://paperswithcode.com/method/transformer) layer, it directly builds its representation from the context and a persistent memory block without going through a feedforward transformation. The additional persistent memory block stores, in the form of key-value vectors, information that does not depend on the context. In terms of parameters, these persistent key-value vectors replace the feedforward sublayer.

All-Attention Layer

Augmenting Self-attention with Persistent Memory

**Learning From Multiple Experts** is a self-paced knowledge distillation framework that aggregates the knowledge from multiple 'Experts' to learn a unified student model. Specifically, the proposed framework involves two levels of adaptive learning schedules: Self-paced Expert Selection and Curriculum Instance Selection, so that the knowledge is adaptively transferred to the 'Student'. The self-paced expert selection automatically controls the impact of knowledge distillation from each expert, so that the learned student model will gradually acquire the knowledge from the experts, and finally exceed the expert. The curriculum instance selection, on the other hand, designs a curriculum for the unified model where the training samples are organized from easy to hard, so that the unified student model will receive a less challenging learning schedule, and gradually learns from easy to hard samples.

LFME

Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification

Z. Průša, P. Balazs and P. L. Søndergaard, "A Noniterative Method for Reconstruction of Phase From STFT Magnitude," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 5, pp. 1154-1164, May 2017, doi: 10.1109/TASLP.2017.2678166.
Abstract: A noniterative method for the reconstruction of the short-time fourier transform (STFT) phase from the magnitude is presented. The method is based on the direct relationship between the partial derivatives of the phase and the logarithm of the magnitude of the un-sampled STFT with respect to the Gaussian window. Although the theory holds in the continuous setting only, the experiments show that the algorithm performs well even in the discretized setting (discrete Gabor transform) with low redundancy using the sampled Gaussian window, the truncated Gaussian window and even other compactly supported windows such as the Hann window. Due to the noniterative nature, the algorithm is very fast and it is suitable for long audio signals. Moreover, solutions of iterative phase reconstruction algorithms can be improved considerably by initializing them with the phase estimate provided by the present algorithm. We present an extensive comparison with the state-of-the-art algorithms in a reproducible manner.
URL: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7890450&isnumber=7895265

PGHI

The **ENet Initial Block** is an image model block used in the [ENet](https://paperswithcode.com/method/enet) semantic segmentation architecture. [Max Pooling](https://paperswithcode.com/method/max-pooling) is performed with non-overlapping 2 × 2 windows, and the [convolution](https://paperswithcode.com/method/convolution) has 13 filters, which sums up to 16 feature maps after concatenation. This is heavily inspired by Inception Modules.

ENet Initial Block

ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation

**MDETR** is an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. It utilizes a [transformer](https://paperswithcode.com/method/transformer)-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. The network is pre-trained on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. The network is then fine-tuned on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation.

MDETR

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

In contrast to typical GANs, a U-Net GAN uses a segmentation network as the discriminator. This segmentation network predicts two classes: real and fake. In doing so, the discriminator gives the generator region-specific feedback. This discriminator design also enables a  [CutMix](https://paperswithcode.com/method/cutmix)-based consistency regularization on the two-dimensional output of the U-Net GAN discriminator, which further improves image synthesis quality.

U-Net GAN

A U-Net Based Discriminator for Generative Adversarial Networks

**LeVIT** is a hybrid neural network for fast inference image classification. LeViT is a stack of [transformer blocks](https://paperswithcode.com/method/transformer), with [pooling steps](https://paperswithcode.com/methods/category/pooling-operation) to reduce the resolution of the activation maps as in classical [convolutional architectures](https://paperswithcode.com/methods/category/convolutional-neural-networks). This replaces the uniform structure of a Transformer by a pyramid with pooling, similar to the [LeNet](https://paperswithcode.com/method/lenet) architecture

LeVIT

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference

Train a convolutional neural network to generate the contents of an arbitrary image region conditioned on its surroundings.

Inpainting

Context Encoders: Feature Learning by Inpainting

**Linformer** is a linear [Transformer](https://paperswithcode.com/method/transformer) that utilises a linear self-attention mechanism to tackle the self-attention bottleneck with [Transformer models](https://paperswithcode.com/methods/category/transformers). The original [scaled dot-product attention](https://paperswithcode.com/method/scaled) is decomposed into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention.

Linformer

Linformer: Self-Attention with Linear Complexity

**Dorylus** is a distributed system for training graph neural networks which uses cheap CPU servers and Lambda threads. It scales to
large billion-edge graphs with low-cost cloud resources.

Dorylus

Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads

**MODERN**, or **Modulated Residual Network**, is an architecture for [visual question answering](https://paperswithcode.com/task/visual-question-answering) (VQA). It employs [conditional batch normalization](https://paperswithcode.com/method/conditional-batch-normalization) to allow a linguistic embedding from an [LSTM](https://paperswithcode.com/method/lstm) to modulate the [batch normalization](https://paperswithcode.com/method/batch-normalization) parameters of a [ResNet](https://paperswithcode.com/method/resnet). This enables the linguistic embedding to manipulate entire feature maps by scaling them up or down, negating them, or shutting them off, etc.

Viet-Anh on Software

AI Glossary

Tech Terminology

Pathology Language and Image Pre-Training

Semantic Reasoning Network

Siamese Multi-depth Transformer-based Hierarchical Encoder

AccoMontage

Spatially-Adaptive Normalization