**Iterative Pseudo-Labeling** (IPL) is a semi-supervised algorithm for speech recognition which efficiently performs multiple iterations of pseudo-labeling on unlabeled data as the acoustic model evolves. In particular, IPL fine tunes an existing model at each iteration using both labeled data and a subset of unlabeled data.

Iterative Pseudo-Labeling for Speech Recognition

A scalable second order optimization algorithm for deep learning.

Optimization in machine learning, both theoretical and applied, is presently dominated by first-order gradient methods such as stochastic gradient descent. Second-order optimization methods, that involve second derivatives and/or second order statistics of the data, are far less prevalent despite strong theoretical properties, due to their prohibitive computation, memory and communication costs. In an attempt to bridge this gap between theoretical and practical optimization, we present a scalable implementation of a second-order preconditioned method (concretely, a variant of full-matrix Adagrad), that along with several critical algorithmic and numerical improvements, provides significant convergence and wall-clock time improvements compared to conventional first-order methods on state-of-the-art deep models. Our novel design effectively utilizes the prevalent heterogeneous hardware architecture for training deep models, consisting of a multicore CPU coupled with multiple accelerator units. We demonstrate superior performance compared to state-of-the-art on very large learning tasks such as machine translation with Transformers, language modeling with BERT, click-through rate prediction on Criteo, and image classification on ImageNet with ResNet-50.

Distributed Shampoo

Towards Practical Second Order Optimization for Deep Learning

Please enter a description about the method here

Attention Sinks

Efficient Streaming Language Models with Attention Sinks

**Disentangled Attribution Curves (DAC)** provide interpretations of tree ensemble methods in the form of (multivariate) feature importance curves. For a given variable, or group of variables, [DAC](https://paperswithcode.com/method/dac) plots the importance of a variable(s) as their value changes.

The Figure to the right shows an example. The tree depicts a decision tree which performs binary classification using two features (representing the XOR function). In this problem, knowing the value of one of the features without knowledge of the other feature yields no information - the classifier still has a 50% chance of predicting either class. As a result, DAC produces curves which assign 0 importance to either feature on its own. Knowing both features yields perfect information about the classifier, and thus the DAC curve for both features together correctly shows that the interaction of the features produces the model’s predictions.

Disentangled Attribution Curves

Disentangled Attribution Curves for Interpreting Random Forests and Boosted Trees

**SEER** is a self-supervised learning approach for training large models on random, uncurated images with no supervision. It trains [RegNet-Y](https://paperswithcode.com/method/regnet-y) architectures with the [SwAV](https://paperswithcode.com/method/swav). Several adjustments are made to self-supervised training to make it work at a larger scale, including using a [cosine learning schedule](https://paperswithcode.com/method/cosine-annealing)

SEER

Self-supervised Pretraining of Visual Features in the Wild

**Varifocal Loss** is a loss function for training a dense object detector to predict the IACS, inspired by [focal loss](https://paperswithcode.com/method/focal-loss). Unlike the focal loss that deals with positives and negatives equally, Varifocal Loss treats them asymmetrically.

$$ VFL\left(p, q\right) = −q\left(q\log\left(p\right) + \left(1 − q\right)\log\left(1 − p\right)\right) \text{ if } q > 0 $$

$$ VFL\left(p, q\right) = −\alpha{p^{\gamma}}\log\left(1-p\right) $$

where $p$ is the predicted IACS and $q$ is the target IoU score.

For a positive training example, $q$ is set as the IoU between the generated bounding box and the ground-truth one
(gt IoU), whereas for a negative training example, the training target $q$ for all classes is $0$.

Varifocal Loss

VarifocalNet: An IoU-aware Dense Object Detector

**lda2vec** builds representations over both words and documents by mixing word2vec’s skipgram architecture with Dirichlet-optimized sparse topic mixtures. 

The Skipgram Negative-Sampling (SGNS) objective of word2vec is modified to utilize document-wide feature vectors while simultaneously learning continuous document weights loading onto topic vectors. The total loss term $L$ is the sum of the Skipgram Negative Sampling Loss (SGNS) $L^{neg}\_{ij}$ with the addition of a Dirichlet-likelihood term over document weights, $L\_{d}$. The loss is conducted using a context vector, $\overrightarrow{c\_{j}}$ , pivot word vector $\overrightarrow{w\_{j}}$, target word vector $\overrightarrow{w\_{i}}$, and negatively-sampled word vector $\overrightarrow{w\_{l}}$:

$$ L = L^{d} + \Sigma\_{ij}L^{neg}\_{ij} $$

$$L^{neg}\_{ij} = \log\sigma\left(c\_{j}\cdot\overrightarrow{w\_{i}}\right) + \sum^{n}\_{l=0}\sigma\left(-\overrightarrow{c\_{j}}\cdot\overrightarrow{w\_{l}}\right)$$

lda2vec

Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec

Apollo

Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization

**ProxylessNet-Mobile** is a convolutional neural architecture learnt with the [ProxylessNAS](https://paperswithcode.com/method/proxylessnas) [neural architecture search](https://paperswithcode.com/method/neural-architecture-search) algorithm that is optimized for mobile devices. It uses inverted residual blocks (MBConvs) from [MobileNetV2](https://paperswithcode.com/method/mobilenetv2) as its basic building block.

ProxylessNet-Mobile

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

The **Adaptive Attention Span Transformer** is a Transformer that utilises an improvement to the self-attention layer called [adaptive masking](https://paperswithcode.com/method/adaptive-masking) that allows the model to choose its own context size. This results in a network where each attention layer gathers information on their own context. This allows for scaling to input sequences of more than 8k tokens.

Their proposals are based on the observation that, with the dense attention of a traditional [Transformer](https://paperswithcode.com/method/transformer), each attention head shares the same attention span $S$ (attending over the full context). But many attention heads can specialize to more local context (others look at the longer sequence). This motivates the need for a variant of self-attention that allows the model to choose its own context size (adaptive masking - see components).

Adaptive Span Transformer

Adaptive Attention Span in Transformers

**Model Agnostic Confidence Estimator**, or **MACEst**, is a model-agnostic confidence estimator. Using a set of nearest neighbours, the algorithm differs from other methods by estimating confidence independently as a local quantity which explicitly accounts for both aleatoric and epistemic uncertainty. This approach differs from standard calibration methods that use a global point prediction model as a starting point for the confidence estimate.

MACEst

MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator

**EfficientNet** is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a *compound coefficient*. Unlike conventional practice that arbitrary scales  these factors, the EfficientNet scaling method uniformly scales network width, depth, and resolution with a set of fixed scaling coefficients. For example, if we want to use $2^N$ times more computational resources, then we can simply increase the network depth by $\alpha ^ N$,  width by $\beta ^ N$, and image size by $\gamma ^ N$, where $\alpha, \beta, \gamma$ are constant coefficients determined by a small grid search on the original small model. EfficientNet uses a compound coefficient $\phi$ to uniformly scales network width, depth, and resolution in a  principled way.

The compound scaling method is justified by the intuition that if the input image is bigger, then the network needs more layers to increase the receptive field and more channels to capture more fine-grained patterns on the bigger image.

The base EfficientNet-B0 network is based on the inverted bottleneck residual blocks of [MobileNetV2](https://paperswithcode.com/method/mobilenetv2), in addition to squeeze-and-excitation blocks.

 EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters.

EfficientNet

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

An **End-to-End Memory Network** is a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of [Memory Network](https://paperswithcode.com/method/memory-network), but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol.

The model takes a discrete set of inputs $x\_{1}, \dots, x\_{n}$ that are to be stored in the memory, a query $q$, and outputs an answer $a$. Each of the $x\_{i}$, $q$, and $a$ contains symbols coming from a dictionary with $V$ words. The model writes all $x$ to the memory up to a fixed buffer size, and then finds a continuous representation for the $x$ and $q$. The continuous representation is then processed via multiple hops to output $a$.

End-To-End Memory Network

End-To-End Memory Networks

**VarifocalNet** is a method aimed at accurately ranking a huge number of candidate detections in object detection. It consists of a new loss function, named [Varifocal Loss](https://paperswithcode.com/method/varifocal-loss), for training a dense object detector to predict the IACS, and a new efficient star-shaped bounding box feature representation for estimating the IACS and refining coarse bounding boxes. Combining these two new components and a bounding box refinement branch, results in a dense object detector on the [FCOS](https://paperswithcode.com/method/fcos) architecture, what the authors call VarifocalNet or VFNet for short.

VFNet

A **Global Convolutional Network**, or **GCN**, is a semantic segmentation building block that utilizes a large kernel to help perform classification and localization tasks simultaneously. It can be used in a [FCN](https://paperswithcode.com/method/fcn)-like structure, where the [GCN](https://paperswithcode.com/method/gcn) is used to generate semantic score maps. Instead of directly using larger kernels or global [convolution](https://paperswithcode.com/method/convolution), the GCN module employs a combination of $1 \times k + k \times 1$ and $k \times 1 + 1 \times k$ convolutions, which enables [dense connections](https://paperswithcode.com/method/dense-connections) within a large
$k\times{k}$ region in the feature map

Global Convolutional Network

Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network

**EMQAP**, or **E-Manual Question Answering Pipeline**, is an approach for answering questions pertaining to electronics devices. Built upon the pretrained [RoBERTa](https://paperswithcode.com/method/roberta), it harbors a supervised multi-task learning framework which efficiently performs the dual tasks of identifying the section in the E-manual where the answer can be found and the exact answer span within that section.

EMQAP

Question Answering over Electronic Devices: A New Benchmark Dataset and a Multi-Task Learning based QA Framework

A generic way of representing and interpolating labels, which allows straightforward extension of any kind of [mixup](https://paperswithcode.com/method/mixup) to deep metric learning for a large class of loss functions.

Metrix

It Takes Two to Tango: Mixup for Deep Metric Learning

**Anycost GAN** is a type of generative adversarial network for image synthesis and editing. Given an input image, we project it into the latent space with encoder $E$ and backward optimization. We can modify the latent code with user input to edit the image. During editing, a sub-generator of small cost is used for fast and interactive preview; during idle time, the full cost generator renders the final, high-quality output. The outputs from the full and sub-generators are visually consistent during projection and editing.

Viet-Anh on Software

AI Glossary

Tech Terminology

Iterative Pseudo-Labeling