**MoGA-B** is a convolutional neural network optimized for mobile latency and discovered via Mobile GPU-Aware (MoGA) [neural architecture search](https://paperswithcode.com/method/neural-architecture-search). The basic building block is MBConvs (inverted residual blocks) from [MobileNetV2](https://paperswithcode.com/method/mobilenetv2). Squeeze-and-excitation layers are also experimented with.

MoGA-B

MoGA: Searching Beyond MobileNetV3

**LOGAN** is a generative adversarial network that uses a latent optimization approach using [natural gradient descent](https://paperswithcode.com/method/natural-gradient-descent) (NGD). For the Fisher matrix in NGD, the authors use the empirical Fisher $F'$ with Tikhonov damping:

$$ F' = g \cdot g^{T} + \beta{I} $$

They also use Euclidian Norm regularization for the optimization step.

For LOGAN's base architecture, [BigGAN-deep](https://paperswithcode.com/method/biggan-deep) is used with a few modifications: increasing the size of the latent source from $186$ to $256$, to compensate the randomness of the source lost
when optimising $z$. 2, using the uniform distribution $U\left(−1, 1\right)$ instead of the standard normal distribution $N\left(0, 1\right)$ for $p\left(z\right)$ to be consistent with the clipping operation, using  leaky [ReLU](https://paperswithcode.com/method/relu) (with the slope of 0.2 for the negative part) instead of ReLU as the non-linearity for smoother gradient flow for $\frac{\delta{f}\left(z\right)}{\delta{z}}$ .

LOGAN

LOGAN: Latent Optimisation for Generative Adversarial Networks

Bi-attention employs the attention-in-attention (AiA) mechanism to capture second-order statistical information: the outer point-wise channel attention vectors are computed from the output of the inner channel attention.

Bi-attention

Bilinear Attention Networks for Person Retrieval

Differentiable simultaneous optimization of hyperparameters and neural network architecture. Also a [Neural Architecture Search](https://paperswithcode.com/method/neural-architecture-search) (NAS) method.

Differentiable Hyperparameter Search

sharpDARTS: Faster and More Accurate Differentiable Architecture Search

**TD-Gammon** is a game-learning architecture for playing backgammon. It involves the use of a $TD\left(\lambda\right)$ learning algorithm and a feedforward neural network.

Credit: [Temporal Difference Learning and
TD-Gammon](https://cling.csd.uwo.ca/cs346a/extra/tdgammon.pdf)

TD-Gammon

RAM adopts RNNs and reinforcement learning (RL) to make the network learn where to pay attention.

Recurrent Models of Visual Attention

The composed video retrieval (CoVR) task is a new task, where the goal is to find a video that matches both a query image and a query text. The query image represents a visual concept that the user is interested in, and the query text specifies how the concept should be modified or refined. For example, given an image of a fountain and the text _during show at night_, the CoVR task is to retrieve a video that shows the fountain at night with a show.

CoVR

CoVR: Learning Composed Video Retrieval from Web Video Captions

Multi-heads of both self and cross attentions

Rendezvous

Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos

**ProphetNet** is a sequence-to-sequence pre-training model that introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by $n$-step ahead prediction that predicts the next $n$ tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and further help predict multiple future tokens.

ProphetNet

ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training

**SpecGAN** is a generative adversarial network method for spectrogram-based, frequency-domain audio generation. The problem is suited for GANs designed for image generation. The model can be approximately inverted. 

To process audio into suitable spectrograms, the authors perform the short-time Fourier transform with 16 ms windows and 8ms stride, resulting in 128 frequency bins, linearly spaced from 0 to 8 kHz. They take the magnitude of the resultant spectra and scale amplitude values logarithmically to better-align with human perception. They then normalize each frequency bin to have zero mean and unit variance. They clip the spectra to $3$ standard deviations and rescale to $\left[−1, 1\right]$.

They then use the [DCGAN](https://paperswithcode.com/method/dcgan) approach on the result spectra.

SpecGAN

Adversarial Audio Synthesis

**Differential Diffusion** is an enhancement of image-to-image diffusion models that adds the ability to control the amount of change applied to each image fragment via a change map.

Differential Diffusion

Differential Diffusion: Giving Each Pixel Its Strength

A **Contractive Autoencoder** is an autoencoder that adds a penalty term to the classical reconstruction cost function. This penalty term corresponds to the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input. This penalty term results in a localized space contraction which in turn yields robust features on the activation layer. The penalty helps to carve a representation that better captures the local directions of variation dictated by the data, corresponding to a lower-dimensional non-linear manifold, while being more invariant to the vast majority of directions orthogonal to the manifold.

Contractive Autoencoder

**ShuffleNet v2** is a convolutional neural network optimized for a direct metric (speed) rather than indirect metrics like FLOPs. It builds upon [ShuffleNet v1](https://paperswithcode.com/method/shufflenet), which utilised pointwise group convolutions, bottleneck-like structures, and a [channel shuffle](https://paperswithcode.com/method/channel-shuffle) operation. Differences are shown in the Figure to the right, including a new channel split operation and moving the channel shuffle operation further down the block.

ShuffleNet v2

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

**Progressive Neural Architecture Search**, or **PNAS**, is a method for learning the structure of convolutional neural networks (CNNs). It uses a sequential model-based optimization (SMBO) strategy, where we search the space of cell structures, starting with simple (shallow) models and progressing to complex ones, pruning out unpromising structures as we go. 

At iteration $b$ of the algorithm, we have a set of $K$ candidate cells (each of size $b$ blocks), which we train and evaluate on a dataset of interest. Since this process is expensive, PNAS also learns a model or surrogate function which can predict the performance of a structure without needing to train it. We then expand the $K$ candidates of size $b$ into $K' \gg K$ children, each of size $b+1$. The surrogate function is used to rank all of the $K'$ children, pick the top $K$, and then train and evaluate them. We continue in this way until $b=B$, which is the maximum number of blocks we want to use in a cell.

PNAS

Progressive Neural Architecture Search

**StreaMRAK** is a streaming version of kernel ridge regression. It divdes the problem into several levels of resolution, which allows continual refinement to the predictions.

StreaMRAK

StreaMRAK a Streaming Multi-Resolution Adaptive Kernel Algorithm

Our tracker, CenterTrack, applies a detection model to a pair of images and detections from the prior frame. Given this minimal input, CenterTrack localizes objects and predicts their associations with the previous frame. That's it. CenterTrack is simple, online (no peeking into the future), and real-time.

CenterTrack

Tracking Objects as Points

**Spectral-Normalized Identity Priors**, or **SNIP**, is a structured pruning approach that penalizes an entire [residual module](https://paperswithcode.com/method/residual-connection) in a [Transformer model](https://paperswithcode.com/method/residual-connection) toward an identity mapping. It is applicable to any structured module, including a single [attention head](https://paperswithcode.com/method/scaled), an [entire attention block](https://paperswithcode.com/method/multi-head-attention), or a [feed-forward subnetwork](https://paperswithcode.com/method/position-wise-feed-forward-layer). The method identifies and discards unimportant non-linear mappings in the [residual connections](https://paperswithcode.com/method/residual-connection) by applying a thresholding operator on the function norm. Furthermore, [spectral normalization](https://paperswithcode.com/method/spectral-normalization) to stabilize the distribution of the post-activation values of the [Transformer](https://paperswithcode.com/method/transformer) layers, further improving the pruning effectiveness of the proposed methodology.

Spectral-Normalized Identity Priors

Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior

**Informative Sample Mining Network** is a multi-stage sample training scheme for GANs to reduce sample hardness while preserving sample informativeness. Adversarial Importance Weighting is proposed to select informative samples and assign them greater weight. The authors also propose Multi-hop Sample Training to avoid the potential problems in model training caused by sample mining. Based on the principle of divide-and-conquer, the authors produce target images by multiple hops, which means the image translation is decomposed into several separated steps.

Viet-Anh on Software

AI Glossary

Tech Terminology

MoGA-B

LOGAN

Bilinear Attention

Differentiable Hyperparameter Search

TD-Gammon

Recurrent models of visual attention

Composed Video Retrieval

Multi-head of Mixed Attention