**NPID++** (Non-Parametric Instance Discrimination) is a self-supervision approach that takes a non-parametric classification approach. It approves upon [NPID](https://paperswithcode.com/method/npid) by using more negative samples and training for more epochs.

**Natural Gradient Descent** is an approximate second-order optimisation method. It has an interpretation as optimizing over a Riemannian manifold using an intrinsic distance metric, which implies the updates are invariant to transformations such as whitening. By using the positive semi-definite (PSD) Gauss-Newton matrix to approximate the (possibly negative definite) Hessian, NGD can often work better than exact second-order methods.

Given the gradient of $z$, $g = \frac{\delta{f}\left(z\right)}{\delta{z}}$, NGD computes the update as:

$$\Delta{z} = \alpha{F}^{−1}g$$

where the Fisher information matrix $F$ is defined as:

$$ F = \mathbb{E}\_{p\left(t\mid{z}\right)}\left[\nabla\ln{p}\left(t\mid{z}\right)\nabla\ln{p}\left(t\mid{z}\right)^{T}\right] $$

The log-likelihood function $\ln{p}\left(t\mid{z}\right)$ typically corresponds to commonly used error functions such as the cross entropy loss.

Source: [LOGAN](https://paperswithcode.com/method/logan)

Image: [Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
](https://arxiv.org/abs/1905.10961)

Natural Gradient Descent

NPID++

Self-Supervised Learning of Pretext-Invariant Representations

A **FuseFormer block** is used in the [FuseFormer](https://paperswithcode.com/method/fuseformer) model for video inpainting. It is the same to standard [Transformer](https://paperswithcode.com/method/transformer) block except that feed forward network is replaced with a Fusion Feed Forward Network (F3N). F3N brings no extra parameter into the standard feed forward net and the difference is that F3N inserts a soft-split and a soft composite operation between the two layer of MLPs.

Source	Self-Supervised Learning of Pretext-Invariant Representations
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com