The **GAN Hinge Loss** is a hinge loss based loss function for [generative adversarial networks](https://paperswithcode.com/methods/category/generative-adversarial-networks):

$$ L\_{D} = -\mathbb{E}\_{\left(x, y\right)\sim{p}\_{data}}\left[\min\left(0, -1 + D\left(x, y\right)\right)\right] -\mathbb{E}\_{z\sim{p\_{z}}, y\sim{p\_{data}}}\left[\min\left(0, -1 - D\left(G\left(z\right), y\right)\right)\right] $$

$$ L\_{G} = -\mathbb{E}\_{z\sim{p\_{z}}, y\sim{p\_{data}}}D\left(G\left(z\right), y\right) $$

Forward gradients are unbiased estimators of the gradient $\nabla f(\theta)$ for a function $f: \mathbb{R}^n \rightarrow \mathbb{R}$, given by $g(\theta) = \langle \nabla f(\theta) , v \rangle v$. 

Here $v = (v_1, \ldots, v_n)$ is a random vector, which must satisfy the following conditions in order for $g(\theta)$ to be an unbiased estimator of $\nabla f(\theta)$

* $v_i \perp v_j$ for all $i \neq j$
* $\mathbb{E}[v_i] = 0$ for all $i$
* $\mathbb{V}[v_i] = 1$ for all $i$

Forward gradients can be computed with a single jvp (Jacobian Vector Product), which enables the use of the forward mode of autodifferentiation instead of the usual reverse mode, which has worse computational characteristics.

Forward gradient

Gradients without Backpropagation

GAN Hinge Loss

Geometric GAN

**CoOp**, or **Context Optimization**, is an automated prompt engineering method that avoids manual prompt tuning by modeling context words with continuous vectors that are end-to-end learned from data. The context could be shared among all classes or designed to be class-specific. During training, we simply minimize the prediction error using the cross-entropy loss with respect to the learnable context vectors, while keeping the pre-trained parameters fixed. The gradients can be back-propagated all the way through the text encoder, distilling the rich knowledge encoded in the parameters for learning task-relevant context.

Source	Geometric GAN
Year	2000
Data Source	Papers with Code archive (CC BY-SA)