CPE is an effective collaborative metric learning to effectively address the problem of sparse and insufficient preference supervision from the margin distribution point-of-view.

**Weight Tying** improves the performance of language models by tying (sharing) the weights of the embedding and [softmax](https://paperswithcode.com/method/softmax) layers. This method also massively reduces the total number of parameters in the language models that it is applied to. 

Language models are typically comprised of an embedding layer, followed by a number of [Transformer](https://paperswithcode.com/method/transformer) or [LSTM](https://paperswithcode.com/method/lstm) layers, which are finally followed by a softmax layer. Embedding layers learn word representations, such that similar words (in meaning) are represented by vectors that are near each other (in cosine distance). [Press & Wolf, 2016] showed that the softmax matrix, in which every word also has a vector representation, also exhibits this property. This leads them to propose to share the softmax and embedding matrices, which is done today in nearly all language models.  

This method was independently introduced by [Press & Wolf, 2016](https://paperswithcode.com/paper/using-the-output-embedding-to-improve) and [Inan et al, 2016](https://paperswithcode.com/paper/tying-word-vectors-and-word-classifiers-a).

Additionally, the Press & Wolf paper proposes Three-way Weight Tying, a method for NMT models in which the embedding matrix for the source language, the embedding matrix for the target language, and the softmax matrix for the target language are all tied. That method has been adopted by the Attention Is All You Need model and many other neural machine translation models.

Weight Tying

Using the Output Embedding to Improve Language Models

Collaborative Preference Embedding against Sparse Labels

**k-Means Clustering** is a clustering algorithm that divides a training set into $k$ different clusters of examples that are near each other. It works by initializing $k$ different centroids {$\mu\left(1\right),\ldots,\mu\left(k\right)$} to different values, then alternating between two steps until convergence:

(i) each training example is assigned to cluster $i$ where $i$ is the index of the nearest centroid $\mu^{(i)}$

(ii) each centroid $\mu^{(i)}$ is updated to the mean of all training examples $x^{(j)}$ assigned to cluster $i$.

Text Source: Deep Learning, Goodfellow et al

Image Source: [scikit-learn](https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_digits.html)

Source	Collaborative Preference Embedding against Sparse Labels
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Collaborative Preference Embedding?

Viet-Anh on Software