**ESACL**, or **Enhanced Seq2Seq Autoencoder via Contrastive Learning**, is a denoising sequence-to-sequence (seq2seq) autoencoder via contrastive learning for abstractive text summarization. The model adopts a standard [Transformer](https://paperswithcode.com/method/transformer)-based architecture with a multilayer bi-directional encoder and an autoregressive decoder. To enhance its denoising ability, self-supervised contrastive learning is incorporated along with various sentence-level document augmentation.

**Principle Components Analysis (PCA)** is an unsupervised method primary used for dimensionality reduction within machine learning.  PCA is calculated via a singular value decomposition (SVD) of the design matrix, or alternatively, by calculating the covariance matrix of the data and performing eigenvalue decomposition on the covariance matrix. The results of PCA provide a low-dimensional picture of the structure of the data and the leading (uncorrelated) latent factors determining variation in the data.

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis#/media/File:GaussianScatterPCA.svg)

ESACL

Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization

**Noisy Student Training** is a semi-supervised learning approach. It extends the idea of self-training
and distillation with the use of equal-or-larger student models and noise added to the student during learning. It has three main steps: 

1. train a teacher model on labeled images
2. use the teacher to generate pseudo labels on unlabeled images
3. train a student model on the combination of labeled images and pseudo labeled images. 

The algorithm is iterated a few times by treating the student as a teacher to relabel the unlabeled data and training a new student.

Noisy Student Training seeks to improve on self-training and distillation in two ways. First, it makes the student larger than, or at least equal to, the teacher so the student can better learn from a larger dataset. Second, it adds noise to the student so the noised student is forced to learn harder from the pseudo labels. To noise the student, it uses input noise such as [RandAugment](https://paperswithcode.com/method/randaugment) data augmentation, and model noise such as [dropout](https://paperswithcode.com/method/dropout) and [stochastic depth](https://paperswithcode.com/method/stochastic-depth) during training.

Source	Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Enhanced Seq2Seq Autoencoder via Contrastive Learning?

Viet-Anh on Software