Nyströmformer replaces the self-attention in [BERT](https://paperswithcode.com/method/bert)-small and BERT-base using the proposed Nyström approximation. This reduces self-attention complexity to $O(n)$ and allows the [Transformer](https://paperswithcode.com/method/transformer) to support longer sequences.

ChebNet involves a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs.

Description from: [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://arxiv.org/pdf/1606.09375.pdf)

ChebNet

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Nyströmformer

Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

The **Truncation Trick** is a latent sampling procedure for generative adversarial networks, where we sample $z$ from a truncated normal (where values which fall outside a range are resampled to fall inside that range). 
The original implementation was in [Megapixel Size Image Creation with GAN](https://paperswithcode.com/paper/megapixel-size-image-creation-using).
In [BigGAN](http://paperswithcode.com/method/biggan), the authors find this provides a boost to the Inception Score and FID.

Source	Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com

Viet-Anh on Software

What is: Nyströmformer?

Viet-Anh on Software