The methon to overcome catastrophic forgetting in neural network while continual learning

An **Invertible Rescaling Network (IRN)** is a network for image rescaling.  According to the Nyquist-Shannon sampling theorem, high-frequency contents are lost during downscaling. Ideally, we hope to keep all lost information to perfectly recover the original HR image, but storing or transferring the high-frequency information is unacceptable. In order to well address this challenge, the Invertible Rescaling Net (IRN) captures some knowledge on the lost information in the form of its distribution and embeds it into model’s parameters to mitigate the ill-posedness. Given an HR image $x$, IRN not only downscales it into a LR image y, but also embeds the case-specific high-frequency content into an auxiliary case-agnostic latent variable $z$, whose marginal distribution
obeys a fixed pre-specified distribution (e.g., isotropic Gaussian). Based on this model,
we use a randomly drawn sample of $z$ from the pre-specified distribution for the inverse upscaling procedure, which holds the most information that one could have in upscaling.

Invertible Image Rescaling

Overcoming catastrophic forgetting in neural networks

**Self-Cure Network**, or **SCN**, is a method for suppressing uncertainties for large-scale facial expression recognition, prventing deep networks from overfitting uncertain facial images. Specifically, SCN suppresses the uncertainty from two different aspects: 1) a self-attention mechanism over mini-batch to weight each training sample with a ranking regularization, and 2) a careful relabeling mechanism to modify the labels of these samples in the lowest-ranked group.

Source	Overcoming catastrophic forgetting in neural networks
Year	2000
Data Source	Papers with Code archive (CC BY-SA)