Viet-Anh on Software Logo

What is: Root Mean Square Layer Normalization?

SourceRoot Mean Square Layer Normalization
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm.