Viet-Anh on Software Logo

What is: Root Mean Square Layer Normalization?

SourceRoot Mean Square Layer Normalization
Data SourceCC BY-SA -

RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm.