Vision Transformer

LayerScale: A Small Change That Stabilizes Deep Vision Transformers

LayerScale multiplies each Transformer residual branch by a learned per-channel gain initialized near zero, helping very deep image Transformers begin close to an identity mapping.