Training Stability
Explore the latest content and insights.
LayerScale: A Small Change That Stabilizes Deep Vision Transformers
LayerScale multiplies each Transformer residual branch by a learned per-channel gain initialized near zero, helping very deep image Transformers begin close to an identity mapping.