LayerScale

Explore the latest content and insights.

LayerScale: A Small Change That Stabilizes Deep Vision Transformers
November 5, 2023
LayerScale
Vision-Transformer
Training-Stability
Glossary
LayerScale multiplies each Transformer residual branch by a learned per-channel gain initialized near zero, helping very deep image Transformers begin close to an identity mapping.