ReGLU
Explore the latest content and insights.
ReGLU, GEGLU, and SwiGLU: Gated Feed-Forward Layers in Transformers
ReGLU, GEGLU, and SwiGLU multiply one linear projection by an activated gate, giving Transformer feed-forward layers a learned control path.