Transformer
Explore the latest content and insights.
Squared ReLU: The Tiny Activation Change Used by Primer
Squared ReLU computes the square of a rectified activation. This note explains why Primer used it, how it changes gradients, and when not to adopt it blindly.
ReGLU, GEGLU, and SwiGLU: Gated Feed-Forward Layers in Transformers
ReGLU, GEGLU, and SwiGLU multiply one linear projection by an activated gate, giving Transformer feed-forward layers a learned control path.