ReGLU, GEGLU, and SwiGLU: Gated Feed-Forward Layers in Transformers

ReGLU, GEGLU, and SwiGLU are gated alternatives to the ordinary activation inside a Transformer feed-forward network. They compute two projections of the same input, activate one projection, and multiply the two element by element.

For projections $a=xW_a+b_a$ and $b=xW_b+b_b$:

$$\operatorname{ReGLU}(x) = a \odot \operatorname{ReLU}(b),$$ $$\operatorname{GEGLU}(x) = a \odot \operatorname{GELU}(b),$$ $$\operatorname{SwiGLU}(x) = a \odot \operatorname{SiLU}(b).$$

The gated hidden state is then projected back to the model dimension.

The multiplication is the important part. Calling SiLU or GELU by itself is not SwiGLU or GEGLU.

Compare parameter budgets fairly

A conventional FFN has two large matrices; a gated FFN has three. Keeping the same hidden width therefore increases parameters and compute. For an apples-to-apples comparison, reduce the gated hidden width—commonly toward two-thirds of the original FFN width—then round to a hardware-friendly multiple.

The exact width should follow the model recipe. “SwiGLU is better” is not meaningful if the SwiGLU model quietly receives a larger parameter budget.

Which variant should I use?

The original GLU-variant study found that several gated variants improved quality over standard ReLU or GELU in its T5-style denoising pretraining and language-understanding experiments. Later language-model families adopted SwiGLU, but the best choice still depends on architecture, scale, optimizer, and runtime support.

• ReGLU: uses ReLU in the gating branch, so negative gate values become zero.
• GEGLU: smooth GELU gate; common in encoder-decoder Transformers.
• SwiGLU: smooth SiLU/Swish gate; common in modern decoder-only language models.

On deployment hardware, the extra projection and elementwise gate can change latency even when model quality improves. Measure the complete model rather than choosing from the activation name alone.

Primary source

• Noam Shazeer, GLU Variants Improve Transformer, 2020.