Squared ReLU

Squared ReLU: The Tiny Activation Change Used by Primer

Squared ReLU computes the square of a rectified activation. This note explains why Primer used it, how it changes gradients, and when not to adopt it blindly.