squared ReLU

Squared ReLU

The squared ReLU activation function is a special case of the GLU family:

$$\begin{array}{rl}\text{SquaredReLU}(x)& =\text{ReLU}(xW)\odot (xW)\\ & =\text{ReLU}(xW{)}^2\end{array}$$