Squared ReLU The squared ReLU activation function is a special case of the GLU family: SquaredReLU(x)=ReLU(xW)⊙(xW)=ReLU(xW)2