Swish

Smoother than ReLU, learnable slope like PReLU:

$${\text{Swish}}_{\beta }(x)=x\cdot \sigma (\beta x)$$

Sigmoid linear unit:

$$\text{SiLU}={\text{Swish}}_1(x)=x\cdot \sigma (x)$$