SwiGLU

SwiGLU

GLU, but the sigmoid is swapped with Swish:

$$\text{SwiGLU}(x)=\text{Swish}(xW)\odot (xV)$$

Most modern open-weight LLMs use SwiGLU.