GLU

GLU

$$\text{GLU}(x)=\sigma (xG)\odot (xV)$$

$\sigma$ …sigmoid
The first half acts as a gate that decides, per-feature, how much of the second half to let through.

Compare to a vanilla MLP:

$$\begin{array}{rl}f(x)& =\varphi (W_L\cdots \varphi (W_{1}x))\\ f_{\text{GLU}}(x)& =\varphi (W_L^g[\cdots ])\odot (W_L^v[\cdots ])\end{array}$$

The computation is factorized, expressed as a product of simpler functions, which gives it combinatorial coverage of multiplicative interactions between features, which an MLP can only approximate.

But this doesn’t make GLU able to do ICL; compare to self-attention:

$$\begin{array}{rl}\text{GLU}(x)& =g(x)\odot (Vx),g(x)=\text{Swish}(Wx)\\ \text{Attn}(X)& =A(X)VX,A(X)=\text{softmax}(Q{K}^T)\end{array}$$

In both cases, one part computes fast weights, another applies those to a projection of the input.
But attention’s dynamic weight matrix mixes across tokens, while GLU is pointwise across tokens.
It’s like dynamically selecting which computation to apply.

Since we add an extra weight matrix, we scale down the parameters per layer, to keep the total count the same:

$$\underset{\text{vanilla}}{\underbrace{2k{d}^2}}=\underset{GLU}{\underbrace{3k{d}^2}}⟹k_{GLU}=\frac{2k}{3}$$

Modern version: SwiGLU, delivers consistent improvements across pre-training and downstream benchmark tasks.
Computationally efficient version ($G$ = $V$; $\varphi =\text{ReLU}$): squared ReLU

Challenge: The quadratic branching can can over long training spans produce outliers, when xG and xV align, which is a problem for FP8 training, addressed in scaling-fp8-training-to-trillion-token-llms.