RMS norm

Just scales the input by the root mean square of the input values.
Because RMSNorm naturally produces outputs orthogonal to the uniform vector ¹, recentering is redundant.

$$\mathrm{RMSNorm}(x;ϵ)=\gamma \odot \frac{x}{\sqrt{\mathrm{mean}(x^2)+ϵ}}$$

$\gamma$ … learnable scaling parameter (vector of same dimension as $x$)
$ϵ$ … small constant added for numerical stability to prevent division by zero.

Like layer normalization, it operates at the level of individual data points, i.e. normalizing across features/tokens rather than across a batch/sequence.

c.f. layernorm

$$\mathrm{LayerNorm}(x;\gamma ,\beta ,ϵ)=\gamma \odot \frac{x-\mu }{\sigma +ϵ}+\beta$$

Where $\mu$ and $\sigma$ are the mean and standard deviation of the features in $x$, and $\gamma$ and $\beta$ are learnable parameters for scaling and shifting.

c.f. batchnorm

$$\mathrm{BatchNorm}(x;\gamma ,\beta ,ϵ)=\gamma \odot \frac{x-{\mu }_B}{{\sigma }_B+ϵ}+\beta$$

Where ${\mu }_B$ and ${\sigma }_B$ are the mean and standard deviation computed over a batch of data, and $\gamma$ and $\beta$ are learnable parameters for scaling and shifting.

Footnotes

1. https://arxiv.org/pdf/2409.12951v1 ↩