layer normalization

https://paperswithcode.com/paper/layer-normalization

Transclude of Normalization---Scaling---Standardization#^f74628

In contrast to batch normalization every datapoint gets normalized individually, but accross all the features (layers). Removes dependency on batch but features might have different scales.

---

Essentially BatchNorm:

Input: Values of $x$ over a mini-batch: $B=\{x_1\dots m\};$
${\mu }_B=\frac{1}{m}{\sum }_{i=1}^m{x}_i$ // mini batch mean
${\sigma }_B^2=\frac{1}{m}{\sum }_{i=1}^m(x_i-{\mu }_B{)}^2$ // mini-batch variance
${\hat{x}}_i=\frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$ // normalize
$ϵ$ … noise parameter in case variance is 0 (div by 0)

Link to original

Since mean and variance are heavily dependent on the batch, we introduce learnable parameters (unit gaussion at initialization but optimization might change that).
$\gamma$ (scale) approximates the true variance of the neuron activation and
$\beta$ (offset) apporximates the true mean of the neuron activation.
$z^{(i)}=\gamma \otimes {\hat{x}}^{(i)}+\beta$

Link to original

but better:

$$\text{LayerNorm}(\mathbf{x})\dot{=}\frac{\mathbf{x}-\mu }{\sqrt{{\sigma }^2+ϵ}},\text{where }\mu \dot{=}\frac{1}{n}\sum _{i=1}^n{x}_i\text{ and }{\sigma }^2\dot{=}\frac{1}{n}\sum _{i=1}^n(x_i-\mu {)}^2$$

where $n$ is the dimensionality of $\mathbf{x}$ and $ϵ$ is a small number used for numerical stability.

Why?

Transclude of Normalization---Scaling---Standardization#batch-normalization-vs-layer-normalization

Example

Refer to batch normalization for more explanation.

layer normalization for fully connected layer $i$, neuron $j$ and feature size $m$

$$\begin{gathered} \text{displaylines}{\text{mean}}_i=\frac{1}{m}\sum _{l=1}^m{\sigma }_{il} \\ {\text{var}}_i=\frac{1}{m}\sum _{l=1}^m({\sigma }_{il}-mea{n}_i{)}^2 \end{gathered}$$

Normalize feature $j:$

$${\hat{\sigma }}_{ij}=\frac{{\sigma }_{ij}-{\text{mean}}_i}{\sqrt{va{r}_i+ϵ}}$$

$ϵ$ … noise parameter in case var is 0, see ^cd1cd1
Scale and shift normalized feature, with learnable params:

$${\gamma }_{ij}={\hat{\sigma }}_{ij}+{\beta }_{ij}$$

source

Code

(from Karpathy)