layer normalization

https://paperswithcode.com/paper/layer-normalization

Types of normalization techniques in machine learning - comparison

Link to original

In contrast to batch normalization every datapoint gets normalized individually, but accross all the features (layers). Removes dependency on batch but features might have different scales.

---

Essentially BatchNorm:

Input: Values of $x$ over a mini-batch: $B=\{x_1\dots m\};$
${\mu }_B=\frac{1}{m}{\sum }_{i=1}^m{x}_i$ // mini batch mean
${\sigma }_B^2=\frac{1}{m}{\sum }_{i=1}^m(x_i-{\mu }_B{)}^2$ // mini-batch variance
${\hat{x}}_i=\frac{x_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$ // normalize
$ϵ$ … noise parameter in case variance is 0 (div by 0)

Link to original

Since mean and variance are heavily dependent on the batch, we introduce learnable parameters (unit gaussion at initialization but optimization might change that).
$\gamma$ (scale) approximates the true variance of the neuron activation and
$\beta$ (offset) apporximates the true mean of the neuron activation.
$z^{(i)}=\gamma \otimes {\hat{x}}^{(i)}+\beta$

Link to original

but better:

$$\text{LayerNorm}(\mathbf{x})\dot{=}\frac{\mathbf{x}-\mu }{\sqrt{{\sigma }^2+ϵ}},\text{where }\mu \dot{=}\frac{1}{n}\sum _{i=1}^n{x}_i\text{ and }{\sigma }^2\dot{=}\frac{1}{n}\sum _{i=1}^n(x_i-\mu {)}^2$$

where $n$ is the dimensionality of $\mathbf{x}$ and $ϵ$ is a small number used for numerical stability.

Why?

Batch Normalization vs Layer Normalization

Batch normalization is dependent on the batch size, as it normalizes on all the batches, whereas Layer Normalization normalizes per data point. (Batch Norm only good with bigger batch sizes $\approx$>8)
Layer Normalization also performs the same operations during training and testing. (BN doesn’t)

Layer Norm good for:

• sequences
• variable batch num workers
• parallelization

Not so good for:

• CNNs
• Features/Layers with different scales

Link to original

Example

Refer to batch normalization for more explanation.

layer normalization for fully connected layer $i$, neuron $j$ and feature size $m$

$$\begin{gathered} \text{displaylines}{\text{mean}}_i=\frac{1}{m}\sum _{l=1}^m{\sigma }_{il} \\ {\text{var}}_i=\frac{1}{m}\sum _{l=1}^m({\sigma }_{il}-mea{n}_i{)}^2 \end{gathered}$$

Normalize feature $j:$

$${\hat{\sigma }}_{ij}=\frac{{\sigma }_{ij}-{\text{mean}}_i}{\sqrt{va{r}_i+ϵ}}$$

$ϵ$ … noise parameter in case var is 0, see ^cd1cd1
Scale and shift normalized feature, with learnable params:

$${\gamma }_{ij}={\hat{\sigma }}_{ij}+{\beta }_{ij}$$

source

Code

(from Karpathy)