LeCun Initialization

LecCun Initialization (LeCun et al. 2002)

$$W\sim U[-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}]$$

where $n$ is the number of inputs to a neuron (“fan-in”).

LeCun Initialization

Consider a neuron receiving $n$ inputs (“fan-in”). Each input has variance 1, and each weight $w$ is sampled uniformly from $[-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}]$. The neuron computes:

$$\text{output}=w_1{x}_1+w_2{x}_2+...+w_n{x}_n$$

For each term $w_i{x}_i$:

$$\text{Var}(w_i{x}_i)=w_i^2\text{Var}(x_i)\approx (\frac{1}{\sqrt{n}}{)}^2\cdot 1=\frac{1}{n}$$

Summing $n$ such terms:

$$\text{Var}(\text{output})=n\cdot \frac{1}{n}=1$$

This maintains unit variance through the network, preventing vanishing or exploding gradients.

Link to original

References

Yann Lecun