variance

The variance is the squared mean deviation / distance of each datapoint from the overall mean of the dataset $\bar{X}$

$$\begin{array}{rl}\text{Var}(X)& =\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2\end{array}$$

It measures how spread out the distribution is. Squaring the distance is nicer - mathematically - than taking absolute values.

Squared distance to the mean of the distribution, but the units are off, because it’s squared… which is corrected in the standard deviation.

Variance with expected value notation:

$$\text{displaylines}\begin{array}{rl}Var(X)& =E\left[(X-E[X]{)}^2\right]\\ & =E[X^2-2XE[X]+E[X{]}^2]\\ & =E[X^2]-2E[X]E[X]+E[X{]}^2\\ & =E[X^2]-2E[X{]}^2+E[X{]}^2\\ & =E[X^2]-E[X{]}^2\end{array}$$

$E[X^2]$ is also called the second moment.

With bra-ket notation:

$$\begin{array}{rl}\text{Var}(X)& =⟨X^2⟩-⟨X{⟩}^2\end{array}$$

When scaling a random variable $X$ by a constant $c$:

$$\text{Var}(cX)=c^2\text{Var}(X)$$

This follows directly from the definition of variance:

$$\begin{array}{rl}\text{Var}(cX)& =E[(cX-c\mu {)}^2]\\ & =E[c^2(X-\mu {)}^2]\\ & =c^{2}E[(X-\mu {)}^2]\\ & =c^2\text{Var}(X)\end{array}$$

For independent random variables, variances add:

$$\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$$

LeCun Initialization

Consider a neuron receiving $n$ inputs (“fan-in”). Each input has variance 1, and each weight $w$ is sampled uniformly from $[-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}]$. The neuron computes:

$$\text{output}=w_1{x}_1+w_2{x}_2+...+w_n{x}_n$$

For each term $w_i{x}_i$:

$$\text{Var}(w_i{x}_i)=w_i^2\text{Var}(x_i)\approx (\frac{1}{\sqrt{n}}{)}^2\cdot 1=\frac{1}{n}$$

Summing $n$ such terms:

$$\text{Var}(\text{output})=n\cdot \frac{1}{n}=1$$

This maintains unit variance through the network, preventing vanishing or exploding gradients.