variance

The variance is the squared mean deviation / distance of each datapoint from the overall mean of the dataset $\bar{X}$

$$\begin{array}{rl}\text{Var}(X)& =\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2\end{array}$$

It measures how spread out the distribution is. Squaring the distance is nicer - mathematically - than taking absolute values. This causes the units to be off, which is corrected in the standard deviation.

Variance with expected value notation:

$$\begin{array}{rl}Var(X)& =E\left[(X-E[X]{)}^2\right]\\ & =E[X^2-2XE[X]+E[X{]}^2]\\ & =E[X^2]-2E[X]E[X]+E[X{]}^2\\ & =E[X^2]-2E[X{]}^2+E[X{]}^2\\ & =E[X^2]-E[X{]}^2\end{array}$$

With bra-ket notation:

$$\text{Var}(X)=⟨X^2⟩-⟨X{⟩}^2$$

Variance is the second central moment.

$E[X^2]$ is the second moment. When data isn’t centered at zero, $E[X^2]$ gets artificially inflated by the mean’s distance from zero.

Example: Heights of 175cm, 180cm, 185cm (mean = 180cm)

• $E[X^2]=\frac{175^2+180^2+185^2}{3}=32,350$ (huge!)
• $E[X{]}^2=180^2=32,400$ (the artificial inflation from being centered at 180)
• $\text{Var}(X)=32,350-32,400=25$ (just the ±5cm spread)

Mathematically, we can decompose any $x^2$:
$x^2=((x-\mu )+\mu {)}^2=(x-\mu {)}^2+2\mu (x-\mu )+{\mu }^2$

Averaging over all $x$:
$E[X^2]=E[(X-\mu {)}^2]+{\mu }^2=\text{Var}(X)+E[X{]}^2$

So $E[X^2]$ contains both the actual spread (variance) and the “artificial inflation” (${\mu }^2$) from being far from zero. Subtracting $E[X{]}^2$ removes this inflation, isolating just the variance.

When scaling a random variable $X$ by a constant $c$:

$$\text{Var}(cX)=c^2\text{Var}(X)$$

This follows directly from the definition of variance:

$$\begin{array}{rl}\text{Var}(cX)& =E[(cX-c\mu {)}^2]\\ & =E[c^2(X-\mu {)}^2]\\ & =c^{2}E[(X-\mu {)}^2]\\ & =c^2\text{Var}(X)\end{array}$$

For independent random variables, variances add:

$$\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$$

Counter-example: $\text{Var}(2X)=4\text{Var}(X)\ne 2\text{Var}(X)$

LeCun Initialization

Consider a neuron receiving $n$ inputs (“fan-in”). Each input has variance 1, and each weight $w$ is sampled uniformly from $[-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}]$. The neuron computes:

$$\text{output}=w_1{x}_1+w_2{x}_2+...+w_n{x}_n$$

For each term $w_i{x}_i$:

$$\text{Var}(w_i{x}_i)=w_i^2\text{Var}(x_i)\approx (\frac{1}{\sqrt{n}}{)}^2\cdot 1=\frac{1}{n}$$

Summing $n$ such terms:

$$\text{Var}(\text{output})=n\cdot \frac{1}{n}=1$$

This maintains unit variance through the network, preventing vanishing or exploding gradients.