law of large numbers

Law of Large Numbers

Given a sequence of $n$ iid random variables $X_1,X_2,\dots ,X_n$ , the sample mean ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ will converge to $\mu$ as $n\to \infty$.
Formally:

$$\underset{n\to \infty }{\lim }P(|{\bar{X}}_n-\mu |>ϵ)=0,\text{for any }ϵ>0$$

Note: The variances don’t actually have to be identical.
This law holds for many but not all distributions, e.g. the cauchy distribution with undefined mean, or distributions with infinite mean, like the pareto distribution with $\alpha \le 1$.

The Central Limit Theorem limit theorem is stronger, also describing the shape the distribution will take.

Proof, handwavy:

$$E[{\bar{X}}_n]=E\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]=\frac{1}{n}\sum _{i=1}^{n}E[X_i]=\mu$$ $$Var({\bar{X}}_n)=Var\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)=\frac{1}{n^2}\sum _{i=1}^{n}Var(X_i)=\frac{{\sigma }^2}{n}$$

As $n\to \infty$, $Var({\bar{X}}_n)\to 0$.
→ Averaging doesn’t change the mean, but it reduces noise.

Proof, rigorous:
By chebyshev inequality, for any $ϵ>0$:

$$\begin{array}{rl}P(|{\bar{X}}_n-\mu |\ge ϵ)& \le \frac{Var({\bar{X}}_n)}{{ϵ}^2}\\ & =\frac{{\sigma }^2}{n{ϵ}^2}\text{variance of sample mean }=\frac{{\sigma }^2}{n}\\ & \to 0\text{as }n\to \infty \end{array}$$

Strong Law of Large Numbers

The strong version of the law holds even if the variables are not identically distributed, as long as they are independent/uncorrelated and satisfy ${\sum }_i^{\infty }\text{Var}(X_i)/i^2<\infty$ → The sample mean converges to the true mean almost surely (with probability 1):

$${\bar{X}}_n\stackrel{a.s.}{\to }\mu \text{as }n\to \infty ⟺P\left(\underset{n\to \infty }{\lim }{\bar{X}}_n=\mu \right)=1$$