multivariate gaussian distribution

A multivariate gaussian distribution contains $n$ random variables, each of which is gaussian, as is their joint distribution.

EXAMPLE

A multivariate distribution with mean $\mu =\left[\begin{array}{c}0\\ 0\end{array}\right]$ and the covariance matrix $\Sigma =\left[\begin{array}{ccc}1& & \frac{3}{5}\\ \frac{3}{5}& & 2\end{array}\right]$

The multivariate distribution $X$ is fully described by its mean vector $\mu$ and a covariance matrix $\Sigma$.

$$X=\left[\begin{array}{c}{X}_1\\ X_2\\ ⋮\\ X_n\end{array}\right]\sim \mathcal{N}(\mu ,\Sigma )$$

The mean vector describes the expected value of the distribution. Each of its components describes the mean of the corresponding dimension.
The covariance matrix models the variance along each dimension, it describes the shape of the distribution.
The main diagonal of the covariance matrix contains the central moments of the $i$-th random variable, and the off-diagonal elements contain the covariances describing the correlation between the variables.

$$\Sigma =\text{Cov}(X_i,X_j)=E[(X_i-{\mu }_i)(X_j-{\mu }_j{)}^{}]=E[X{X}^T]-\mu {\mu }^T$$

Visualization of different covariance matrices of a bivariate gaussian distribution.

Note: There is an interactive version at distill.

Depicted are multivariate gaussian distributions with a $\mu$ of $0$ and covariance matrixes:

$\left[\begin{array}{cc}1& -0.8\\ -0.8& 1\end{array}\right]$,$\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$ and $\left[\begin{array}{cc}1& 0.8\\ 0.8& 1\end{array}\right]$

The one in the middle is a diagonal gaussian distribution: The two variables do not correlatete with eachother. The covariance matrix is zero except on the main diagonal.

Marginalization and conditioning

Gaussian distributions are closed under marginalization and conditioning, i.e. they return a modified gaussian distribution.
E.g. for the following bivariate distribution:

$$P_{X,Y}=\left[\begin{array}{c}X\\ Y\end{array}\right]\sim N(\mu ,\Sigma )=\mathcal{N}(\left[\begin{array}{c}{\mu }_X\\ {\mu }_Y\end{array}\right],\left[\begin{array}{cc}{\Sigma }_{XX}& {\Sigma }_{XY}\\ {\Sigma }_{YX}& {\Sigma }_{YY}\end{array}\right])$$

Marginalization

Marginalization lets us extract partial information from multivariate probability distributions. Given a normal probability distribution $P(X,Y)$ over vectors of random variables $X$ and $Y$, we can determine their marginalized probability distributions like this:

$$\begin{array}{rl}X& \sim \mathcal{N}({\mu }_X,\Sigma XX)\\ Y& \sim \mathcal{N}({\mu }_Y,{\Sigma }_{YY})\end{array}$$

This means that each partition $X$ and $Y$ only depends on its corresponding entries in $\mu$ and $\Sigma$.

Another way to express this, mathematically, is that we view every possible value of $X$ under the consideration of all possible values of $Y$, e.g.: $P(X=x_1)=P(X=x_1,Y=y_1)+\cdots +P(X=x_1,Y=y_n))$, everaging out Y’s contribution:

$$p_X(x)={\int }_y{p}_{X,Y}(x,y)dy={\int }_y{p}_{X|Y}(x|y)p_Y(y)dy$$

Conditioning

Conditioning is used to determine the probability distribution of one variable depending on another variable, i.e. how one variable behaves when another one is known.

$$\begin{array}{rl}X|Y& \sim \mathcal{N}({\mu }_X+{\Sigma }_{XY}{\Sigma }_{YY}^{-1}(Y-{\mu }_Y),{\Sigma }_{XX}-{\Sigma }_{XY}{\Sigma }_{YY}^{-1}{\Sigma }_{YX})\\ Y|X& \sim \mathcal{N}({\mu }_Y+{\Sigma }_{YX}{\Sigma }_{XX}^{-1}(X-{\mu }_X),{\Sigma }_{YY}-{\Sigma }_{YX}{\Sigma }_{XX}^{-1}{\Sigma }_{XY})\end{array}$$

The mean gets shifted by how much the known variable differs from its expected value $(Y-{\mu }_Y)$ , which is normalized by ${\Sigma }_{YY}^{-1}$ (think $(Y-{\mu }_y)/Y$), and scaled by the covariance between the two variables ${\Sigma }_{XY}$ . This product can be thought of as translating the normalized deviation in $Y$ to corresponding changes in $X$’s scale.
→ ${\Sigma }_{XY}{\Sigma }_{YY}^{-1}{\Sigma }_{YX}$ represents the amount of variance in $X$ that can be explained by $Y$.

Conditioning is like taking a slice of the distribution at the known value.

Visualization of marginalization and conditioning

Note: There is an interactive version at distill.

The blue curve shows the entire underlying distribution of the random variable $x_a$, and the red curve shows a slice of the joint distribution of $x_a$ and $x_b$ at a specific value of $x_b$.
i

For normal distributions, uncorrelated variables are independent if and only if they are jointly normally distributed.

A pair $(X,Y)$ is jointly normal exactly when every linear combination $aX+bY$ is normally distributed, i.e. the resulting distribution is a multivariate normal distribution.

Visual intuition $X\sim \mathcal{N}(0,1)$ and $Y=WX$ where $W$ is $\pm 1$ with equal probability:

Individually $X$ and $Y$ each look normal, but together their joint distribution is unusual: When $X$ is positive, $Y$ is either $+X$ or $-X$ with 50/50 chance, when $X$ is negative, same thing.

→ $X$ and $Y$ cannot possibly be independent, even though they’re uncorrelated!

References

Lesson 4: Multivariate Normal Distribution
probability distribution