covariance

Covariance is just like correlation - it tells you how two random variables change together - with the difference that covariance is not necessarily bounded / normalized between $[-1;1]$.

In fact, the covariance matrix is the unnormalized correlation matrix.

Covariance

The covariance between two variables $X$ and $Y$ is calculated as the average of the product of their deviations from their respective means $\bar{X}$ and $\bar{Y}$.

$$\begin{array}{rl}\text{Cov}(X,Y)& =\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})\\ & =E[(X-[E[X]])(Y-E[Y])]\\ & =E[XY]-E[X]E[Y]\end{array}$$

This final version gives us a nice interpretation:
Covariance is the difference in expected value of two random variables taken from their joint behavior vs as if they are independent. So if they are independent, both terms will be the same and the covariance 0. See this.

Derivation Properties:

Rearranging, using

$$\begin{array}{rl}& =E[(X-[E[X]])(Y-E[Y])]\\ & =E[XY-E[X]Y-E[Y]X+E[X]E[Y]]\\ & =E[XY]-E[E[X]]E[Y]-E[E[Y]]E[X]+E[E[X]]E[E[Y]]\\ & =E[XY]-E[X]E[Y]-E[X]E[Y]+E[X]E[Y]\\ & =E[XY]-E[X]E[Y]\end{array}$$

Covariance can be interpreted as:

$\text{Cov}(X,Y)>0$ →when $X$ increases, $Y$ tends to increase as well, and vice versa. It’s more likely to find $X$ and $Y$ both above or both below their means, and to it’s more likely to sample $(x,y)$ where both are large/small.
$\text{Cov}(X,Y)<0$ → when $X$ increases, $Y$ tends to decrease, and vice versa. It’s less likely to find $X$ and $Y$ both above or both below their means, and to it’s more likely to sample $(x,y)$ where one is large and the other is small.
$\text{Cov}(X,Y)=0$ → no linear relationship between $X$ and $Y$, i.e. you can’t tell how large $y$ will be if you know $x$.

The covariance of a random variable and itself is its variance

$$\begin{array}{r}\text{Cov}(X,X)=\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2=\text{Var}(X)\end{array}$$

Covariance is commutative

$$\begin{array}{rl}\text{Cov}(A,B)& =\text{Cov}(B,A)\\ {\Sigma }_{AB}& ={\Sigma }_{AB}^T\end{array}$$

Limitations

Covariance doesn’t indicate the strength of the relationship, nor its causality. So if two distributions are independent, their covariance will be zero but the reverse it not true because it does not take non-linear relationsips into account.
However, for multivariate normal distribution, a zero covariance implies independence.

It is also sensitive to the scale of the variables, making it difficult to compare covariances across different datasets.

![[covariance-20240731122500640.webp|center|https://distill.pub/2019/visual-exploration-gaussian-processes/]]

Covariance Matrix

Split this + all the references into separate note…

Covariance matrix

The second moment of a random variable $X$ is $E[X^2]$ → The second moment matrix contains all pairwise $E[X_i\cdot X_j]$ – dot products ($E[X{X}^T]$).
When centered (mean subtracted), this becomes the covariance matrix (or “central second moment”):

$$\text{Covariance Matrix}=E[(X-\mu )(X-\mu {)}^T]=E[X{X}^T]-\mu {\mu }^T$$

The diagonal elements represent the variance of each variable, while the off-diagonal elements represent the covariance between pairs of variables:

$$\left[\begin{array}{ccccccc}\text{Var}(X_1)& & \text{Cov}(X_1,X_2)& & \dots & & \text{Cov}(X_1,X_n)\\ \text{Cov}(X_2,X_1)& & \text{Var}(X_2)& & \dots & & \text{Cov}(X_2,X_n)\\ ⋮& & ⋮& & \ddots & & ⋮\\ \text{Cov}(X_n,X_1)& & \text{Cov}(X_n,X_2)& & \dots & & \text{Var}(X_n)\end{array}\right]$$

The covariance matrix is always positive semidefinite.

See also multivariate gaussian distribution.

Illustrated with a simple example

How much somone likes some fruite example table (higher score means they like the fruit more):

Apple	Banana
1	1
3	0
-1	-1

The means of both random variables:

$$\begin{array}{rl}{\mu }_A& =(1+3-1)/3=1\\ {\mu }_B& =(1+0-1)/3=0\end{array}$$

The covariance matrix of these two random variables has the following form:

$$\begin{array}{rl}{K}_{AB}& =\left[\begin{array}{ccc}\text{Cov}(A,A)& & \text{Cov}(A,B)\\ \text{Cov}(B,A)& & \text{Cov}(B,B)\end{array}\right]\\ & \\ & =\left[\begin{array}{ccc}\text{Var}(A)& & \text{Cov}(A,B)\\ \text{Cov}(B,A)& & \text{Var}(B)\end{array}\right]\end{array}$$

Since, $\text{Cov}(A,B)=\text{Cov}(B,A)$ → $K_{AB}=K_{AB}^T$ (we know the elements of one triangle, we can just mirror it).

$$\begin{array}{rl}\text{Cov}(A,B)& =E[AB]-E[A]E[B]\\ & =\frac{(1\cdot 1+3\cdot 0+-1\cdot -1)}{3}-1\cdot 0=\frac{2}{3}\\ Cov(A,A)& =E[A^2]-E[A{]}^2=\frac{11}{3}-1=\frac{8}{3}\\ Cov(B,B)& =E[B^2]-0=\frac{2}{3}\\ K_{AB}& =\left[\begin{array}{ccc}\frac{8}{3}& & \frac{2}{3}\\ \frac{2}{3}& & \frac{2}{3}\end{array}\right]\end{array}$$

References

gpt
cov matrix yt