covariance

Covariance is just like correlation - it tells you how two random variables change together - with the difference that covariance is not necessarily bounded / normalized between $[-1;1]$.

In fact, the covariance matrix is the unnormalized correlation matrix.

The covariance between two variables $X$ and $Y$ is calculated as the average of the product of their deviations from their respective means $\bar{X}$ and $\bar{Y}$.

$$\begin{array}{rl}\text{Cov}(X,Y)& =\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})\\ & =E[(X-[E[X]])(Y-E[Y])]\\ & =E[XY]-E[X]E[Y]\end{array}$$

Derivation Properties:

$$\begin{array}{rl}& =E[(X-[E[X]])(Y-E[Y])]\\ & =E[XY-E[X]Y-E[Y]X+E[X]E[Y]]\\ & =E[XY]-E[E[X]]E[Y]-E[E[Y]]E[X]+E[E[X]]E[E[Y]]\\ & =E[XY]-E[X]E[Y]-E[X]E[Y]+E[X]E[Y]\\ & =E[XY]-E[X]E[Y]\end{array}$$

Covariance can be interpreted as:

$\text{Cov}(X,Y)>0$ →when $X$ increases, $Y$ tends to increase as well, and vice versa.
$\text{Cov}(X,Y)<0$ → when $X$ increases, $Y$ tends to decrease, and vice versa.
$\text{Cov}(X,Y)=0$ → no linear relationship between $X$ and $Y$.

The covariance of a random variable and itself is its variance

$$\begin{array}{r}\text{Cov}(X,X)=\frac{1}{n}\sum _{i=1}^n(X_i-\bar{X}{)}^2=\text{Var}(X)\end{array}$$

The order of the variables does not matter (commutative property)

$$\begin{array}{rl}\text{Cov}(A,B)& =\text{Cov}(B,A)\\ {\Sigma }_{AB}& ={\Sigma }_{AB}^T\end{array}$$

Limitations

Covariance does not indicate the strength of the relationship, nor its causality. So if two distributions are independent, their covariance will be zero but the reverse it not true because it does not take non-linear relationsips into account. However, if distributions are multivarate normal, a zero covariance implies independence.
It is also sensitive to the scale of the variables, making it difficult to compare covariances across different datasets.

Covariance Matrix

Covariance matrix

The second moment of a random variable $X$ is $E[X^2]$ → The second moment matrix contains all pairwise $E[X_i\cdot X_j]$ – dot products ($E[X{X}^T]$).
When centered (mean subtracted), this becomes the covariance matrix (or “central second moment”):

$$\text{Covariance Matrix}=E[(X-\mu )(X-\mu {)}^T]=E[X{X}^T]-\mu {\mu }^T$$

See also multivariate gaussian distribution for a more detailed and visual overview.

Illustrated with a simple example

How much somone likes some fruite example table (higher score means they like the fruit more):

Apple	Banana
1	1
3	0
-1	-1

The means of both random variables:

$$\begin{array}{rl}{\mu }_A& =(1+3-1)/3=1\\ {\mu }_B& =(1+0-1)/3=0\end{array}$$

The covariance matrix of these two random variables has the following form:

$$\begin{array}{rl}{K}_{AB}& =\left[\begin{array}{ccc}\text{Cov}(A,A)& & \text{Cov}(A,B)\\ \text{Cov}(B,A)& & \text{Cov}(B,B)\end{array}\right]\\ & \\ & =\left[\begin{array}{ccc}\text{Var}(A)& & \text{Cov}(A,B)\\ \text{Cov}(B,A)& & \text{Var}(B)\end{array}\right]\end{array}$$

Since, $\text{Cov}(A,B)=\text{Cov}(B,A)$ → $K_{AB}=K_{AB}^T$ (we know the elements of one triangle, we can just mirror it).

$$\begin{array}{rl}\text{Cov}(A,B)& =E[AB]-E[A]E[B]\\ & =\frac{(1\cdot 1+3\cdot 0+-1\cdot -1)}{3}-1\cdot 0=\frac{2}{3}\\ Cov(A,A)& =E[A^2]-E[A{]}^2=\frac{11}{3}-1=\frac{8}{3}\\ Cov(B,B)& =E[B^2]-0=\frac{2}{3}\\ K_{AB}& =\left[\begin{array}{ccc}\frac{8}{3}& & \frac{2}{3}\\ \frac{2}{3}& & \frac{2}{3}\end{array}\right]\end{array}$$

References

gpt
cov matrix yt