chain rule of probability

Chain rule of probability

By simply rearranging the formula for conditional probability, we get:

$$\begin{array}{rl}P(A\cap B)& =P(A|B)\cdot P(B)\end{array}$$

… for $A$ and $B$ to happen, $B$ has to happen, and then $A$ has to happen given that $B$ has happened.
This always holds.

For $n$ vars:

$$P(\bigcap _{i=1}^n{A}_i)=P(A_1)\cdot P(A_2|A_1)\cdot P(A_3|A_1\cap A_2)\cdots P(A_n|\bigcap _{j=1}^{n-1}{A}_j)$$

It looks a bit nicer with random variables:

$$P(X_1,X_2,\dots ,X_n)=P(X_1)\cdot P(X_2|X_1)\cdot P(X_3|X_1,X_2)\cdots P(X_n|X_1,\dots ,X_{n-1})$$

Independent events

Two events $A$ and $B$ are independent if knowing $A$ gives no extra information about $B$, and vice versa:

$$\begin{array}{r}P(A\mid B)=P(A)\\ P(B\mid A)=P(B)\end{array}$$

Equivalently, using the formula of conditional probability, for independent events, we can say:

$$\begin{array}{rl}P(A)& =P(A\mid B)=\frac{P(A\cap B)}{P(B)}\\ ⟹P(A\cap B)& =P(A)\cdot P(B)\end{array}$$

So the chain rule of probability simplifies to multiplication for independent events.

Link to original

This works for multivariate distribution too:

Conditional probability

The conditional probability of $X$ given $Y=y$ is given by:

$$\begin{array}{rl}P(X=x\mid Y=y)& =\frac{P(x,y)}{P_Y(y)}\end{array}$$

This is almost identical to the standard conditional probability (probability of $X$ AND $Y$ divided by the probability of $Y$), but more general, as it’s a conditional density function of all values of $x$ and $y$.
This is the same formula but using density notation - $f_{X|Y}(x|y)$ denotes the conditional density function of $X$ given $Y=y$:

$$f_{X|Y}(x|y)=\frac{f(x,y)}{f_Y(y)}$$

And the chain rule of probability follows directly from this:

$$\begin{array}{rl}P(x,y)& =P(X=x\mid Y=y)\cdot P_Y(y)\\ f(x,y)& =f_{X|Y}(x|y)\cdot f_Y(y)\end{array}$$

Link to original