probability

Steve for a 🐐ed intro.

---

Probability

“Laplace’s Rule”:

$$P(A)=\frac{|A|}{|\Omega |}$$

$|A|$ … number of events where $A$ occurs / number of ways $A$ can happen.
$|\Omega |$ … number of possible events.
→ $0\le P(A)\le 1$

Flip a fair coin twice. What’s the probability of at least one heads occuring?

$\Omega =\{hh,ht,th,tt\}$
$A=\{hh,ht,th\}$
$P(A)=\frac{3}{4}$

Probability is just combinatorics, i.e. the math of combinations and proportions

Thinking of probability geometrically, as proportions is incredibly helpful:

Pictured: The $6\times 6$ sample space formed by all possible outcomes when rolling two dice.
The event $X=5$, where $X$ is a random variable representing the dice sum can be represented as an area (yellow) in this space, and its probability $P(X=5)$ is the proportion of the area of this event to the total area of the sample space → $P(X=5)=\frac{4}{36}=\frac{1}{9}$
Generally, you can think of the sample space as a unit square (area = 1).

Roll two dice. What is the probability that at least one die is a 5?

By tedious counting:
$\Omega =\{(1,1),(1,2),\dots ,(6,6)\}$
$A=\{(5,1),(5,2),\dots ,(6,5),(6,6)\}$
$P(A)=\frac{11}{36}$
… or we take the problem apart by looking at the complement of $A$: Either the first die is a 5 or it’s not and the second die is a 5.
$P(A)=P(X_1=5)+P(X_1\ne 5)\cdot P(X_2=5)=\frac{1}{6}+\frac{5}{6}\cdot \frac{1}{6}$

Visually, it’s easy to see the relation to binomial coefficients (→ probability of getting a 5 on at least one die is the same as the probability of getting a 5 on exactly one die) and how probabilities stack up.

Probabilistic models are not just useful for truly random things, but also for things that are too complex to model exactly.

If there are $n$ people in a room, how large does $n$ need to be for at least $50\%$ chance of at least $2$ people sharing a birthday?

Each person has to compare with each other person $(n-1)+(n-2)+\cdots +2+1=\frac{n(n-1)}{2}$, but this won’t help us get to an exact probability, as there’s over-counting, etc.
$P(x\ge 2)=1-P(x=0)$, where $x$ is the number of people with the same birthday.
The expression on the left is very hard to compute as again, you need to keep track of all the combinations, overlaps, …
For the expression on the right (the complement), we just need to divide the number of unique birthdays (sampling without replacement) by the possible number of birthdays (sampling with replacement):

$$\begin{array}{r}\frac{365!}{(365-n)!}\cdot \frac{1}{365^n}=(365\cdot 364\cdots (365-(n-1)))\cdot \frac{1}{365^n}==\prod _{i=1}^n\frac{365-i}{365}\end{array}$$

>>> P_different_bday = lambda n: math.prod([(365 - i) / 365 for i in range(1, n)])
>>> [round(1 - P_different_bday(i), 3) for i in range(0,10)]
[0, 0.0, 0.003, 0.008, 0.016, 0.027, 0.04, 0.056, 0.074, 0.095]
>>> 1 - P_different_bday(360)
1.0

¹

Properties of probability

The probability function $P$ is a map from subsets of the sample space $\Omega$ (the set of all possible outcomes of a random experiment) to the real numbers:

$$\begin{array}{rl}P:2^{\Omega }& \to \mathbb{R}[0,1]\in \mathbb{R}\end{array}$$

$P(\Omega )=1$ … the probability of something happening is 1.
$P(\varnothing )=0$ … the probability of nothing happening is 0.
$A\subseteq \Omega ⟹P(A)\ge 0$ … probability of an event is always non-negative.
$A\subseteq B⟹P(A)\le P(B)$ … a subset of events has a smaller probability than the set it’s a subset of.
$P(A^c)=1-P(A)$ … the probability of $A$ not happening. Also sometimes denoted as $P(\neg A)$ (surprise, but without the log).
If $A,B\in \Omega$ are disjoint (can’t occur at the same time), the probability of either happening is the sum of the probability of each happening:
$(A\cap B)=\varnothing ⟹P(A\cup B)=P(A)+P(B)$
For non-disjoint events, we need to subtract the intersection, so we don’t count it twice (inclusion-exclusion principle).
$P(A\cup B)=P(A)+P(B)-P(A\cap B)$

The “counting” from earlier is just asking about the relative sizes of sets; proportions:
“$A$ and $B$ happened”: “$A$ or $B$ happened”:
Now it’s also clearer what happens when we multiply or add probabilities: $\cap ⟺\cdot \cup ⟺+$
With the caveat of overcounting for non-disjoint events and the union, and for the intersection: If they are not independent, we need to use the chain rule of probability (see below).

conditional probability

Conditional probability

$A,B$ are two events (outcomes) of a random experiment. The probability of an event $A$ given that another event $B$ has occurred is:

$$P(A|B)=\frac{P(A\cap B)}{P(B)}$$

If we know that $B$ has occured, $B$ becomes the new $\Omega$:
→
In this case, $A$ becomes a lot more likely, as it occupies a larger fraction of the sample space than it did before.
It’s easy to see geometrically: The reverse is not true (completely different proportions).

$P(A|B)\ne P(B|A)$

Link to original

independent

Independent events

Two events $A$ and $B$ are independent if knowing $A$ gives no extra information about $B$, and vice versa:

$$\begin{array}{r}P(A\mid B)=P(A)\\ P(B\mid A)=P(B)\end{array}$$

Equivalently, using the formula of conditional probability, for independent events, we can say:

$$\begin{array}{rl}P(A)& =P(A\mid B)=\frac{P(A\cap B)}{P(B)}\\ ⟹P(A\cap B)& =P(A)\cdot P(B)\end{array}$$

So the chain rule of probability simplifies to multiplication for independent events.

EXAMPLE

$\Omega$ = deck of 52 cards
$A$ = card is spade → $P(A)=\frac{1}{4}$
$B$ = card is queen → $P(B)=\frac{1}{13}$
$P(A\cap B)=P(A)P(B)=\frac{1}{52}$
The probability of getting a spade or queen doesn’t change if we restrict ourselves to the set $A$ or $B$:

Independent random variables

Two random variables are independent:

$$X\perp Y⟺p(X,Y)=p(X)p(Y)$$

Knowing one variable gives no information about the other.

Mutual independence (more than 2 variables):

$$p(X_1,X_2,\dots ,X_n)=\prod _{i=1}^{n}p(X_i)$$

Link to original

Two dice are rolled. $A$ … one dice is a 3. $B$ … the sum of the dice 6. What’s $P(A\mid B)$?

→ $P(A\mid B)=\frac{1}{5}$, since there’s one out of the 5 possible events (given C), where either of the dices is a 3.

law of total probability

Law of total probability

Given a partition of the sample space $\Omega$ into $n$ disjoint events ($\Omega ={\bigcup }_{i=1}^n{B}_i,B_i\cap B_j=\varnothing$), the probability of an event $A$ is the sum of the probabilities of $A$ given each of the $B_i$, weighted by the probability of each $B_i$:

$$\begin{array}{r}P(A)=\sum _{i=1}^{n}P(A\mid B_i)\cdot P(B_i)\end{array}$$

$P(A)=P(A\mid B)P(B)+P(A\mid B^c)P(B^c)$

Link to original

Further reading:
bayes theorem
probability distribution
joint probability distribution
…

The general view / measure-theoretic scaffold:

Probability as a measure

The probability is just a measure with the additional normalization constraint that

$$P(\Omega )=1$$

Probability space: $(\Omega ,\mathcal{F},\mathbb{P})$
Random variable: $X:\Omega \to \mathbb{R}$ … a measurable function

Continue with this

https://chatgpt.com/g/g-p-68bcc91a795c8191b02e92abaf3daa0c-tutorials/c/68c084fa-88f0-8321-976e-8c44dcfad57e
https://chatgpt.com/c/68c070c4-87fc-8329-b5e4-38b7423e9d36

Continue cleaning up the below

https://chatgpt.com/c/68bf85f5-1860-8320-9794-f5b3341b5468
https://chatgpt.com/c/68c322e2-c778-832a-865b-35fed8b3d912 (last one)

A note on notation (needs refactor!)

$\Omega$ … the sample space, the set of all possible outcomes of a random experiment.
$A,B,C\subseteq \Omega$ … events.
$\mathbb{P}:\mathcal{F}\to [0,1]$ … probability measure $\mathbb{P}(A)$ aka $\mathrm{Pr}(A)$, aka $P(A)$.

$X,Y,X_i$ … random variables; $\mathbf{X}=(X_1,X_2,\dots ,X_d)$ … a vector of random variables.
$x,y,z$ … realized values; $\mathbf{x}=(x_1,x_2,\dots ,x_d)$

$P_X$ … the probability distribution “law” of a random variable $X$.
$P(y|x)=P(Y=y|X=x)P(x_i|y)=P(X=x_i,Y=y)$ … shorthand notation for probabilities of random variables taking on specific values.

$p(x)=p_X(x)$ … the probability of a random variable $X$ taking on value $x$, general notation; specific:
… $=P(X=x)$ PMF (discrete; is a probability)
… $=f_X(x)$ PDF (continuous; not a probability, probabilities come from integrals $P(a\le X\le b)={\int }_a^b{f}_X(x)dx$)
$P(A|B),p(x|\theta )$ … conditional probability, discrete and continuous, respectively.

$\{p_{\theta }:\theta \in \Theta \}$ … a family of distributions indexed by $\theta$.
Its use can depend on context:
$\text{PMF/PDF}:=p(x;\theta )=p_{\theta }(x)=p(x|\theta )$ … the probability of data $x$ given fixed parameters $\theta$. (PMF/PDF of $x$)
$L(\theta ;x):=p(x;\theta )=p_{\theta }(x)=p(x|\theta )$ … the likelihood of fixed data $x$ given parameters $\theta$. (not a density in $\theta$!)

$p(x)$ … a distribution without explicit parameters, e.g. the true data distribution, or a marginal distribution.
$p(\theta )$ … a distribution over parameters, e.g. a prior.
$p(\theta |x)$ or $p(\theta ;x)$ … the posterior distribution over parameters given data $x$.

Wth was the context of this | (Hypergeometric Distribution (sampling without replacement))

give the general form, make the connection to multinomial (sampling with replacement), move into separate form and maybe make a note for sampling with/without replacement

---

If we have $N=N_A+N_B$ items, of which $a$ are of type and $b$ are of another type.
Then the probability of sampling $n=a+b$ items, of which $a$ are of type $A$ and $b$ are of type $B$, without replacement is given by the bivariate hypergeometric distribution:

$$\begin{array}{r}P(X_A=a,X_B=b)=\frac{\left(\genfrac{}{}{0ex}{}{A}{a}\right)\left(\genfrac{}{}{0ex}{}{B}{b}\right)}{\left(\genfrac{}{}{0ex}{}{N}{n}\right)}\end{array}$$

Introduction (old and bad)

Let $\pi (n)$ denote the nth digit of $\pi$.
Consider the quanity (number) of $d=\pi (3)$.
Is it even or odd? $d=4$ → even.
What about $\pi (10^{1000})$?
There are two approaches to probability:

1. “d is even with probability $0.5$”
   1. Probability: Representing uncertainty about certain values.
   2. Baysian approach. Common in Machine Learning
2. “d is even with probablility 0 or 1 but I don’t know which”
   4. Probability: Mathematically defineable thing about frequencies
   5. More common, esp. in rigorous mathematical theory.

   ---

   Important points:
   The total probability is alwys 1 $${\int }_X^{}p(x)dx=1$$ We care about indepence: $$p(x,y)=p(x)\cdot p(y)$$ We care about expectation: The probabiliy times some other function (e.g. darts player points and dart positions) $${\mathbb{E}}_p[f]={\int }_X^{}f(x)p(x)dx$$ Wandb YT

References

mathematics

Footnotes

1. ↩