probability

Steve for a 🐐ed intro.

---

Probability

“Laplace’s Rule”:

$$P(A)=\frac{|A|}{|\Omega |}$$

$|A|$ … number of events where $A$ occurs / number of ways $A$ can happen.
$|\Omega |$ … number of possible events.
→ $0\le P(A)\le 1$

The probability $P$ is a map from subsets of the sample space $\Omega$ (the set of all possible outcomes of a random experiment) to the real numbers:

$$\begin{array}{rl}P:2^{\Omega }& \to \mathbb{R}[0,1]\in \mathbb{R}\end{array}$$

$P(\Omega )=1$ … the probability of something happening is 1.
$P(\varnothing )=0$ … the probability of nothing happening is 0.
$A\subseteq \Omega ⟹P(A)\ge 0$ … probability of an event is always non-negative.
$A\subseteq B⟹P(A)\le P(B)$ … a subset of events has a smaller probability than the set it’s a subset of.
$P(A^c)=1-P(A)$ … the probability of $A$ not happening. Also sometimes denoted as $P(\neg A)$ (surprise, but without the log).

Flip a fair coin twice. What’s the probability of at least one heads occuring?

$\Omega =\{hh,ht,th,tt\}$
$A=\{hh,ht,th\}$
$P(A)=\frac{3}{4}$

Probability is just combinatorics, i.e. the math of combinations and proportions

Thinking of probability geometrically, as proportions (relative sizes of sets) is incredibly helpful:

Pictured: The $6\times 6$ sample space formed by all possible outcomes when rolling two dice.
The event $X=5$, where $X$ is a random variable representing the dice sum can be represented as an area (yellow) in this space, and its probability $P(X=5)$ is the proportion of the area of this event to the total area of the sample space → $P(X=5)=\frac{4}{36}=\frac{1}{9}$
Generally, you can think of the sample space as a unit square (area = 1).

Roll two dice. What is the probability that at least one die is a 5?

By tedious counting:
$\Omega =\{(1,1),(1,2),\dots ,(6,6)\}$
$A=\{(5,1),(5,2),\dots ,(6,5),(6,6)\}$
$P(A)=\frac{11}{36}$
… or we take the problem apart by looking at the complement of $A$: Either the first die is a 5 or it’s not and the second die is a 5.
$P(A)=P(X_1=5)+P(X_1\ne 5)\cdot P(X_2=5)=\frac{1}{6}+\frac{5}{6}\cdot \frac{1}{6}$

Visually, it’s easy to see the relation to binomial coefficients (→ probability of getting a 5 on at least one die is the same as the probability of getting a 5 on exactly one die) and how probabilities stack up.

Probabilistic models are not just useful for truly random things, but also for things that are too complex to model exactly.

If there are $n$ people in a room, how large does $n$ need to be for at least $50\%$ chance of at least $2$ people sharing a birthday?

Each person has to compare with each other person $(n-1)+(n-2)+\cdots +2+1=\frac{n(n-1)}{2}$, but this won’t help us get to an exact probability, as there’s over-counting, etc.
$P(x\ge 2)=1-P(x=0)$, where $x$ is the number of people with the same birthday.
The expression on the left is very hard to compute as again, you need to keep track of all the combinations, overlaps, …
For the expression on the right (the complement), we just need to divide the number of unique birthdays (sampling without replacement) by the possible number of birthdays (sampling with replacement):

$$\begin{array}{r}\frac{365!}{(365-n)!}\cdot \frac{1}{365^n}=(365\cdot 364\cdots (365-(n-1)))\cdot \frac{1}{365^n}==\prod _{i=1}^n\frac{365-i}{365}\end{array}$$

>>> P_different_bday = lambda n: math.prod([(365 - i) / 365 for i in range(1, n)])
>>> [round(1 - P_different_bday(i), 3) for i in range(0,10)]
[0, 0.0, 0.003, 0.008, 0.016, 0.027, 0.04, 0.056, 0.074, 0.095]
>>> 1 - P_different_bday(360)
1.0

Probability of $A$ or $B$ (at least one) occuring

$$P(A\cup B)=P(A)+P(B)-P(A\cap B)$$

For non-disjoint events $A,B\in \Omega$ (i.e. they can co-occur), we need to subtract the intersection, so we don’t count it twice → $\cup \leftrightarrow +$ (with a correction).
For disjoint events, it simplifies:

$$(A\cap B)=\varnothing ⟹P(A\cup B)=P(A)+P(B)$$

”$A$ and $B$ happened”: “$A$ or $B$ happened”:

Further reading (roughly in order):
conditional probability
law of total probability
bayes theorem
probability distribution
joint probability distribution
…

Slop below

The general view / measure-theoretic scaffold:

Probability as a measure

The probability is just a measure with the additional normalization constraint that

$$P(\Omega )=1$$

Probability space: $(\Omega ,\mathcal{F},\mathbb{P})$
Random variable: $X:\Omega \to \mathbb{R}$ … a measurable function

Continue with this

https://chatgpt.com/g/g-p-68bcc91a795c8191b02e92abaf3daa0c-tutorials/c/68c084fa-88f0-8321-976e-8c44dcfad57e
https://chatgpt.com/c/68c070c4-87fc-8329-b5e4-38b7423e9d36

Continue cleaning up the below

https://chatgpt.com/c/68bf85f5-1860-8320-9794-f5b3341b5468
https://chatgpt.com/c/68c322e2-c778-832a-865b-35fed8b3d912 (last one)

A note on notation (needs refactor!)

$\Omega$ … the sample space, the set of all possible outcomes of a random experiment.
$A,B,C\subseteq \Omega$ … events.
$\mathbb{P}:\mathcal{F}\to [0,1]$ … probability measure $\mathbb{P}(A)$ aka $\mathrm{Pr}(A)$, aka $P(A)$.

$X,Y,X_i$ … random variables; $\mathbf{X}=(X_1,X_2,\dots ,X_d)$ … a vector of random variables.
$x,y,z$ … realized values; $\mathbf{x}=(x_1,x_2,\dots ,x_d)$

$P_X$ … the probability distribution “law” of a random variable $X$.
$P(y|x)=P(Y=y|X=x)P(x_i|y)=P(X=x_i,Y=y)$ … shorthand notation for probabilities of random variables taking on specific values.

$p(x)=p_X(x)$ … the probability of a random variable $X$ taking on value $x$, general notation; specific:
… $=P(X=x)$ PMF (discrete; is a probability)
… $=f_X(x)$ PDF (continuous; not a probability, probabilities come from integrals $P(a\le X\le b)={\int }_a^b{f}_X(x)dx$)
$P(A|B),p(x|\theta )$ … conditional probability, discrete and continuous, respectively.

$\{p_{\theta }:\theta \in \Theta \}$ … a family of distributions indexed by $\theta$.
Its use can depend on context:
$\text{PMF/PDF}:=p(x;\theta )=p_{\theta }(x)=p(x|\theta )$ … the probability of data $x$ given fixed parameters $\theta$. (PMF/PDF of $x$)
$L(\theta ;x):=p(x;\theta )=p_{\theta }(x)=p(x|\theta )$ … the likelihood of fixed data $x$ given parameters $\theta$. (not a density in $\theta$!)

$p(x)$ … a distribution without explicit parameters, e.g. the true data distribution, or a marginal distribution.
$p(\theta )$ … a distribution over parameters, e.g. a prior.
$p(\theta |x)$ or $p(\theta ;x)$ … the posterior distribution over parameters given data $x$.

Wth was the context of this | (Hypergeometric Distribution (sampling without replacement))

give the general form, make the connection to multinomial (sampling with replacement), move into separate form and maybe make a note for sampling with/without replacement

---

If we have $N=N_A+N_B$ items, of which $a$ are of type and $b$ are of another type.
Then the probability of sampling $n=a+b$ items, of which $a$ are of type $A$ and $b$ are of type $B$, without replacement is given by the bivariate hypergeometric distribution:

$$\begin{array}{r}P(X_A=a,X_B=b)=\frac{\left(\genfrac{}{}{0ex}{}{A}{a}\right)\left(\genfrac{}{}{0ex}{}{B}{b}\right)}{\left(\genfrac{}{}{0ex}{}{N}{n}\right)}\end{array}$$

Introduction (old and bad)

Let $\pi (n)$ denote the nth digit of $\pi$.
Consider the quanity (number) of $d=\pi (3)$.
Is it even or odd? $d=4$ → even.
What about $\pi (10^{1000})$?
There are two approaches to probability:

1. “d is even with probability $0.5$”
   1. Probability: Representing uncertainty about certain values.
   2. Baysian approach. Common in Machine Learning
2. “d is even with probablility 0 or 1 but I don’t know which”
   4. Probability: Mathematically defineable thing about frequencies
   5. More common, esp. in rigorous mathematical theory.

   ---

   Important points:
   The total probability is alwys 1 $${\int }_X^{}p(x)dx=1$$ We care about indepence: $$p(x,y)=p(x)\cdot p(y)$$ We care about expectation: The probabiliy times some other function (e.g. darts player points and dart positions) $${\mathbb{E}}_p[f]={\int }_X^{}f(x)p(x)dx$$ Wandb YT

References

mathematics