inference

Statistical inference is the process of learning about unknown quantities from observed data.

The Inference Problem

We observe data $X$ generated by some process with unknown parameters $\theta$. What can we say about $\theta$ given $X$?
This involves two directions of reasoning:

• Forward: $P(X\mid \theta )$ - If we knew the parameters, what data would we expect?
• Inverse: $P(\theta \mid X)$ - Given the data we observed, what parameters are plausible?

We typically know how to compute $P(X\mid \theta )$ (the likelihood) from our model, but we want $P(\theta \mid X)$ (the posterior).

Approaches to Inference

Baysian Inference - Uses Bayes theorem to update beliefs.
Frequentist Inference - Treats parameters as fixed but unknown. Uses procedures that have good properties when repeated.

The key philosophical difference: Bayesians quantify uncertainty about parameters directly via $P(\theta \mid X)$, while frequentists quantify uncertainty about their procedures.