inference

Statistical inference is the process of learning about unknown quantities from observed data.

This has nothing to do with inference (ML) ("inference-time"/"test-time")

The only link is the notion of “inferring something”.
Statistical inference is about learning/uncertainty, ML inference is about making predictions with a trained model.

The Inference Problem

We observe data $X$ generated by some process with unknown parameters $\theta$. What can we say about $\theta$ given $X$?
This involves two directions of reasoning:

• Forward: $P(X\mid \theta )$ - If we knew the parameters, what data would we expect?
• Inverse: $P(\theta \mid X)$ - Given the data we observed, what parameters are plausible?

We typically know how to compute $P(X\mid \theta )$ (the likelihood) from our model, but we want $P(\theta \mid X)$ (the posterior).

Approaches to Inference

bayesian inference - Uses Bayes theorem to update beliefs.
Frequentist Inference - Treats parameters as fixed but unknown. Uses procedures that have good properties when repeated.
Bayesians quantify uncertainty about parameters directly via $P(\theta \mid X)$, while frequentists quantify uncertainty about their procedures.

variational inference - Approximates complex posteriors with simpler distributions.