marginal likelihood

Marginal LIkelihood / Evidence

The marginal likelihood is the probability of the observed data $\mathbf{x}$ under a model after marginalizing (integrating out) all unobserved variables $U$ (e.g. latent variables; if bayesian, the model parameters too), then using the chain rule of probability to decompose the joint distribution:

$$\begin{array}{rlr}p(\mathbf{x})& ={\int }_{U}p(\mathbf{x},U)dU& ={\int }_{U}p(\mathbf{x}|U)p(U)dU\end{array}$$

→ We get the total probability of the data under the model:

Below are some common formulations of the marginal likelihood in different contexts:

Marginal likelihood over latents / "data likelihood"

$$\begin{array}{rl}{p}_{\theta }(\mathbf{x})& ={\int }_{\mathbf{z}}{p}_{\theta }(\mathbf{x},\mathbf{z})d\mathbf{z}\\ & ={\int }_{\mathbf{z}}{p}_{\theta }(\mathbf{x}|\mathbf{z})p_{\theta }(\mathbf{z})d\mathbf{z}\end{array}$$

Here we integrate over the latent variables $\mathbf{z}$, given fixed model parameters $\theta$.
This is often called the data likelihood in the context of latent variable models, as it gives the likelihood of the observed data $\mathbf{x}$ under the model, accounting for all possible configurations of the latent variables $\mathbf{z}$.

Marginal likelihood over parameters / "model evidence"

$$\begin{array}{rl}p(\mathbf{x})& ={\int }_{\theta }p(\mathbf{x}|\theta )p(\theta )d\theta \\ & ={\int }_{\theta }\mathcal{L}(\theta )p(\theta )d\theta \end{array}$$

Todo

https://chatgpt.com/g/g-p-68bcc91a795c8191b02e92abaf3daa0c-tutorials/c/68bef229-82cc-8326-aea0-f5021deae1ff
https://chatgpt.com/g/g-p-68bcc91a795c8191b02e92abaf3daa0c-tutorials/c/68bf196e-bcc4-8328-ae79-ebb7e8857ce6
https://chatgpt.com/c/68be0327-367c-8323-b066-3efddd854212

It’s also called evidence, because it quantifies how much the data “evidences” or supports the model.

# Fix mixing conccepts of model evidence and latent variable posterior evidence…

The marginal likelihood normalizes the posterior

The marginal likelihood represents the total probability of observing our data, accounting for all possible parameter settings according to our prior beliefs. It serves as the normalizing constant in Bayes Theorem:

$$p_{\theta }(\mathbf{z}|\mathbf{x})=\frac{p_{\theta }(\mathbf{x}|\mathbf{z})p_{\theta }(\mathbf{z})}{p_{\theta }(\mathbf{x})}$$

Unlike the likelihood, which evaluates parameter settings individually, the marginal likelihood evaluates the model as a whole, integrating over the entire parameter space.

The marginal likelihood naturally implements occam’s razor for model comparison.

Complex models with greater flexibility can fit the observed data better, but they spread their prior probability over a larger parameter space. A model with the highest marginal likelihood balances goodness-of-fit against model complexity, as it must assign high probability to the observed data across its entire parameter space, not just at the best-fitting parameters (i.e. it can’t “cheat” by having a few extreme parameter settings that fit well but most others that don’t).