likelihood

Likelihood function

A likelihood scores how plausible different $\theta$ make the observed data $x$.
It has the same formula as the probability distribution in $x$, but viewed as a function of $\theta$, data $x$ fixed.

$$\begin{array}{r}L(\theta ;x):=p(x\mid \theta )\end{array}$$

We vary $\theta$ to see how likely they make the observed data.

Note: While $\int p(x|\theta )dx=1$ for each $\theta$, the likelihood need not integrate to 1 over $\theta$: $\int p(x|\theta )d\theta \ne 1$, i.e. it is not a probability distribution (over $\theta$).
Note the equivalent and equally context dependent notation $p_{\theta }(x)=p(x|\theta )$. Writing $L(\theta ;x)$ avoids this ambiguity by making the variable explicit.

Careful: $p(\theta |x)$ (the posterior) is not a mirror image of the probability distribution $p(x|\theta )$ regardless of whether it is viewed as a function of $\theta$ or $x$. ¹ They are related through Bayes Theorem:

Bayes Theorem: Normalized likelihood = posterior

The likelihood is not normalized over $\theta$. That’s just another way of saying that it’s not a probability distribution in $\theta$, i.e. $\int L(\theta ;x)d\theta \ne 1$.
Baye’s theorem normalizes the likelihood to get a proper probability distribution over $\theta$ (the posterior):

$$\begin{array}{rl}p(\theta |x)& =\frac{L(\theta ;x)p(\theta )}{p(x)}\end{array}$$

Since $p(x)$ does not depend on $\theta$, we can write this as a proportionality:

$$\begin{array}{rl}p(\theta |x)& \propto L(\theta ;x)p(\theta )\end{array}$$

→ The posterior is the prior weighted by the likelihood, up to multiplication by a constant $1/p(x)$.,
→ Dropping this constant changes nothing for inference.

iid factorization of the likelihood

$$p(x\mid \theta )\stackrel{\text{iid}}{=}\prod _{i=1}^{n}p(x_i|\theta ),x=(x_1,\dots ,x_n)$$

Independence turns the joint probability of the data into a product via the chain rule of probability, without it we’d have: $p(x\mid \theta )={\prod }_{i}p(x_i|x_1,\dots x_{i-1},\theta )$
Being identically distributed makes it the same $p$ in every factor (otherwise each $x_i$ would have its own marginal $p_i$).

What does the likelihood function $L(\theta ;x)$ represent?

It scores how plausible different $\theta$ make the observed data $x=(x_1,\dots ,x_n)$.
It’s the (joint) probability/density that drawing $n=|x|$ iid samples from $p_{\theta }(X)$ — $X$ where $X$ is the random variable of a single observation — produces exactly the observed data $x$.

Properties of the likelihood

Some but not all properties of the probability carry over.
Nonnegativity: $L(\theta |x)\ge 0$
chain rule of probability still works
…
Some properties don’t carry over:
Not a distribution over $\theta$.
No countable additivity over disjoint parameter sets: Likelihoods don’t add like probabilities, they multiply (for independent data). The log-likelihood adds.
Marginalization over $\theta$ is not defined without a prior /measure.

Footnotes

1. ^cc8b69. I certainly would never confuse these… ↩