likelihood

Likelihood function

A likelihood scores how plausible different $\theta$ make the observed data $x$.
It has the same formula as the probability distribution in $x$, but viewed as a function of $\theta$, data $x$ fixed.

$$\begin{array}{r}L(\theta ;x):=p(x\mid \theta )\end{array}$$

We vary $\theta$ to see how likely they make the observed data.

Note: While $\int p(x|\theta )dx=1$ for each $\theta$, the likelihood need not integrate to 1 over $\theta$: $\int p(x|\theta )d\theta \ne 1$, i.e. it is not a probability distribution.
Note the equivalent and equally context dependent notation $p_{\theta }(x)=p(x|\theta )$. Writing $L(\theta ;x)$ avoids this ambiguity by making the variable explicit.

Careful: $p(\theta |x)$ (the posterior) is not a mirror image of the probability distribution $p(x|\theta )$ regardless of whether it is viewed as a function of $\theta$ or $x$. ¹ They are related through Bayes Theorem:

Bayes Theorem: Normalized likelihood = posterior

The likelihood is not normalized over $\theta$. That’s just another way of saying that it’s not a probability distribution in $\theta$, i.e. $\int L(\theta ;x)d\theta \ne 1$.
Baye’s theorem normalizes the likelihood to get a proper probability distribution over $\theta$ (the posterior):

$$\begin{array}{rl}p(\theta |x)& =\frac{L(\theta ;x)p(\theta )}{p(x)}\end{array}$$

Since $p(x)$ does not depend on $\theta$, we can write this as a proportionality:

$$\begin{array}{rl}p(\theta |x)& \propto L(\theta ;x)p(\theta )\end{array}$$

→ The posterior is the prior weighted by the likelihood, up to multiplication by a constant $1/p(x)$.,
→ Dropping this constant changes nothing for inference.

Properties of the likelihood

Some but not all properties of the probability carry over.
Nonnegativity: $L(\theta |x)\ge 0$
chain rule of probability still works
…
Some properties don’t carry over:
Not a distribution over $\theta$.
No countable additivity over disjoint parameter sets: Likelihoods don’t add like probabilities, they multiply (for independent data). The log-likelihood adds.
Marginalization over $\theta$ is not defined without a prior /measure.

Footnotes

1. ^cc8b69. I certainly would never confuse these… ↩