score function

The term “score function” is a leftover from the context in which it was first used in (genetic analysis).

Score function

The score function (or score) is the gradient of the log-likelihood function with given a parameter vector $\theta$ and observed data $x$:

$$\begin{array}{r}{s}_{\theta }(x):={\nabla }_{\theta }\log p_{\theta }(x)\end{array}$$

It tells us which direction in parameter space we should adjust $\theta$ to increase the likelihood of observing our data $x$.
A positive score suggests increasing $\theta$ would improve the fit, while a negative score suggests decreasing it.

The score function has mean zero under the true parameter value

If ${\theta }_{\text{true}}$ is the parameter that generated our observed data, not an estimate:

$${\mathbb{E}}_{{\theta }_{\text{true}}}[s({\theta }_{\text{true}};X)]=0$$

… we are at a stationary point of the log-likelihood function.
Note: A zero gradient is not sufficient for global optimality, but could also be satisfied by local maxima, local minima, saddle points, … unless the parameter space is convex and the likelihood function is concave.

The expected value of the score function w.r.t the model is zero.

$$\begin{array}{rl}{\mathbb{E}}_{X\sim p_{\theta }}[s_{\theta }(X)]& ={\mathbb{E}}_{X\sim p_{\theta }}[{\nabla }_{\theta }\log p_{\theta }(X)]\\ & =\int {\nabla }_{\theta }\log p_{\theta }(x)p_{\theta }(x)dx\\ & =\int \frac{{\nabla }_{\theta }{p}_{\theta }(x)}{p_{\theta }(x)}{p}_{\theta }(x)dx\\ & =\int {\nabla }_{\theta }{p}_{\theta }(x)dx\\ & ={\nabla }_{\theta }\int p_{\theta }(x)dx\\ & ={\nabla }_{\theta }1\\ & =0\end{array}$$

Line 3: Take the derivative of the log
Line 5: The $p_{\theta }$ is a PDF

The covariance of the score function is called the fisher information

It gives us a sense of the uncertainty of our estimate, i.e. measuring how much information our data provides about the parameter $\theta$:

$$\begin{array}{rl}{\mathbb{E}}_{X\sim p_{\theta }}[(s_{\theta }(X)-0)(s_{\theta }(X)-0{)}^T]& ={\mathbb{E}}_{X\sim p_{\theta }}[s_{\theta }(X)s_{\theta }(X{)}^T]\\ & ={\mathbb{E}}_{X\sim p_{\theta }}[({\nabla }_{\theta }\log p_{\theta }(X))({\nabla }_{\theta }\log p_{\theta }(X){)}^T]\\ & :=I_F(\theta )\end{array}$$

---

https://agustinus.kristia.de/blog/fisher-information/
https://andrewcharlesjones.github.io/journal/log-derivative.html