fisher information

“average local curvature of log-likelihood per observation” (i.e., expected squared score);
or, informally: “how quickly the data punish small parameter mistakes, per sample.”
“FIM tells you how sharply the model reacts to parameter nudges (geometry/curvature).”

Motivation

Proof + relation to hessian, read after done with Monte Carlo Gradient Estimation in Machine Learning

→ https://agustinus.kristia.de/blog/fisher-information/
https://claude.ai/chat/cdfedbde-bb26-47cd-8db0-c4df38a4a49d

The covariance of the score function is called the fisher information

It gives us a sense of the uncertainty of our estimate, i.e. measuring how much information our data provides about the parameter $\theta$:

$$\begin{array}{rl}{\mathbb{E}}_{X\sim p_{\theta }}[(s_{\theta }(X)-0)(s_{\theta }(X)-0{)}^T]& ={\mathbb{E}}_{X\sim p_{\theta }}[s_{\theta }(X)s_{\theta }(X{)}^T]\\ & ={\mathbb{E}}_{X\sim p_{\theta }}[({\nabla }_{\theta }\log p_{\theta }(X))({\nabla }_{\theta }\log p_{\theta }(X){)}^T]\\ & :=I_F(\theta )\end{array}$$

Link to original

Fisher Information

The Fisher information $I(\theta )$ quantifies how much information a random variable $X$ carries about an unknown parameter $\theta$ that describes its probability distribution. For a PDF $f(x;\theta )$, it is defined as:

$$I(\theta )=\mathbb{E}\left[{\left(\frac{\partial }{\partial \theta }\log f(X;\theta )\right)}^2\right]=-\mathbb{E}\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(X;\theta )\right]$$

where the expectation is taken with respect to $f(x;\theta )$.

Fisher information measures the curvature of the log-likelihood function around the true parameter value.
A higher Fisher information indicates:

• The distribution is more sensitive to changes in the parameter $\theta$
• We can estimate $\theta$ more precisely from observations
• The likelihood function has a sharper peak around the true value

The first form of the definition shows that fisher information is the variance of the score function.

The second form of the definition (using second derivative) directly shows this connection to curvature, since the second derivative measures how quickly the slope changes – a more negative second derivative indicates a sharper peak in the log-likelihood.

Cramér-Rao bound: The variance of any unbiased estimator $\hat{\theta }$ is bounded below by the inverse of the Fisher information:

$$\text{Var}(\hat{\theta })\ge \frac{1}{I(\theta )}$$

→ Fisher information directly determines the best possible precision we can achieve when estimating parameters from data.

For a normal distribution $\mathcal{N}(\mu ,{\sigma }^2)$, the Fisher information with respect to the mean $\mu$ is:

$$I(\mu )=\frac{1}{{\sigma }^2}$$

→ We can estimate the mean more precisely (higher Fisher information) when the variance is smaller.

FIM … Fisher information Matrix

Fisher information is the covariance of the score function