fisher information

Fisher Information

The Fisher information $I(\theta )$ quantifies how much information a random variable $X$ carries about an unknown parameter $\theta$ that describes its probability distribution. For a PDF $f(x;\theta )$, it is defined as:

$$I(\theta )=\mathbb{E}\left[{\left(\frac{\partial }{\partial \theta }\log f(X;\theta )\right)}^2\right]=-\mathbb{E}\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(X;\theta )\right]$$

where the expectation is taken with respect to $f(x;\theta )$.

Fisher information measures the curvature of the log-likelihood function around the true parameter value.
A higher Fisher information indicates:

• The distribution is more sensitive to changes in the parameter $\theta$
• We can estimate $\theta$ more precisely from observations
• The likelihood function has a sharper peak around the true value

The first form of the definition shows that fisher information is the variance of the score function.

The second form of the definition (using second derivative) directly shows this connection to curvature, since the second derivative measures how quickly the slope changes – a more negative second derivative indicates a sharper peak in the log-likelihood.

Cramér-Rao bound: The variance of any unbiased estimator $\hat{\theta }$ is bounded below by the inverse of the Fisher information:

$$\text{Var}(\hat{\theta })\ge \frac{1}{I(\theta )}$$

→ Fisher information directly determines the best possible precision we can achieve when estimating parameters from data.

For a normal distribution $\mathcal{N}(\mu ,{\sigma }^2)$, the Fisher information with respect to the mean $\mu$ is:

$$I(\mu )=\frac{1}{{\sigma }^2}$$

→ We can estimate the mean more precisely (higher Fisher information) when the variance is smaller.