information

self-information (surprise) … information content of a single event (code length of a single event)
$I(x)=-\log p(x)$

entropy … average surprise of a random variable (expected code length)
$H(X)={\mathbb{E}}_{x\sim P}[-\log p(x)]$

cross-entropy … average surprise of the true data $P$ when assuming a model $Q$ (expected code length using $Q$ to encode $P$)
$H(P,Q)={\mathbb{E}}_{x\sim P}[-\log q(x)]$

KL-divergence … extra average surprise regret when assuming model $Q$ instead of the true distribution $P$ (inefficiency of using $Q$ to encode $P$)
$D_{KL}(P||Q)=H(P,Q)-H(P)={\mathbb{E}}_{x\sim P}[\log p(x)-\log q(x)]$

mutual information … average reduction in uncertainty about one random variable given knowledge of another (shared information between them)
$I(X;Y)=H(X)-H(X|Y)$

fisher information … average local curvature of log-likelihood, per observation (sqaured sensitivity of the model to parameter changes)
$I(\theta )=\mathbb{E}\left[{\left(\frac{\partial }{\partial \theta }\log p(X;\theta )\right)}^2\right]=-\mathbb{E}\left[\frac{{\partial }^2}{\partial {\theta }^2}\log f(X;\theta )\right]$

information gain …