cross-entropy

Cross-entropy

The cross-entropy is the average surprise (entropy) you get observing a random variable $X$ governed by probability distribution $P$, while believing in a model $Q$.

$$\begin{array}{rl}H(P)+D_{\text{KL}}(P\Vert Q)& ={\mathbb{E}}_{x\sim P}[-\log p(x)]+{\mathbb{E}}_{x\sim P}[\log \frac{p(x)}{q(x)}]\\ & ={\mathbb{E}}_{x\sim P}[-\log p(x)]+{\mathbb{E}}_{x\sim P}[\log p(x)-\log q(x)]\\ & ={\mathbb{E}}_{x\sim P}[-\log q(x)]\\ & :=H(P,Q)\end{array}$$

Where $H(P)$ is the entropy of $X$, $D_{KL}(P\Vert Q)$ is the KL-divergence between $P$ and $Q$.
$P$ represents data/observations/a measured probability distribution,
$Q$ represents a theory/model/description/approximation of $P$.

CE can tell you how good your model is.
If you model is perfect, i.e. $P=Q⟹D_{\text{KL}}=0$, then the crossentropy is simply equal to the entropy (uncertainty) of the true distribution.

The cross-entropy can never be lower than the entropy of the generating distribution:

$$H(P,Q)\ge H(P)$$

$P$ and $Q$ are not interchangable

$$H(P,Q)\ne H(Q,P)$$

E.g.: If you believe a coin is fair (0.5/0.5), but it is rigged (0.99/0.01), then the CE is: $0.99\log \left(\frac{1}{0.5}\right)+0.01\log (\frac{1}{0.5})\approx 0.7$
If you believe it is rigged when it is actually fair, then the CE is: $0.5\log \left(\frac{1}{0.99}\right)+0.5\log (\frac{1}{0.01})\approx 2.4$
In the second case, the entropy is much larger, as half of the time you are extremely surprsied to see tails, this extreme surprise dominates the average surprise.