surprise

Surprise

$$\text{displaylines}s(x)=log(\frac{1}{p(x)})=-\log p(x)$$

Surprise is the log of the inverse probability (negative log likelihood).
It is also called “shannon information”.

(s is the state in the above image, h is the surprise function)
Extreme cases:
p(x) = 1 → 0 surprise
p(x) = 0 → $\infty$ surprise

Log-properties

Because we take the log of the probability, if two unrelated suprising things happen, we add suprises instead of multiplying like with probabilities:

$$\begin{gathered} \text{displaylines}P(A\wedge B)=P(A)\cdot P(B) \\ S(A\wedge B)=S(A)+S(B) \end{gathered}$$

entropy is the expected value of surprise

$$H(x)=\mathbb{E}[p_X(x)s_X(x)]={\int }_X^{}{s}_X(x)p_X(x)dx={\int }_X{p}_X(x)\log (\frac{1}{p_X(x)})dx=-{\int }_X{p}_X(x)\log (p_X(x))dx$$

→ Average surprise of a random variable weighted by its probability.
High entropy = high uncertainty → higher avg. surprise. E.g.: uniform distribution = highest surprise (unpredictable - every outcome is equally surprising).
Low entropy = low uncertainty → less avg. surprise. Certain outcome (1 possibility) → 0 entropy, no surprise, predictable.
→ entropy = uncertainty/unpredictabiltiy

We can also write it like this in the discrete case:

$$H(x)=\sum _x{p}_x\cdot s(x)=-\sum _x{p}_x\cdot \log \left(\frac{1}{p_x}\right)=-\sum _x{p}_x\cdot \log (p_x)$$

More surprising things are harder to compress (in information theory).
In machine learning, for example if we predict how likely a football team is to win, we want to achieve a very low suprise (loss), so we can estimate confidently.

In machine learning, we want the next target to have a very high probability, i.e. low surprise → negative log-likelihood loss

References

The Key Equation Behind Probability