KL-divergence

The KL divergence measures the difference between the probability distribution of a model and the true distribution, disregarding the uncertainty of the true distribution itself.

The KL-divergence subtracts the uncertainty (entropy) about the true distribution from cross-entropy, leaving us with a measure for the similarity of these distributions.

$$\begin{array}{rl}H(P,Q)-H(P)& =\\ \sum _s{p}_s\log \left(\frac{1}{q_s}\right)-\sum _s{p}_s\log \left(\frac{1}{p_s}\right)& =\\ \sum _s{p}_s\left(\log \left(\frac{1}{q_s}\right)-\log \left(\frac{1}{p_s}\right)\right)& =\\ \sum _s{p}_s\log \left(\frac{p_s}{q_s}\right)& =D_{KL}(P,Q)\end{array}$$

Link to original

KL divergence is not a metric (distance) because it is not symmetric! $D_{KL}(P||Q)\ne D_{KL}(Q||P)$

→ When using KL divergence as a loss function, the choice of which distribution is the “reference” (P) versus the “approximation” (Q) matters significantly. For example, minimizing $D_{KL}(P||Q)$ tends to produce Q that covers all modes of P (zero-avoiding), while minimizing $D_{KL}(Q||P)$ produces Q that concentrates on high-probability regions of P (zero-forcing).

Why aren’t we using KL-divergence as a loss function, if it seems to have better properties?

Since we cannot change $P$, we can only minimize w.r.t. $Q$, and hence the KL divergent is equivalent to cross-entropy loss:

We don’t care about the exact value, so we don’t bother to estimate the entropy of the training data:

References

The Key Equation Behind Probability