perplexity

W.r.t. cross-entropy loss $\mathcal{L}=-\log p$, the perplexity is defined as:

$$\begin{array}{rl}\text{PPL}& =\exp (\mathcal{L})\\ & =\exp \left(-\frac{1}{N}\sum _{i=1}^N\log p_i\right)\\ & =\exp \left(\sum _{i=1}^N\log p_i^{-\frac{1}{N}}\right)\\ & ={\left(\prod _{i=1}^N{p}_i\right)}^{-\frac{1}{N}}\\ & ={\left(\prod _{i=1}^N\frac{1}{p_i}\right)}^{\frac{1}{N}}\\ & =\sqrt[N]{\prod _{i=1}^N\frac{1}{p_i}}\end{array}$$

It’s the geometric mean of the inverse predicted probabilities.
→ Perplexity is dimensionless and multiplicative: halving perplexity doubles the geometric mean of the probabilities the model assigned to the true labels.

Effective branching factor

Perplexity: The model is, on average (geometric mean), as confused as if it had to pick uniformly from $k$ candidates.

EXAMPLE

For a uniform distribution over $k$ candidates (e.g. vocabulary size), we have $\mathcal{L}=-\log \frac{1}{k}=\log k$, so $\text{PPL}=\exp (\log k)=k$.

The lowest possible perplexity is at $\mathcal{L}=0$, i.e. $\text{PPL}=1$.