cross-entropy loss

Cross-Entropy Loss

The cross-entropy loss is often used for classification tasks, in conjunction with softmax:

$${\ell }_{\text{CE}}^{(i)}(x_i,q_i,\theta )=-\sum _k{q}_i(k)\log (p_{\theta }(k|x_i))=H(q_i,p_{\theta }(\cdot |x_i))$$

$i$ … $i^{th}$ sample
$k$ … class index
$q$ … true distribution (soft labels, one-hot, …)
$p$ … predicted probability distribution (from softmax, …)
$\theta$ … model parameters
$x$ … input features
→ ${\hat{p}}_i=\text{softmax}(f_{\theta }(x_i)),{\ell }_{\text{CE}}^{(i)}=-y_i^T\log {\hat{p}}_i$

Categorical Cross-Entropy Loss

For one-hot encodings, the cross-entropy loss simplifies to Categorical Cross-Entropy, aka negative log-likelihood loss:

$${\ell }_{\text{CCE}}^{(i)}=-\sum _k{y}_{i,k}\log (p_{\theta }(k|x_i))=-\log (p_{\theta }(y_{i,k}|x_i))$$

$i$ … $i^{th}$ sample
$k$ … index of the correct class
$y$ … true label (one-hot, all probability mass is on the true class)
$p$ … predicted probability distribution (from softmax, …)
$\theta$ … model parameters
$x$ … input features
→ ${\hat{p}}_i=\text{softmax}(f_{\theta }(x_i)),{\ell }_{\text{CCE}}^{(i)}=-\log {\hat{p}}_{i,y_i}$

Link to original

nn.CrossentropyLoss <=> nn.LogSoftmax & nn.NLLoss
nn.BCEWithLogitsLoss <=> nn.Sigmoid & nn.BCELoss

When does it not perform well?

• strong class imbalence (just becomes more confident in predicting the majority class and neglects the minority class)
• fails to differentiate between easy and hard samples. Hard example = model makes significant errors; easy example = straightforward to classify. CE doesn’t allocate more attention to hard samples

Mitigations:

Addressing class imbalance: balanced CE loss

Adding a weighting factor for each class resolves the issue with class imbalances:

$$CE(p_t)=-{\alpha }_t\log (p_t)$$

$p_i\dots$ predicted probability of class (from softmax, …)
${\alpha }_t\dots$ weight for class $t$
$\alpha$ is usually calculated as the inverse of the class distribution like so:

$${\alpha }_t=\frac{\text{number of samples}}{\text{number of classes}\times {\text{class count}}_t}$$

sample_labels = torch.cat([dataset[i][1] for i in len(dataset)])  
unique, counts = torch.unique(sample_labels, return_counts=True)  
class_weights = counts.sum() / (len(unique) * counts)

(if the dataset is very large, take a random sample that fits into memory)

Addressing hard-negatives: focal loss

https://towardsdatascience.com/cross-entropy-negative-log-likelihood-and-all-that-jazz-47a95bd2e81
classification