How is cross-entropy loss calculated?
For one example, the loss is -log p, where p is the probability the model gave the true class. The general form, -Σ y·log(p) summed over classes, reduces to that because the one-hot label y is 1 for the true class and 0 elsewhere. Training averages it over a batch and gradient descent pushes it down.
A worked example with three classes (cat, dog, bird), natural log throughout:
| Step | Cat | Dog | Bird |
|---|---|---|---|
| Logits from the network | 2.0 | 1.0 | 0.1 |
| exp(logit) | 7.389 | 2.718 | 1.105 |
| Softmax probability (divide by 11.212) | 0.659 | 0.242 | 0.099 |
| Loss if this is the true class | 0.417 | 1.417 | 2.317 |
If the image is a cat, the loss is -ln(0.659) = 0.417. If it is a bird, the same prediction costs 2.317. A model that guesses uniformly over three classes scores ln(3) = 1.099, a useful baseline: a validation loss at or above it means the model does no better than guessing. The penalty is logarithmic, so the last bit of confidence in a wrong answer costs the most: p = 0.5 costs 0.693, p = 0.01 costs 4.605. That is why cross-entropy rewards honest probabilities, the property model calibration measures.
What is the difference between binary and categorical cross-entropy?
They are the same idea applied to different label shapes.
| Variant | Use it for | Output layer | PyTorch | Keras |
|---|---|---|---|---|
| Binary cross-entropy | Two classes, or several independent yes/no labels | One sigmoid per label | BCEWithLogitsLoss |
BinaryCrossentropy |
| Categorical cross-entropy | One label out of many, one-hot targets | Softmax over classes | CrossEntropyLoss |
CategoricalCrossentropy |
| Sparse categorical cross-entropy | One label out of many, integer targets | Softmax over classes | CrossEntropyLoss (class indices) |
SparseCategoricalCrossentropy |
Binary cross-entropy is -[y·log p + (1 - y)·log(1 - p)], the two-class case of the same formula listed with other losses under loss function. Pick by label structure, not by class count: a multi-label classification problem with ten tags still uses binary cross-entropy, ten times.
What is the loss function for softmax, and why do frameworks want logits?
Softmax is almost always paired with cross-entropy, and frameworks fuse the two. PyTorch’s CrossEntropyLoss expects “the unnormalized logits for each class” and, for class-index targets, is “equivalent to applying LogSoftmax on an input, followed by NLLLoss”; its defaults are reduction='mean' and label_smoothing=0.0 (PyTorch 2.15 CrossEntropyLoss, read 16 November 2026). Keras goes the other way: its cross-entropy losses default to from_logits=False and assume y_pred already holds probabilities (Keras probabilistic losses, read 16 November 2026). Applying softmax yourself and then passing the result to PyTorch’s loss is a silent bug: training runs, but on a doubly squashed signal. Feed logits to a loss that takes logits, which also keeps the log of tiny probabilities numerically stable.
Language models use the same loss over the vocabulary: each next token is one class, and perplexity is the average cross-entropy exponentiated.
Is focal loss better than cross-entropy?
Only when easy examples swamp the hard ones. Focal loss was introduced for object detectors facing “extreme foreground-background class imbalance”, by “reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples” (Lin et al., Focal Loss for Dense Object Detection, read 16 November 2026). For balanced classification, plain cross-entropy remains the default, and class weights (PyTorch’s weight argument) are the simpler first fix for imbalance. Judge either choice on held-out metrics such as recall, not on the loss value alone, since a falling training loss can still be overfitting.