Cross-Entropy Loss: Formula, Worked Example and PyTorch Use

LLM foundationsModels and inferencePublished By Simon Budziak

Cross-entropy loss is the negative log of the probability a model assigned to the correct class. It is the standard training loss for classification and for language models, which predict the next token as a class. It costs almost nothing when the model is confident and right, and grows without limit when it is confident and wrong.

How is cross-entropy loss calculated?

For one example, the loss is -log p, where p is the probability the model gave the true class. The general form, -Σ y·log(p) summed over classes, reduces to that because the one-hot label y is 1 for the true class and 0 elsewhere. Training averages it over a batch and gradient descent pushes it down.

A worked example with three classes (cat, dog, bird), natural log throughout:

Step Cat Dog Bird
Logits from the network 2.0 1.0 0.1
exp(logit) 7.389 2.718 1.105
Softmax probability (divide by 11.212) 0.659 0.242 0.099
Loss if this is the true class 0.417 1.417 2.317

If the image is a cat, the loss is -ln(0.659) = 0.417. If it is a bird, the same prediction costs 2.317. A model that guesses uniformly over three classes scores ln(3) = 1.099, a useful baseline: a validation loss at or above it means the model does no better than guessing. The penalty is logarithmic, so the last bit of confidence in a wrong answer costs the most: p = 0.5 costs 0.693, p = 0.01 costs 4.605. That is why cross-entropy rewards honest probabilities, the property model calibration measures.

What is the difference between binary and categorical cross-entropy?

They are the same idea applied to different label shapes.

Variant Use it for Output layer PyTorch Keras
Binary cross-entropy Two classes, or several independent yes/no labels One sigmoid per label BCEWithLogitsLoss BinaryCrossentropy
Categorical cross-entropy One label out of many, one-hot targets Softmax over classes CrossEntropyLoss CategoricalCrossentropy
Sparse categorical cross-entropy One label out of many, integer targets Softmax over classes CrossEntropyLoss (class indices) SparseCategoricalCrossentropy

Binary cross-entropy is -[y·log p + (1 - y)·log(1 - p)], the two-class case of the same formula listed with other losses under loss function. Pick by label structure, not by class count: a multi-label classification problem with ten tags still uses binary cross-entropy, ten times.

What is the loss function for softmax, and why do frameworks want logits?

Softmax is almost always paired with cross-entropy, and frameworks fuse the two. PyTorch’s CrossEntropyLoss expects “the unnormalized logits for each class” and, for class-index targets, is “equivalent to applying LogSoftmax on an input, followed by NLLLoss”; its defaults are reduction='mean' and label_smoothing=0.0 (PyTorch 2.15 CrossEntropyLoss, read 16 November 2026). Keras goes the other way: its cross-entropy losses default to from_logits=False and assume y_pred already holds probabilities (Keras probabilistic losses, read 16 November 2026). Applying softmax yourself and then passing the result to PyTorch’s loss is a silent bug: training runs, but on a doubly squashed signal. Feed logits to a loss that takes logits, which also keeps the log of tiny probabilities numerically stable.

Language models use the same loss over the vocabulary: each next token is one class, and perplexity is the average cross-entropy exponentiated.

Is focal loss better than cross-entropy?

Only when easy examples swamp the hard ones. Focal loss was introduced for object detectors facing “extreme foreground-background class imbalance”, by “reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples” (Lin et al., Focal Loss for Dense Object Detection, read 16 November 2026). For balanced classification, plain cross-entropy remains the default, and class weights (PyTorch’s weight argument) are the simpler first fix for imbalance. Judge either choice on held-out metrics such as recall, not on the loss value alone, since a falling training loss can still be overfitting.

This entry was drafted with AI assistance.

Frequently asked questions

What is the difference between cross-entropy and log loss?

For classification they are the same quantity. Log loss is the name statisticians and scikit-learn users tend to use, cross-entropy is the name in deep learning frameworks.

What is a good cross-entropy loss value?

There is no universal threshold. Compare it with the loss of a model that guesses uniformly, which is the natural log of the number of classes, and with your own validation loss over time.

Summarize this page with

Train your team to build this