Lesson 11 of 16Choosing a Loss Function

Multi-Class Classification: Softmax and Cross-Entropy

Hinge loss works when there are exactly two classes and one output score. Most real classification problems have more than two classes, is this photo a cat, a dog, a bird, or a fish? That needs K output neurons, one raw score (a logit) per class, and a way to turn K arbitrary numbers into something that behaves like a probability distribution: nonnegative, and summing to exactly 1.

Softmax does this: exponentiate every logit (so everything becomes positive), then divide each by the sum of all of them (so they add up to 1).

softmax(z)_i = exp(z_i) / Σⱼ exp(z_j)

The exponential is doing real work here, not just enforcing positivity: it means a logit just slightly larger than the others turns into a much larger share of the probability. Softmax doesn't just rank the classes, it amplifies confidence.

Once you have a probability distribution, cross-entropy loss scores it against the true class y:

loss = -log(P(y))

If the model puts nearly all its probability mass on the correct class, P(y) is close to 1 and -log(P(y)) is close to 0, almost no loss. If the model is confidently wrong, P(y) is close to 0 and -log(P(y)) shoots toward infinity, a severe penalty. Notice the shape: cross-entropy never bottoms out at exactly 0 the way hinge loss does, it always wants a little more confidence, right up until P(y) = 1.

Raw logits in, a probability distribution out, click a bar to set the true class

cat logit: 1.20dog logit: 0.40bird logit: -0.80fish logit: 2.10temperature: 1.00

cross-entropy loss = -log(P(fish)) = 0.497

Click any bar to mark it the true class and watch the loss respond. Dragging temperature below 1 sharpens the distribution toward the largest logit; above 1 flattens it toward uniform.

Drag any logit up and watch every bar move, not just that one, softmax is a joint function of all the logits at once, since they all share the same denominator. Click a low-probability bar to mark it the true class and watch the loss spike. The temperature slider previews something you'll see again in language model sampling: divide every logit by a number greater than 1 before softmax and the distribution flattens toward uniform; divide by a number less than 1 and it sharpens toward whichever class already has the highest logit.

The same two functions, matching the widget's default logits:

Python

import math

def softmax(logits, temperature=1.0):
    scaled = [z / temperature for z in logits]
    m = max(scaled)
    exps = [math.exp(z - m) for z in scaled]  # subtract max for numerical stability
    total = sum(exps)
    return [e / total for e in exps]

def cross_entropy(probs, true_index):
    return -math.log(max(probs[true_index], 1e-12))

classes = ["cat", "dog", "bird", "fish"]
logits = [1.2, 0.4, -0.8, 2.1]

probs = softmax(logits)
for c, p in zip(classes, probs):
    print(f"{c:>5}: {p:.3f}")

true_index = classes.index("fish")
print("loss =", round(cross_entropy(probs, true_index), 4))

import math

def softmax(logits, temperature=1.0):
    scaled = [z / temperature for z in logits]
    m = max(scaled)
    exps = [math.exp(z - m) for z in scaled]  # subtract max for numerical stability
    total = sum(exps)
    return [e / total for e in exps]

def cross_entropy(probs, true_index):
    return -math.log(max(probs[true_index], 1e-12))

classes = ["cat", "dog", "bird", "fish"]
logits = [1.2, 0.4, -0.8, 2.1]

probs = softmax(logits)
for c, p in zip(classes, probs):
    print(f"{c:>5}: {p:.3f}")

true_index = classes.index("fish")
print("loss =", round(cross_entropy(probs, true_index), 4))

Your turn

Implement cross_entropy_loss(probs, true_index): return -log(probs[true_index]), the negative log-probability the model assigned to the correct class.

Python

Softmax and cross-entropy generalize classification to any number of classes, and they're exactly what the capstone at the end of this course will train with. But loss functions aren't only for classification, sometimes the target is a number, not a category. That's next.

Why does softmax use an exponential instead of, say, just normalizing the logits directly (dividing each by their sum)?

← A Loss Function for Regression Training on a 2D Toy Dataset →