Lesson 10 of 16Choosing a Loss Function

A Loss Function for Regression

Hinge loss and cross-entropy both assume the target is a category: which class is this? But plenty of real problems ask for a number instead: what temperature will it be tomorrow, how many seconds until this process finishes, what's this house worth? That's regression, and it needs a different loss, one that scores how far a predicted number is from the true number, not which side of a decision boundary it landed on.

Mean squared error (MSE) is the standard choice:

loss = (prediction − target)²

averaged across every example. Squaring does two things at once: it makes the loss always nonnegative (an error of -2 and an error of +2 are equally bad), and it penalizes large errors much more than small ones, an error twice as big contributes four times the loss. The output neuron for a regression network also usually drops its activation function entirely, no tanh, no sigmoid, just the raw weighted sum, since the target can be any real number and squashing it into (−1, 1) would make large or small targets impossible to represent.

Fitting a curve to noisy points: minimizing mean squared error

epoch 0 · mean squared error = 0.2243

The red segments are each point's residual, the gap between its true y and the curve's current prediction. MSE is the average of those gaps squared. Click Train Step and watch the residuals shrink as the blue curve bends to reduce that average.

Each red segment is one point's residual: the vertical gap between its true value and the curve's current prediction directly below it. MSE is just the average of those gaps, squared. Click Train Step repeatedly and watch the blue curve bend to reduce that average, the residual segments visibly shrink as it does. Notice this is the exact same forward pass, backward pass, update loop from the training-a-classifier lesson, only the loss function and the output layer's activation changed.

MSE and its gradient, the only two things that differ from a classifier's training loop:

Python

def mse(predictions, targets):
    return sum((p - t) ** 2 for p, t in zip(predictions, targets)) / len(predictions)

def mse_gradient(prediction, target):
    # d/d(prediction) of (prediction - target)^2 is 2 * (prediction - target)
    return 2 * (prediction - target)

predictions = [0.9, -0.2, 1.4, 0.1]
targets =     [1.0, -0.5, 1.1, 0.6]

print("MSE:", round(mse(predictions, targets), 4))
for p, t in zip(predictions, targets):
    print(f"prediction={p:+.1f}  target={t:+.1f}  gradient={mse_gradient(p, t):+.2f}")

def mse(predictions, targets):
    return sum((p - t) ** 2 for p, t in zip(predictions, targets)) / len(predictions)

def mse_gradient(prediction, target):
    # d/d(prediction) of (prediction - target)^2 is 2 * (prediction - target)
    return 2 * (prediction - target)

predictions = [0.9, -0.2, 1.4, 0.1]
targets =     [1.0, -0.5, 1.1, 0.6]

print("MSE:", round(mse(predictions, targets), 4))
for p, t in zip(predictions, targets):
    print(f"prediction={p:+.1f}  target={t:+.1f}  gradient={mse_gradient(p, t):+.2f}")

Your turn

Implement mse(predictions, targets): the average of (prediction - target) squared across all pairs.

Python

You've now seen the full loss-function picture: hinge for binary margins, cross-entropy for multi-class probabilities, MSE for continuous targets. Every training loop in this course has updated its weights after looking at the entire dataset at once. Real datasets are far too large for that. Next: what changes when you only look at a few examples per update.

Why does a regression network's output neuron usually skip the activation function entirely?

← A Loss Function for Classification Multi-Class Classification: Softmax and Cross-Entropy →