A Loss Function for Classification
Suppose the MLP's single output is meant to answer a yes/no question: positive means one class, negative means the other. To train it, you need a loss function — something that scores how wrong a prediction is, in a way that's differentiable, so gradient descent (from the derivatives lesson, all the way back in Zero to GPT) has something to work with.
A simple, effective choice for this is hinge loss. For a true label y of +1 or −1 and the network's raw score:
loss = max(0, 1 − y·score)
Read it as: if the score is confidently on the correct side (large and same sign as y), the loss is exactly 0 — no penalty, nothing more to learn from this example. If it's on the wrong side, or right-but-not-confident, the loss is positive and grows the more wrong it is.
Hinge loss: max(0, 1 − label × score)
Total loss: 3.85
Notice the two flagged rows: a wrong-side prediction and a right-side-but-unconfident one both get penalized, while confidently-correct predictions contribute nothing. Add up every point's loss and you get a single number summarizing "how bad is the network right now" — exactly what gradient descent needs to improve it.
Matching numbers, in Python:
Your turn
Using the hinge_loss function above, implement total_loss(points): sum hinge_loss across every (x, label, score) triple in points.
You now have every ingredient: a network that produces a score, and a loss that scores the score. All that's left is running gradient descent to actually make the loss go down — on real, two-dimensional data.
Under hinge loss, what happens to a prediction that's already confidently correct?