Lesson 16 of 16

Capstone: A Multi-Class Spiral

Every idea from this course lands in one place: a network that takes a 2D point and predicts which of three interleaved spiral arms it belongs to. No straight line, or even two straight lines, can separate three arms that wind around and around each other, this genuinely needs everything a hidden layer's nonlinearity can do, trained properly.

The network itself is nothing new: 2 inputs into 8 hidden neurons (tanh), into 3 output logits, one per class, softmax and cross-entropy loss exactly as in the multi-class classification lesson. What's new is that every training-mechanics lesson from this course now has a real dial to turn on the exact same network: batch size, L2 strength, and dropout.

Capstone: a 3-class spiral, no straight line (or two) can untangle

batch size

train loss

1.090

val loss

1.094

validation accuracy

42%

epoch 0

Filled dots are training points, white-ringed dots are the held-out validation set. The shaded background is the network's predicted class at every point in the plane, three colors, softmax-and-argmax over the raw logits, no straight line or pair of lines could ever separate three interleaved spiral arms. Click Train repeatedly and watch the shaded regions bend into matching spirals. Toggle batch size, L2, and dropout to see every piece from this course working together on one problem.

Filled dots are what the network trains on, white-ringed dots are held out and only ever used to compute validation loss and accuracy, exactly the train/val split from the overfitting lesson. Click Train repeatedly with the defaults (batch size 8, no L2, no dropout) and watch the shaded background bend from a meaningless initial pattern into three genuinely spiral-shaped regions, each color chasing its arm. Then start flipping switches: batch size 1 updates far more often per epoch but noisily, full-batch is smoother but slower to bend into shape, L2 and dropout both trade a bit of raw accuracy for a boundary that doesn't overfit the training points' exact noise.

The full forward and backward pass for one hidden layer feeding a 3-way softmax, the same math as the widget, just without a browser to draw it in:

Python

import math

def tanh(x):
    return math.tanh(x)

def dtanh(y):
    return 1 - y * y

def softmax(logits):
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    total = sum(exps)
    return [e / total for e in exps]

def forward(hidden, output, x):
    h = [tanh(n["w"][0] * x[0] + n["w"][1] * x[1] + n["b"]) for n in hidden]
    logits = [sum(w * hi for w, hi in zip(o["w"], h)) + o["b"] for o in output]
    return h, softmax(logits)

def train_step(hidden, output, x, true_class, lr):
    h, probs = forward(hidden, output, x)
    # softmax + cross-entropy gradient collapses to this one clean expression:
    # d(loss)/d(logit_k) = predicted_probability_k - (1 if k is the true class else 0)
    d_logits = [p - (1.0 if k == true_class else 0.0) for k, p in enumerate(probs)]

    for k, o in enumerate(output):
        for j in range(len(h)):
            o["w"][j] -= lr * d_logits[k] * h[j]
        o["b"] -= lr * d_logits[k]

    for j, n in enumerate(hidden):
        d_h = sum(d_logits[k] * output[k]["w"][j] for k in range(len(output)))
        d_z = d_h * dtanh(h[j])
        n["w"][0] -= lr * d_z * x[0]
        n["w"][1] -= lr * d_z * x[1]
        n["b"] -= lr * d_z

print("One training step ready: forward, softmax, cross-entropy gradient, backward, done.")

import math

def tanh(x):
    return math.tanh(x)

def dtanh(y):
    return 1 - y * y

def softmax(logits):
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    total = sum(exps)
    return [e / total for e in exps]

def forward(hidden, output, x):
    h = [tanh(n["w"][0] * x[0] + n["w"][1] * x[1] + n["b"]) for n in hidden]
    logits = [sum(w * hi for w, hi in zip(o["w"], h)) + o["b"] for o in output]
    return h, softmax(logits)

def train_step(hidden, output, x, true_class, lr):
    h, probs = forward(hidden, output, x)
    # softmax + cross-entropy gradient collapses to this one clean expression:
    # d(loss)/d(logit_k) = predicted_probability_k - (1 if k is the true class else 0)
    d_logits = [p - (1.0 if k == true_class else 0.0) for k, p in enumerate(probs)]

    for k, o in enumerate(output):
        for j in range(len(h)):
            o["w"][j] -= lr * d_logits[k] * h[j]
        o["b"] -= lr * d_logits[k]

    for j, n in enumerate(hidden):
        d_h = sum(d_logits[k] * output[k]["w"][j] for k in range(len(output)))
        d_z = d_h * dtanh(h[j])
        n["w"][0] -= lr * d_z * x[0]
        n["w"][1] -= lr * d_z * x[1]
        n["b"] -= lr * d_z

print("One training step ready: forward, softmax, cross-entropy gradient, backward, done.")

Your turn

Implement softmax_cross_entropy_gradient(probs, true_class): the gradient of cross-entropy loss with respect to each logit, given the softmax probabilities and the true class index. (It's the clean expression from the code above: prob - 1 for the true class, prob - 0 for every other class.)

Python

That's the full course: a single neuron's weighted sum and squash, layers of them stacked into depth, weights started from a sensible scale, gradients checked and watched for vanishing or exploding, three different loss functions for three different kinds of target, batches traded off for noise and speed, and three separate ways to keep a network honest instead of just memorizing. Every one of those ideas, unchanged in kind, only in scale, is what's running underneath a transformer the size of GPT. You've now built the machinery both worlds share.

Why does the gradient of softmax + cross-entropy loss simplify to just (predicted probability − true label), with no extra chain-rule factors?

← Dropout and Early Stopping Finish course →