Lesson 5 of 6

Training on a 2D Toy Dataset

Everything comes together now. The training loop is exactly four steps, repeated over and over: forward pass (compute the network's output), loss (score how wrong it is), backward pass (compute how every weight contributed to that wrongness), update (nudge every weight opposite its gradient). That's it — that's the entire training loop, for this tiny network and for GPT-scale ones alike.

To make this non-trivial, the data below is arranged in an XOR pattern: points in the top-right and bottom-left quadrants belong to one class, points in the top-left and bottom-right belong to the other. No straight line can separate these two classes — you need a hidden layer's nonlinearity to solve it. This exact problem is famous in neural network history for exactly that reason.

Predict before you look

Before hitting Play: what shape do you think the final boundary needs to take to correctly separate all four quadrant clusters — and can a network with just 4 hidden neurons actually pull it off?

Hit Play and watch the shaded background — the network's prediction at every point in the plane — bend from a meaningless initial pattern into two blue and two amber regions that actually separate the dots by color. That shifting shape is gradient descent, visualized directly: every epoch nudges every weight a little, and a few hundred nudges later, the network has carved out a boundary no single straight line ever could.

The dataset itself isn't fixed either — click empty space on the grid to add your own point of either color, or click an existing point to remove it. Every change retrains the same network from scratch on whatever's on the board, so you can build a pattern of your own and watch this exact training loop try to solve it.

The complete training loop, including the hand-derived backward pass:

Python

import math

def tanh(x):
    return math.tanh(x)

def dtanh(y):
    return 1 - y * y

points = [
    ((0.8, 0.9), 1), ((1.1, 0.7), 1), ((0.6, 1.2), 1), ((1.3, 1.1), 1), ((0.9, 0.6), 1),
    ((-0.9, -0.8), 1), ((-1.2, -0.6), 1), ((-0.7, -1.1), 1), ((-1.1, -1.3), 1), ((-0.6, -0.9), 1),
    ((0.9, -0.8), -1), ((1.2, -0.6), -1), ((0.7, -1.1), -1), ((1.1, -1.3), -1), ((0.6, -0.9), -1),
    ((-0.8, 0.9), -1), ((-1.1, 0.7), -1), ((-0.6, 1.2), -1), ((-1.3, 1.1), -1), ((-0.9, 0.6), -1),
]

hidden = [
    {"w": [0.5, -0.3], "b": 0.1},
    {"w": [-0.4, 0.6], "b": -0.1},
    {"w": [0.3, 0.4], "b": 0.05},
    {"w": [-0.6, -0.2], "b": 0.15},
]
output = {"w": [0.4, -0.5, 0.3, 0.6], "b": 0.0}

def forward(x):
    h = [tanh(sum(w * xi for w, xi in zip(n["w"], x)) + n["b"]) for n in hidden]
    o = tanh(sum(w * hi for w, hi in zip(output["w"], h)) + output["b"])
    return h, o

def train_step(x, y, lr):
    h, o = forward(x)
    dL_dOut = 2 * (o - y)
    dL_dZ2 = dL_dOut * dtanh(o)

grad_out_w = [dL_dZ2 * hi for hi in h]
    grad_out_b = dL_dZ2
    grad_hidden_z = [dL_dZ2 * output["w"][j] * dtanh(h[j]) for j in range(4)]
    grad_hidden_w = [[grad_hidden_z[j] * xi for xi in x] for j in range(4)]

output["w"] = [w - lr * g for w, g in zip(output["w"], grad_out_w)]
    output["b"] -= lr * grad_out_b
    for j, n in enumerate(hidden):
        n["w"] = [w - lr * g for w, g in zip(n["w"], grad_hidden_w[j])]
        n["b"] -= lr * grad_hidden_z[j]

def total_loss():
    return sum((forward(x)[1] - y) ** 2 for x, y in points) / len(points)

lr = 0.1
for epoch in range(151):
    if epoch % 30 == 0:
        print(f"epoch {epoch:>3}: loss = {total_loss():.4f}")
    for x, y in points:
        train_step(x, y, lr)

print(f"epoch 150: loss = {total_loss():.4f}")

One last question before this course wraps up: what happens if the network has more than enough capacity to fit the data — including its noise?

Go deeper

Gradient descent, how neural networks learn | Deep learning, chapter 23Blue1Brown

Explains cost functions and gradient descent visually — the same training loop you just watched carve out a decision boundary above.

Why can't a single straight line separate the XOR-pattern data in this lesson?

← A Loss Function for Classification Overfitting and Regularization →