Lesson 6 of 6

Overfitting and Regularization

The network from the last lesson has 4 hidden neurons and 2 inputs each — 8 weights, plus 4 hidden biases, plus 4 output weights and a bias: 17 numbers, all free to move however gradient descent likes. Given enough capacity and few enough training points, a network can drive the loss on those exact points to nearly zero by memorizing quirks and outliers instead of learning the actual pattern. That's overfitting.

One common fix is L2 regularization: add a penalty to the loss proportional to the sum of the squared weights, loss + λ · Σ(w²). Now the network isn't just rewarded for fitting the data — it's also taxed for having large weights. Since large, wild weights are exactly what's needed to carve tight, jagged regions around individual noisy points, this penalty pushes the network toward smoother, simpler boundaries.

Same noisy data, trained with vs. without L2

final loss: 0.205

Without regularization, the network is free to carve out tight little regions around noisy/mislabeled points — high training accuracy, but a boundary that likely won't generalize well.

This is the same XOR data as before, but with a handful of mislabeled-looking points scattered near the center. Toggle regularization off and the boundary twists itself to carve out little pockets around those noisy points — technically lower training loss, but almost certainly worse at generalizing to new data. Toggle it on and the boundary stays smoother, accepting a few "wrong" points near the noisy region rather than contorting around every one of them.

The only change from the previous lesson's training loop: the L2 term added directly into each gradient before the update:

Python

import math

def tanh(x):
    return math.tanh(x)

def dtanh(y):
    return 1 - y * y

def forward(hidden, output, x):
    h = [tanh(sum(w * xi for w, xi in zip(n["w"], x)) + n["b"]) for n in hidden]
    o = tanh(sum(w * hi for w, hi in zip(output["w"], h)) + output["b"])
    return h, o

def train_step(hidden, output, x, y, lr, l2):
    h, o = forward(hidden, output, x)
    dL_dOut = 2 * (o - y)
    dL_dZ2 = dL_dOut * dtanh(o)

# each weight's gradient now includes "+ 2 * l2 * w" — the L2 penalty's
    # own derivative, pulling every weight toward zero a little each step
    grad_out_w = [dL_dZ2 * hi + 2 * l2 * w for hi, w in zip(h, output["w"])]
    grad_out_b = dL_dZ2
    grad_hidden_z = [dL_dZ2 * output["w"][j] * dtanh(h[j]) for j in range(4)]
    grad_hidden_w = [
        [grad_hidden_z[j] * xi + 2 * l2 * w for xi, w in zip(x, n["w"])]
        for j, n in enumerate(hidden)
    ]

output["w"] = [w - lr * g for w, g in zip(output["w"], grad_out_w)]
    output["b"] -= lr * grad_out_b
    for j, n in enumerate(hidden):
        n["w"] = [w - lr * g for w, g in zip(n["w"], grad_hidden_w[j])]
        n["b"] -= lr * grad_hidden_z[j]

def weight_magnitude(hidden, output):
    total = sum(w * w for n in hidden for w in n["w"])
    total += sum(w * w for w in output["w"])
    return total

def fresh_network():
    hidden = [
        {"w": [0.5, -0.3], "b": 0.1}, {"w": [-0.4, 0.6], "b": -0.1},
        {"w": [0.3, 0.4], "b": 0.05}, {"w": [-0.6, -0.2], "b": 0.15},
    ]
    output = {"w": [0.4, -0.5, 0.3, 0.6], "b": 0.0}
    return hidden, output

for l2 in [0.0, 0.05]:
    hidden, output = fresh_network()
    for epoch in range(151):
        for x, y in [((0.2, 0.3), -1), ((-0.3, -0.2), -1), ((0.3, -0.2), 1), ((-0.2, 0.3), 1)]:
            train_step(hidden, output, x, y, 0.1, l2)
    print(f"l2={l2}: sum of squared weights = {weight_magnitude(hidden, output):.3f}")

That's the full arc of this course: a single neuron, a layer of them, stacked into an MLP, given a loss function, trained with gradient descent, and finally kept honest with regularization. Every one of these ideas — weighted sums, nonlinearities, backpropagation, gradient descent — is the exact same machinery underneath a transformer, just at a scale small enough to watch happen in your browser.

What does an L2 regularization penalty do to a network's training?

← Training on a 2D Toy Dataset Finish course →