Overfitting and Regularization

The network from earlier lessons has 4 hidden neurons and 2 inputs each — 8 weights, plus 4 hidden biases, plus 4 output weights and a bias: 17 numbers, all free to move however gradient descent likes. Given enough capacity and few enough training points, a network can drive the loss on those exact points to nearly zero by memorizing quirks and outliers instead of learning the actual pattern. That's overfitting, and the only way to actually detect it is to check performance on data the network never trained on.

That's what a validation set is for: a held-out slice of data, set aside before training starts, that the network's weights never once get updated from. Every training loop so far in this course only ever looked at one number, the training loss. From here on, the real question is two numbers: training loss (how well it fits what it's seen) and validation loss (how well it does on what it hasn't). When those two numbers track together, the network is learning the actual pattern. When training loss keeps falling while validation loss stalls or climbs back up, that gap is overfitting, made visible.

One common fix is L2 regularization: add a penalty to the loss proportional to the sum of the squared weights, loss + λ · Σ(w²). Now the network isn't just rewarded for fitting the data, it's also taxed for having large weights. Since large, wild weights are exactly what's needed to carve tight, jagged regions around individual noisy points, this penalty pushes the network toward smoother, simpler boundaries, ones that tend to generalize better precisely because they stop chasing every training point.

The training set includes a handful of deliberately mislabeled points near the center. The validation set is entirely separate points, some sitting right next to those mislabeled ones, but carrying their true, correct label. With L2 off, the blue training-loss line settles low while the red validation-loss line stays noticeably higher, the shaded gap between them, the network twisted its boundary to accommodate the mislabeled training points, and nearby validation points paid the price. Toggle L2 on and watch that shaded gap visibly shrink.

The only change from a plain training loop: the L2 term added directly into each gradient before the update, and a second, separate loss computed on validation data that never appears in train_step at all:

That's the full arc from a single neuron to a network you can genuinely trust the evaluation of: a weighted sum, a nonlinearity, stacked into layers, given a loss function, trained with gradient descent, checked against gradients you never trust blindly, and now evaluated on data it never got to memorize. Every one of these ideas, weighted sums, nonlinearities, backpropagation, gradient descent, generalization, is the exact same machinery underneath a transformer, just at a scale small enough to watch happen in your browser.