Lesson 15 of 16Generalization

Dropout and Early Stopping

L2 regularization fights overfitting by taxing large weights. There are two other tools that work completely differently, one changes what happens during a forward pass, the other changes when you stop training altogether.

Dropout randomly silences a fraction of a layer's neurons on every single training forward pass, a different random subset each time. A neuron that got dropped contributes nothing that step, no output, no gradient. The surviving neurons get their outputs scaled up by 1 / (1 - rate) to keep the layer's total signal roughly the same size whether or not dropout is active. The effect: no neuron can afford to become the only one responsible for detecting some pattern, since it might vanish on any given step, so the network is pushed toward spreading responsibility across many neurons redundantly. At evaluation time (checking validation loss, or making real predictions), dropout is turned off entirely, every neuron participates, nothing is scaled.

Early stopping is almost embarrassingly simple by comparison: track validation loss during training, and if it keeps climbing while training loss keeps falling, that's the overfitting gap widening. Instead of training for a fixed number of epochs regardless of what happens, save the weights from whichever epoch had the lowest validation loss, and use those, even if training kept running long past that point.

Dropout on each forward pass, and knowing when to stop

Dropout is off: every neuron participates in every forward pass, as in every earlier lesson.

The green dot marks the lowest validation loss reached, epoch 22. Everything after it (faded) is the network continuing to overfit: training loss keeps improving, validation loss climbs back up. Early stopping just means saving the weights at that green dot and stopping there, instead of at whatever epoch training happened to end on.

best val loss (epoch 22)

0.374

val loss at epoch 399

1.135

Click "Resample forward pass" a few times with dropout on and watch a different random subset of neurons gray out each time, exactly what happens silently on every step of training. Now look at the loss curve: with dropout off, validation loss bottoms out early and climbs steadily for the rest of training, that green dot and the long faded tail after it is early stopping made visible. Toggle dropout on and re-run the comparison: the best validation loss lands in a similar range, but the climb afterward is far gentler, dropout is quietly doing some of the same job L2 does, just through a completely different mechanism.

Both, on top of the training loop you already know:

Python

import random

def dropout_forward(hidden_outputs, rate, rng):
    if rate <= 0:
        return hidden_outputs
    kept = []
    for h in hidden_outputs:
        if rng.random() >= rate:
            kept.append(h / (1 - rate))  # scale up survivors
        else:
            kept.append(0.0)
    return kept

rng = random.Random(0)
hidden_outputs = [0.8, -0.3, 0.5, 0.1]
for _ in range(3):
    print(dropout_forward(hidden_outputs, rate=0.3, rng=rng))

def early_stopping(val_losses, patience=20):
    best_epoch = min(range(len(val_losses)), key=lambda i: val_losses[i])
    stopped_early = (len(val_losses) - 1 - best_epoch) >= patience
    return best_epoch, val_losses[best_epoch], stopped_early

losses = [0.9, 0.5, 0.3, 0.25, 0.28, 0.35, 0.4, 0.45, 0.5, 0.55]
best_epoch, best_loss, stopped = early_stopping(losses, patience=3)
print(f"best epoch: {best_epoch}, best val loss: {best_loss}, would trigger early stop: {stopped}")

import random

def dropout_forward(hidden_outputs, rate, rng):
    if rate <= 0:
        return hidden_outputs
    kept = []
    for h in hidden_outputs:
        if rng.random() >= rate:
            kept.append(h / (1 - rate))  # scale up survivors
        else:
            kept.append(0.0)
    return kept

rng = random.Random(0)
hidden_outputs = [0.8, -0.3, 0.5, 0.1]
for _ in range(3):
    print(dropout_forward(hidden_outputs, rate=0.3, rng=rng))

def early_stopping(val_losses, patience=20):
    best_epoch = min(range(len(val_losses)), key=lambda i: val_losses[i])
    stopped_early = (len(val_losses) - 1 - best_epoch) >= patience
    return best_epoch, val_losses[best_epoch], stopped_early

losses = [0.9, 0.5, 0.3, 0.25, 0.28, 0.35, 0.4, 0.45, 0.5, 0.55]
best_epoch, best_loss, stopped = early_stopping(losses, patience=3)
print(f"best epoch: {best_epoch}, best val loss: {best_loss}, would trigger early stop: {stopped}")

Your turn

Implement best_epoch(val_losses): return the index of the lowest value in val_losses (the epoch you'd roll back to under early stopping).

Python

You now have every core tool this course set out to cover: neurons, layers, depth, initialization, activation choice, backpropagation, gradient checking, batching, three flavors of loss function, and three ways to keep a network honest instead of just memorizing. The capstone brings all of it together on one dataset a single straight line could never solve.

Why does dropout scale up the surviving neurons' outputs by 1 / (1 - rate) instead of leaving them unchanged?

← Overfitting and Regularization Capstone: A Multi-Class Spiral →