Lesson 7 of 16Backpropagation & Gradient Flow

Vanishing and Exploding Gradients

Backpropagation computes a gradient by multiplying local derivatives together, one per layer, all the way from the loss back to the weight you care about. That's fine for a handful of layers. But multiplication compounds: multiply ten numbers each a bit less than 1 and the product collapses toward 0. Multiply ten numbers each a bit more than 1 and the product explodes toward infinity. Depth, the very thing that makes networks powerful, is also what makes this a real problem.

Vanishing gradients: if every layer's local derivative is comfortably under 1, perhaps tanh sitting a bit into its saturated region, the product shrinks geometrically with depth. By the time the gradient reaches an early layer, it can be so close to 0 that the update is imperceptible. That layer stops learning, not because it's already correct, but because the signal that would correct it never arrives with any strength.

Exploding gradients are the mirror image: local derivatives comfortably over 1, and the product grows geometrically instead. Weight updates become enormous, the loss swings wildly instead of descending smoothly, and training can diverge outright. tanh's own saturation actually puts a natural brake on this (a saturating function's derivative can't stay above 1 for long), which is part of why exploding gradients show up more with unbounded activations like ReLU, nothing caps how large its output, or its local derivative's contribution through a large weight, can get.

Watching one gradient chain through a stack of layers

depth: 8 layersweight scale: 1.00

1.0000

out

Bar height is log-scaled so both vanishing and exploding gradients stay visible on the same chart. Each step multiplies the running gradient by one more layer's local derivative and the weight scale, exactly the chain rule, applied 8 times in a row.

Set weight scale low and depth high with either activation, and step backward: the gradient shrinks by roughly the same factor every layer, a classic vanishing chain. Now switch to ReLU and push the weight scale above 1, step backward again: the gradient grows every layer instead, genuinely exploding. Try that same high weight scale with tanh, it still vanishes, because tanh's own saturation shrinks its local derivative faster than the weight scale can grow the product. Same chain-rule math, very different outcomes depending on what's being chained.

The same backward chain, computed directly, watch how fast it collapses:

Python

import math

def tanh(x):
    return math.tanh(x)

def dtanh(y):
    return 1 - y * y

def chain_gradient(depth, weight_scale, x0=0.8):
    h = x0
    hs = [h]
    for _ in range(depth):
        h = tanh(weight_scale * h)
        hs.append(h)

    grad = 1.0  # seed: dLoss/dh at the last layer
    grads = [grad]
    for i in range(depth, 0, -1):
        grad = grad * dtanh(hs[i]) * weight_scale
        grads.append(grad)
    return list(reversed(grads))

for depth in [2, 4, 8, 16]:
    grads = chain_gradient(depth, weight_scale=0.6)
    print(f"depth={depth:>2}: gradient reaching layer 0 = {grads[0]:.2e}")

import math

def tanh(x):
    return math.tanh(x)

def dtanh(y):
    return 1 - y * y

def chain_gradient(depth, weight_scale, x0=0.8):
    h = x0
    hs = [h]
    for _ in range(depth):
        h = tanh(weight_scale * h)
        hs.append(h)

    grad = 1.0  # seed: dLoss/dh at the last layer
    grads = [grad]
    for i in range(depth, 0, -1):
        grad = grad * dtanh(hs[i]) * weight_scale
        grads.append(grad)
    return list(reversed(grads))

for depth in [2, 4, 8, 16]:
    grads = chain_gradient(depth, weight_scale=0.6)
    print(f"depth={depth:>2}: gradient reaching layer 0 = {grads[0]:.2e}")

Your turn

Implement chained_derivative(local_derivatives): multiply a list of per-layer local derivatives together to get the total gradient through the whole chain (the chain rule, applied repeatedly).

Python

Weight initialization, activation choice, and now depth itself all feed into whether gradients survive the trip backward. There's a second, completely separate way loss functions get used, though: not to classify categories, but to predict a number directly. That's next.

Why does tanh tend to vanish rather than explode, even when the weight scale multiplying it is large?

← Backpropagation: The Chain Rule, by Hand Gradient Checking →