Lesson 5 of 16Setting Up a Network

Choosing a Nonlinearity

Every neuron you've built so far squashes its weighted sum through tanh. That was one choice among several, and the choice matters: it's not just about what the output looks like, it's about what the derivative looks like, since that derivative is exactly what backpropagation multiplies through on its way backward.

Sigmoid, σ(x) = 1 / (1 + e⁻ˣ), squashes into (0, 1). tanh, which you've already used, squashes into (−1, 1) and is really just a rescaled sigmoid. Both share the same weakness: far from x = 0, both curves go nearly flat, and a nearly flat curve has a derivative close to 0. That's saturation, the same phenomenon from the weight initialization lesson, now viewed from the activation function's side rather than the weight-scale side.

ReLU (Rectified Linear Unit), max(0, x), takes a completely different approach: instead of squashing, it just zeroes out anything negative and passes anything positive through unchanged. Its derivative is either exactly 1 (for positive inputs) or exactly 0 (for negative inputs), never a fraction shrinking toward zero. That's why ReLU became the default in most deep networks: no saturation on the positive side, cheap to compute, and gradients that pass through at full strength instead of shrinking with every layer.

ReLU's cost is the dying ReLU problem: any neuron whose weighted sum lands negative has an exactly-zero derivative, exactly zero gradient reaches its weights, and if that keeps happening across training, that neuron is permanently stuck contributing nothing. Leaky ReLU patches this with a small nonzero slope on the negative side, usually 0.01, so a neuron in the negative region still gets a (small) gradient and has a chance to recover.

Four activation functions, one shared derivative story

x: 1.50

Sigmoid(x)

0.818

Sigmoid'(x)

0.149

The red band marks where |Sigmoid'(x)| < 0.008, the gradient is nearly dead there. Sigmoid and tanh both saturate on both sides. ReLU is dead for every negative input, exactly zero derivative, not just small. Leaky ReLU's whole point is keeping that negative-side slope at 0.01 instead of 0, so it never fully dies.

Switch between the four functions and drag x from one side to the other. Sigmoid and tanh both light up the red danger band on both ends, ReLU's danger band covers the entire negative half exactly, not approximately, and Leaky ReLU never enters the danger band at all, its derivative never drops all the way to 0.

All four, with their derivatives, in one place:

Python

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def dsigmoid(x):
    s = sigmoid(x)
    return s * (1 - s)

def tanh(x):
    return math.tanh(x)

def dtanh(x):
    return 1 - math.tanh(x) ** 2

def relu(x):
    return max(0, x)

def drelu(x):
    return 1.0 if x > 0 else 0.0

def leaky_relu(x, alpha=0.01):
    return x if x > 0 else alpha * x

def dleaky_relu(x, alpha=0.01):
    return 1.0 if x > 0 else alpha

for x in [-3, -0.5, 0.5, 3]:
    print(f"x={x:>5}: sigmoid'={dsigmoid(x):.4f}  tanh'={dtanh(x):.4f}  relu'={drelu(x):.2f}  leaky_relu'={dleaky_relu(x):.2f}")

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def dsigmoid(x):
    s = sigmoid(x)
    return s * (1 - s)

def tanh(x):
    return math.tanh(x)

def dtanh(x):
    return 1 - math.tanh(x) ** 2

def relu(x):
    return max(0, x)

def drelu(x):
    return 1.0 if x > 0 else 0.0

def leaky_relu(x, alpha=0.01):
    return x if x > 0 else alpha * x

def dleaky_relu(x, alpha=0.01):
    return 1.0 if x > 0 else alpha

for x in [-3, -0.5, 0.5, 3]:
    print(f"x={x:>5}: sigmoid'={dsigmoid(x):.4f}  tanh'={dtanh(x):.4f}  relu'={drelu(x):.2f}  leaky_relu'={dleaky_relu(x):.2f}")

Your turn

Implement leaky_relu(x, alpha=0.01): return x for positive x, and alpha * x for negative or zero x.

Python

Saturation in a single activation function is one thing. Chain many layers of saturating activations together, or many layers of anything, and the small derivatives (or large ones) compound multiplicatively. That's next: vanishing and exploding gradients.

What makes ReLU's dying-neuron problem different from sigmoid/tanh saturation?

← Starting From Somewhere Sensible Backpropagation: The Chain Rule, by Hand →