Lesson 4 of 16Setting Up a Network

Starting From Somewhere Sensible

Every weight in the MLP you've been building had to start somewhere. So far you've been handed fixed numbers without asking why those numbers. It turns out the starting point matters enormously, badly chosen initial weights can make a network unable to learn at all, long before backpropagation ever gets a say.

The most tempting bad idea: initialize every weight to exactly 0. It seems harmless, but every neuron in a layer then computes the exact same weighted sum from the exact same inputs, produces the exact same output, and (once you get to backprop) receives the exact same gradient. Every neuron in that layer updates identically, forever. You could have a hidden layer with 1,000 neurons and it would behave like a layer of 1, permanently. This is called the symmetry problem, and it's not something training can ever fix on its own, since nothing ever pushes the neurons apart.

The other tempting bad idea: initialize weights to large random numbers, hoping randomness alone breaks the symmetry. It does, but if the weights are too large, the weighted sums z going into each neuron's tanh land far out in its saturated tails, where the output barely changes no matter how z moves and the local gradient is nearly zero. A network that starts saturated is a network that starts numb.

The fix is to pick a scale for the random weights that keeps the weighted sums in a sane range, not too small (which wastes the network's capacity, everything stays near-linear) and not too large (which saturates it). Two standard formulas do this by looking at how many inputs (fan_in) and outputs (fan_out) a layer has: Xavier/Glorot init samples uniformly from ±√(6 / (fan_in + fan_out)), tuned for tanh-like activations. He init samples from a wider ±√(6 / fan_in), tuned for ReLU-like activations that zero out half their input range. Both are just "random, but scaled on purpose."

How a preset spreads out a layer's starting weights

scale: 1.00×

Hidden layer weights (4 neurons × 2 inputs each)

-2.44

-2.19

2.38

1.00

0.11

-0.47

-0.17

-1.30

Different starting weights per neuron is what lets each one eventually specialize.

Pre-activation z values across 40 sample inputs

-606

Amber bars (|z| > 3) are deep in tanh's saturated tails, barely responsive to further changes. 14% of sampled activations are saturated at this scale.

Switch to Zero and watch every weight cell read the exact same number, that's the symmetry problem made visible. Switch to Large Random and push the scale slider up: the histogram's amber bars (saturated activations) take over almost the whole distribution. Xavier and He keep the histogram centered even as you nudge the scale, because they were derived specifically to counteract the number of inputs feeding each neuron.

The same three ideas, in code, sampling a hidden layer's weights each way:

Python

import random
import math

random.seed(0)

FAN_IN = 2
FAN_OUT = 4

def zero_init():
    return [0.0 for _ in range(FAN_IN)]

def large_random_init():
    return [random.uniform(-2.5, 2.5) for _ in range(FAN_IN)]

def xavier_init():
    limit = math.sqrt(6 / (FAN_IN + FAN_OUT))
    return [random.uniform(-limit, limit) for _ in range(FAN_IN)]

def he_init():
    limit = math.sqrt(6 / FAN_IN)
    return [random.uniform(-limit, limit) for _ in range(FAN_IN)]

for name, init_fn in [("zero", zero_init), ("large", large_random_init), ("xavier", xavier_init), ("he", he_init)]:
    weights = [init_fn() for _ in range(4)]  # one per neuron
    print(f"{name:>6}: {weights}")

import random
import math

random.seed(0)

FAN_IN = 2
FAN_OUT = 4

def zero_init():
    return [0.0 for _ in range(FAN_IN)]

def large_random_init():
    return [random.uniform(-2.5, 2.5) for _ in range(FAN_IN)]

def xavier_init():
    limit = math.sqrt(6 / (FAN_IN + FAN_OUT))
    return [random.uniform(-limit, limit) for _ in range(FAN_IN)]

def he_init():
    limit = math.sqrt(6 / FAN_IN)
    return [random.uniform(-limit, limit) for _ in range(FAN_IN)]

for name, init_fn in [("zero", zero_init), ("large", large_random_init), ("xavier", xavier_init), ("he", he_init)]:
    weights = [init_fn() for _ in range(4)]  # one per neuron
    print(f"{name:>6}: {weights}")

Your turn

Implement xavier_limit(fan_in, fan_out): return the Xavier/Glorot uniform limit sqrt(6 / (fan_in + fan_out)).

Python

Initialization sets the starting point. What a neuron does with its inputs from there, tanh, ReLU, or something else, shapes everything about how gradients behave once training begins. That's next.

Why is initializing every weight to zero a problem, even though it's not literally saturated?

← Stacking into an MLP Choosing a Nonlinearity →