Lesson 6 of 16Backpropagation & Gradient Flow

Backpropagation: The Chain Rule, by Hand

You've built a neuron, a layer, and a full MLP so far, but their weights are still just the arbitrary numbers you were handed. To actually train them, you need to know how much nudging any one weight would change the loss. A real network's loss isn't one function of one variable, it's a long chain of operations composed together: multiply, add, square, multiply again, thousands of times over. The chain rule from calculus tells you how to find the derivative of a composed function: multiply the derivatives of each step together.

Backpropagation is just the chain rule, applied systematically, walking backward through every operation in the chain.

Let's trace it through a tiny example: a = 2, b = -3, c = a·b, d = c + 5, L = d². We want to know: how much does L change if we nudge a? Or b? That's ∂L/∂a and ∂L/∂b, and backprop computes both by starting at L and working backward, multiplying local derivatives as it goes.

Step backward through L = (a·b + 5)² to see gradients flow

Forward pass computed. Click "Step backward" to compute gradients via the chain rule.

Each step backward is one application of the chain rule: ∂L/∂d = 2d (the derivative of squaring), then ∂L/∂c = ∂L/∂d · ∂d/∂c = ∂L/∂d · 1 (since d = c + 5, addition just passes the gradient through), then ∂L/∂a = ∂L/∂c · ∂c/∂a = ∂L/∂c · b (since c = a·b, the local derivative is the other factor). Every gradient is the product of local derivatives chained together, nothing more exotic than that.

Now let's build the thing that actually does this: a tiny autograd engine. You already met a piece of it back in the neuron lesson, a Value class with a .tanh() method bolted on for the forward pass. Here's where that class actually comes from. Each Value remembers what operation created it and how to push gradients backward through that operation, a miniature version of what PyTorch's .backward() does under the hood:

Python

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def __pow__(self, power):
        out = Value(self.data ** power, (self,), f"**{power}")
        def _backward():
            self.grad += power * (self.data ** (power - 1)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

print("Value class defined.")

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def __pow__(self, power):
        out = Value(self.data ** power, (self,), f"**{power}")
        def _backward():
            self.grad += power * (self.data ** (power - 1)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

print("Value class defined.")

Now build the exact same expression from the diagram above and call .backward(). Watch the gradients match what you saw in the widget:

Python

a = Value(2.0)
b = Value(-3.0)
c = a * b
d = c + 5
L = d ** 2

L.backward()

print("L =", L.data)
print("a.grad =", a.grad)
print("b.grad =", b.grad)
print("c.grad =", c.grad)
print("d.grad =", d.grad)

a = Value(2.0)
b = Value(-3.0)
c = a * b
d = c + 5
L = d ** 2

L.backward()

print("L =", L.data)
print("a.grad =", a.grad)
print("b.grad =", b.grad)
print("c.grad =", c.grad)
print("d.grad =", d.grad)

That's the whole trick: build a graph as you compute forward, then walk it backward once, multiplying local derivatives. Stack millions of these operations together (the exact same +, *, and a few more) and you get the training loop behind every modern network, from this four-neuron toy up to GPT-scale models. But chaining derivatives has a dark side: what happens when you multiply many small numbers together, or many large ones? That's next.

Go deeper

Backpropagation calculus | Deep learning, chapter 43Blue1Brown

Formalizes the chain-rule math behind backpropagation with the same visual style, a good companion to the graph-walking intuition above.

Neural Networks: Zero to HeroAndrej Karpathy

A from-scratch, code-first series starting with micrograd, an autograd engine built the same way as this lesson's, then scaled all the way up to GPT.

Given a=2, b=-3, c=a·b, d=c+5, L=d², what is a.grad after calling L.backward()?

← Choosing a Nonlinearity Vanishing and Exploding Gradients →