Lesson 3 of 20

Backpropagation: The Chain Rule, by Hand

A real model's loss isn't one function of one variable, it's a long chain of operations composed together: multiply, add, square, multiply again, thousands of times over. The chain rule from calculus tells you how to find the derivative of a composed function: multiply the derivatives of each step together.

Backpropagation is just the chain rule, applied systematically, walking backward through every operation in the chain.

Let's trace it through a tiny example: a = 2, b = -3, c = a·b, d = c + 5, L = d². We want to know: how much does L change if we nudge a? Or b? That's ∂L/∂a and ∂L/∂b, and backprop computes both by starting at L and working backward, multiplying local derivatives as it goes.

Step backward through L = (a·b + 5)² to see gradients flow

Forward pass computed. Click "Step backward" to compute gradients via the chain rule.

Each step backward is one application of the chain rule: ∂L/∂d = 2d (the derivative of squaring), then ∂L/∂c = ∂L/∂d · ∂d/∂c = ∂L/∂d · 1 (since d = c + 5, addition just passes the gradient through), then ∂L/∂a = ∂L/∂c · ∂c/∂a = ∂L/∂c · b (since c = a·b, the local derivative is the other factor). Every gradient is the product of local derivatives chained together, nothing more exotic than that.

Now let's build the thing that actually does this: a tiny autograd engine. Each Value remembers what operation created it and how to push gradients backward through that operation. This is a miniature version of what PyTorch's .backward() does under the hood:

Python

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def __pow__(self, power):
        out = Value(self.data ** power, (self,), f"**{power}")
        def _backward():
            self.grad += power * (self.data ** (power - 1)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

print("Value class defined.")

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def __pow__(self, power):
        out = Value(self.data ** power, (self,), f"**{power}")
        def _backward():
            self.grad += power * (self.data ** (power - 1)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

print("Value class defined.")

Now build the exact same expression from the diagram above and call .backward(). Watch the gradients match what you saw in the widget:

Python

a = Value(2.0)
b = Value(-3.0)
c = a * b
d = c + 5
L = d ** 2

L.backward()

print("L =", L.data)
print("a.grad =", a.grad)
print("b.grad =", b.grad)
print("c.grad =", c.grad)
print("d.grad =", d.grad)

a = Value(2.0)
b = Value(-3.0)
c = a * b
d = c + 5
L = d ** 2

L.backward()

print("L =", L.data)
print("a.grad =", a.grad)
print("b.grad =", b.grad)
print("c.grad =", c.grad)
print("d.grad =", d.grad)

That's the whole trick: build a graph as you compute forward, then walk it backward once, multiplying local derivatives. Stack millions of these operations together (the exact same +, *, and a few more) and you get the training loop behind every modern language model. From here, we go back to language: tokenization, embeddings, and the self-attention mechanism, now with the machinery to eventually understand how they'd be trained.

Go deeper

Backpropagation calculus | Deep learning, chapter 43Blue1Brown

Formalizes the chain-rule math behind backpropagation with the same visual style, a good companion to the graph-walking intuition above.

Neural Networks: Zero to HeroAndrej Karpathy

A from-scratch, code-first series starting with micrograd, an autograd engine built the same way as this lesson's, then scaled all the way up to GPT.

Given a=2, b=-3, c=a·b, d=c+5, L=d², what is a.grad after calling L.backward()?

← Derivatives: How Change Flows Character-Level & Word-Level Tokenization →