Backpropagation: The Chain Rule, by Hand

A real model's loss isn't one function of one variable — it's a long chain of operations composed together: multiply, add, square, multiply again, thousands of times over. The chain rule from calculus tells you how to find the derivative of a composed function: multiply the derivatives of each step together.

Backpropagation is just the chain rule, applied systematically, walking backward through every operation in the chain.

Let's trace it through a tiny example: a = 2, b = -3, c = a·b, d = c + 5, L = d². We want to know: how much does L change if we nudge a? Or b? That's ∂L/∂a and ∂L/∂b — and backprop computes both by starting at L and working backward, multiplying local derivatives as it goes.

Each step backward is one application of the chain rule: ∂L/∂d = 2d (the derivative of squaring), then ∂L/∂c = ∂L/∂d · ∂d/∂c = ∂L/∂d · 1 (since d = c + 5, addition just passes the gradient through), then ∂L/∂a = ∂L/∂c · ∂c/∂a = ∂L/∂c · b (since c = a·b, the local derivative is the other factor). Every gradient is the product of local derivatives chained together — nothing more exotic than that.

Now let's build the thing that actually does this — a tiny autograd engine. Each Value remembers what operation created it and how to push gradients backward through that operation. This is a miniature version of what PyTorch's .backward() does under the hood:

Now build the exact same expression from the diagram above and call .backward(). Watch the gradients match what you saw in the widget:

That's the whole trick: build a graph as you compute forward, then walk it backward once, multiplying local derivatives. Stack millions of these operations together — the exact same +, *, and a few more — and you get the training loop behind every modern language model. From here, we go back to language: tokenization, embeddings, and the self-attention mechanism, now with the machinery to eventually understand how they'd be trained.