Derivatives: How Change Flows

Before we can talk about how a language model learns, we need one idea from calculus: the derivative.

A derivative answers a simple question: if I nudge the input a tiny bit, how much — and in which direction — does the output move? That's it. No more mysterious than that.

Let's use a toy function: f(x) = (x − 3)². It's shaped like a bowl, with its lowest point at x = 3. Away from the bottom, the function has a slope — steep far from x = 3, flat right at it. That slope, at any point, is the derivative.

Notice: to the left of x = 3 the slope is negative (the curve is falling as x increases), to the right it's positive (rising), and exactly at x = 3 the slope is zero — the bottom of the bowl. This is exactly why derivatives matter for learning: the bottom of a bowl-shaped loss function is where a model's predictions are best, and the slope tells you which way to move to get closer to it.

You don't need calculus rules to compute a derivative — you can approximate it directly from the definition: nudge x by a tiny amount h, see how much f(x) changed, and divide. Run this to watch the approximation converge to the true derivative as h shrinks:

Once you have the slope, you have a strategy for finding the bottom of the bowl: if the slope is positive, decrease x; if it's negative, increase x. Repeat that, taking small steps, and you'll walk downhill to the minimum. This is gradient descent — the exact loop that trains neural networks, just on a toy function instead of billions of parameters:

One honest gap: every training loop in this course uses that exact update rule — plain gradient descent. Almost no real model actually trains that way. On a lopsided loss surface (steep in one direction, shallow in another — extremely common in practice), plain gradient descent zig-zags along the steep direction while crawling along the shallow one. Two real refinements fix this, and you'll see their names constantly: momentum and Adam.

Momentum keeps a running average of past gradients, which smooths out the zig-zag the same way a ball rolling downhill carries momentum through small bumps. Adam goes further: it tracks a running average of squared gradients too, so it can automatically take bigger steps in directions where the gradient has stayed small and consistent, and smaller steps where it's been large or noisy. That adaptive, per-dimension step size is why Adam (or its close variant AdamW) is the default choice for training almost every modern neural network, this course's toy examples aside.

Real models don't have one input — they have billions of parameters, and the function being minimized (the loss) is a deep chain of operations rather than one line of algebra. Computing the slope with respect to every one of those parameters, efficiently, is what backpropagation does — that's next.