Gradient Checking
Every backward pass you've written so far was hand-derived: you worked out the chain rule for each operation and typed the resulting formula into code. That's exactly the kind of step where a sign flip, a missing factor, or a swapped index slips in silently, the code runs, the loss even goes down most of the time, and the bug hides for a long time. Gradient checking catches this by comparing your analytic gradient against a completely different way of estimating the same number.
The idea comes straight from the definition of a derivative: nudge a weight by a tiny amount ε in each direction, and see how much the loss moved.
numerical_gradient ≈ (loss(w + ε) − loss(w − ε)) / (2ε)
This is the slope of the secant line through two nearby points on the loss curve, and as ε shrinks toward 0, that secant slope converges to the true tangent slope, exactly what backprop's chain rule computes directly. Two totally different computations ("perturb and measure" vs. "chain rule by hand") should agree, if they don't, the analytic derivation has a bug.
Checking one weight's analytic gradient against a numerical estimate
Loss as hidden neuron 1's first weight varies, everything else held fixed. Amber dots are the two finite-difference evaluations; the amber line is the secant through them, the numerical gradient is its slope.
Drag epsilon small and the secant line hugs the true tangent, error stays under 1%. Drag it large and the secant starts cutting across the curve's bend instead of following it, the estimate drifts away from the true slope. This is exactly how you'd sanity-check a hand-derived backward pass in real code: if the two numbers disagree by more than about 1%, there's a bug in the analytic gradient, not in the check.
Drag epsilon small and the amber secant line sits almost exactly on top of the true curve, the numerical and analytic gradients agree to several decimal places. Drag epsilon large and the secant starts cutting across the curve's bend instead of tracing it, the two numbers visibly diverge. In practice you'd pick an epsilon small enough that the error stays under roughly 1%, that's your sign the hand-derived gradient is trustworthy.
A gradient check on a tiny function, f(x) = x³, whose derivative you can verify by hand (f'(x) = 3x²):
Your turn
Implement numerical_gradient(f, x, eps): the centered finite-difference estimate (f(x+eps) - f(x-eps)) / (2*eps).
You now have a way to verify any backward pass you write, in this course or anywhere else. Next, a different lens on the same chained-derivative machinery: what happens when you chain many layers deep, and every one of them is a little too small, or a little too large?
Why compare against the centered difference (f(x+ε) − f(x−ε)) / (2ε) rather than the simpler (f(x+ε) − f(x)) / ε?