Lesson 1 of 6

A Single Neuron

Back in Zero to GPT, you built a tiny autograd engine — a Value class that tracks how numbers combine and can compute gradients automatically. This course picks up exactly there and builds something real out of it: a neural network, one piece at a time.

The smallest piece is a single neuron. Strip away the biological metaphor and it's just two steps: take a weighted sum of the inputs, then squash the result through a nonlinear function.

Given inputs x1, x2 and the neuron's own learned numbers — weights w1, w2 and a bias b — the weighted sum is:

z = w1·x1 + w2·x2 + b

That's identical to the dot-product-plus-bias math from the self-attention lessons, just with far fewer dimensions. On its own, this is only a linear function — stack many of them and you'd still only ever be able to represent straight lines and planes.

The fix is a nonlinearity. This course uses tanh, which squashes any real number into the range (−1, 1): large positive z saturates toward 1, large negative z saturates toward −1, and z near 0 passes through almost unchanged. That nonlinear bend is what eventually lets a network represent curves and boundaries far more interesting than a straight line.

A neuron: weighted sum, then squash through tanh

x1 (input): 0.60x2 (input): -0.40w1 (weight): 0.80w2 (weight): -0.60b (bias): 0.10

z = w1·x1 + w2·x2 + b = 0.82
tanh(z) = 0.68

Drag the weights and you're directly reshaping what this neuron responds to; drag the inputs and you're moving the dot along the tanh curve. Push z far enough in either direction and notice the output stops changing much — that's saturation, and it's a big part of why training deep networks needs the careful gradient bookkeeping from the backpropagation lesson.

The exact same neuron, built on the Value autograd engine from Zero to GPT — extended with one new operation, .tanh():

Python

import math

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,), "tanh")
        def _backward():
            self.grad += (1 - t ** 2) * out.grad
        out._backward = _backward
        return out

def backward(self):
        topo, visited = [], set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

x1, x2 = Value(0.6), Value(-0.4)
w1, w2 = Value(0.8), Value(-0.6)
b = Value(0.1)

z = x1 * w1 + x2 * w2 + b
out = z.tanh()

print("z =", round(z.data, 4))
print("out = tanh(z) =", round(out.data, 4))

out.backward()
print("dL/dw1 =", round(w1.grad, 4), " dL/dw2 =", round(w2.grad, 4), " dL/db =", round(b.grad, 4))

Your turn

Implement neuron_forward(x1, x2, w1, w2, b): compute the weighted sum z = w1*x1 + w2*x2 + b, then return tanh(z).

Python

One neuron can only learn one weighted-and-squashed view of its inputs. Next: put several of them side by side.

Go deeper

Neural Networks: Zero to HeroAndrej Karpathy

Starts with micrograd — building the exact same Value-based autograd engine this course extended, from a single neuron up to a full network.

Why does a neuron need a nonlinear activation function like tanh?

← Syllabus A Layer of Neurons →