Lesson 1 of 16

A Single Neuron

Back in Zero to GPT, you built a tiny autograd engine — a Value class that tracks how numbers combine and can compute gradients automatically. This course picks up exactly there and builds something real out of it: a neural network, one piece at a time.

The smallest piece is a single neuron. Strip away the biological metaphor and it's just two steps: take a weighted sum of the inputs, then squash the result through a nonlinear function.

Given inputs x1, x2 and the neuron's own learned numbers — weights w1, w2 and a bias b — the weighted sum is:

z = w1·x1 + w2·x2 + b

That's identical to the dot-product-plus-bias math from the self-attention lessons, just with far fewer dimensions. On its own, this is only a linear function — stack many of them and you'd still only ever be able to represent straight lines and planes.

The fix is a nonlinearity. This course uses tanh, which squashes any real number into the range (−1, 1): large positive z saturates toward 1, large negative z saturates toward −1, and z near 0 passes through almost unchanged. That nonlinear bend is what eventually lets a network represent curves and boundaries far more interesting than a straight line.

A neuron: weighted sum, then squash through tanh

x1 (input): 0.60x2 (input): -0.40w1 (weight): 0.80w2 (weight): -0.60b (bias): 0.10

z = w1·x1 + w2·x2 + b = 0.82
tanh(z) = 0.68

Drag the weights and you're directly reshaping what this neuron responds to; drag the inputs and you're moving the dot along the tanh curve. Push z far enough in either direction and notice the output stops changing much — that's saturation, and it's a big part of why training deep networks needs the careful gradient bookkeeping from the backpropagation lesson coming up.

The exact same neuron, built on the Value autograd engine from Zero to GPT — extended with one new operation, .tanh():

Python

import math

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,), "tanh")
        def _backward():
            self.grad += (1 - t ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo, visited = [], set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

x1, x2 = Value(0.6), Value(-0.4)
w1, w2 = Value(0.8), Value(-0.6)
b = Value(0.1)

z = x1 * w1 + x2 * w2 + b
out = z.tanh()

print("z =", round(z.data, 4))
print("out = tanh(z) =", round(out.data, 4))

out.backward()
print("dL/dw1 =", round(w1.grad, 4), " dL/dw2 =", round(w2.grad, 4), " dL/db =", round(b.grad, 4))

import math

class Value:
    def __init__(self, data, _children=(), _op=""):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,), "tanh")
        def _backward():
            self.grad += (1 - t ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo, visited = [], set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

x1, x2 = Value(0.6), Value(-0.4)
w1, w2 = Value(0.8), Value(-0.6)
b = Value(0.1)

z = x1 * w1 + x2 * w2 + b
out = z.tanh()

print("z =", round(z.data, 4))
print("out = tanh(z) =", round(out.data, 4))

out.backward()
print("dL/dw1 =", round(w1.grad, 4), " dL/dw2 =", round(w2.grad, 4), " dL/db =", round(b.grad, 4))

Your turn

Implement neuron_forward(x1, x2, w1, w2, b): compute the weighted sum z = w1*x1 + w2*x2 + b, then return tanh(z).

Python

One neuron can only learn one weighted-and-squashed view of its inputs. Next: put several of them side by side.

Go deeper

Neural Networks: Zero to HeroAndrej Karpathy

Starts with micrograd — building the exact same Value-based autograd engine this course extended, from a single neuron up to a full network.

Why does a neuron need a nonlinear activation function like tanh?

← Syllabus A Layer of Neurons →