From Counting to Neural Networks

We just built up embeddings and self-attention — real machinery. Let's step all the way back down to the simplest possible language model, so the next lessons about assembling and training a transformer have something humble to compare against.

The question: given only the single previous character, what character comes next?

No neural network needed to start — just count. Take a small corpus of words, and for every pair of adjacent characters, tally how often it occurs. c is followed by a a lot (cat, cab, can); a is followed by t, n, b, m in various amounts. Normalize each character's tallies into a probability distribution over 'what comes next,' and you have a working (if very limited) language model.

Click through a few rows — notice some characters are highly predictable (q would almost always be followed by u in real English), while others are wide open. Hit Generate and watch it walk the grid: start at the • (start-of-word marker), sample a next character from that row's distribution, move to that row, repeat until it samples • again (end of word).

Try replacing the training text at the top with your own — a name, a sentence, a paragraph pasted from somewhere. The grid retrains instantly on whatever you type, so you can watch this exact counting process run on text you actually care about instead of the toy corpus above.

Here's the important part: that grid of probabilities is a language model, in the exact same sense a transformer is. Both take a context and output a probability distribution over the next token.

In fact, this specific grid is mathematically identical to a one-layer neural network: one-hot encode the current character, multiply by a weight matrix, apply softmax — and if you trained that tiny network with gradient descent on this corpus, it would converge to the exact same probabilities you just counted directly. Counting and gradient descent are two different roads to the same answer here.

So why not just use bigger and bigger count tables? The real limitation isn't counting versus neural networks — it's context window of one. This model has no memory of anything before the single previous character. It can't tell that "the cat that chased the mouse across the yard sat..." should probably be followed by something related to a cat sitting, because by the time it's predicting after "sat", everything before "t" is already forgotten.

That's exactly the gap self-attention closes: instead of only ever looking at one previous token, it lets a prediction draw on every token in the context, weighted by relevance.

Here's the whole counting model in plain Python — build the table, read off a row, generate a few words:

Next, we'll assemble the actual building blocks — self-attention, feed-forward layers, and the residual connections that hold them together — into one real transformer block.