Assembling the Transformer

You now have every individual ingredient: embeddings, self-attention, and (from the bigram lesson) the idea of a network layer that transforms a vector. Time to assemble them into an actual transformer block — the repeating unit stacked to build every modern LLM.

One thing missing so far: self-attention computes a weighted sum of value vectors, and weighted sums don't care about order. "The dog bit the man" and "the man bit the dog" would produce identical attention math if word order weren't encoded somewhere. So before anything else happens, a positional encoding — a unique pattern of sine and cosine values per position — gets added directly onto each token's embedding.

Different frequencies per dimension mean nearby positions get similar-but-distinct patterns, while far-apart positions look very different — giving the model a consistent, learnable sense of relative distance, not just an arbitrary index.

Next: residual connections. Recall from the backpropagation lesson that gradients are products of local derivatives chained together — stack enough layers and that product can shrink toward zero (or blow up), making deep networks hard to train. The fix is almost embarrassingly simple: instead of replacing a vector with f(x), compute x + f(x). The +x gives gradients a direct shortcut path backward through every layer, no matter how deep the stack gets.

Two more pieces round out a block:

• Layer norm rescales a vector to have zero mean and unit variance before the next sublayer, keeping activations in a well-behaved range as they pass through many stacked blocks.
• A feed-forward network — a small two-layer network (expand, ReLU, contract) applied independently to each token's vector. Where attention mixes information across tokens, the feed-forward layer lets each token "think" about what it just gathered. This sublayer alone holds a large fraction of a transformer's total parameters.

That's a complete transformer block: embed → add position → attend → residual → norm → feed-forward → residual → norm. A real model stacks dozens of these, each with its own learned weights, and every stack refines the same token vectors a little further before a final layer turns them into next-token predictions.

Here's the entire block in plain Python, reusing the exact same sentence and attention math from the self-attention lesson, continuing with the token "it":

One more refinement before moving on. The attention above was single-head: one set of Q/K/V projections, capturing one kind of relationship. But a sentence has many kinds of relationships at once — which word is the subject, which pronoun refers to which noun, which word disambiguates another. A single head has to compromise across all of them.

The fix: run several attention computations in parallel — multiple heads — each with its own learned Q/K/V projections, so each head is free to specialize. Their outputs get concatenated and mixed back together with one more projection.

Head A and Head B here use different (fixed, toy) projections, and you can see them genuinely disagree about what a token should attend to — that's the entire point. A real model might have 8, 12, or dozens of heads, each nudged by training toward noticing something different, and it's the combination of all of them, mixed by the output projection, that gives a transformer block its expressive power.

The block output is still just a vector per token — a richer, context-aware one, but not yet a prediction. Turning it into an actual next-token probability, and training the whole stack to get better at that, is next.