Capstone: From GPT-2 to Modern LLMs

Recap, in one line each: text becomes tokens, tokens become embeddings, position gets injected, self-attention lets tokens gather context from each other, a feed-forward layer lets each token process what it gathered, residuals and norms keep it all trainable, gradient descent on cross-entropy loss teaches the whole stack, and temperature/top-k sampling turns its output into actual generated text.

That's the whole recipe. Every large language model you've heard of runs on some version of it.

Here's the payoff: every piece above, composed into one small program that actually runs — tokenize (already done, our "tokens" are just characters), embed, attend, feed-forward, score, sample, repeat. Untrained weights mean the output is gibberish, but the mechanism is completely real — this is architecturally a GPT, just tiny and never trained:

Zoom out from this toy version to real systems, and almost nothing changes structurally — only the numbers get enormous. More layers stacked, wider vectors, many attention "heads" running in parallel instead of one, vastly more training text, vastly more compute.

What separates a raw next-token predictor from something like a modern assistant is mostly what happens after this core training — instruction tuning and RLHF, covered in the previous two lessons, nudge the model to follow instructions and prefer helpful, safe responses. On top of that: tool use lets it call external systems (search, code execution, calculators), and architectural tricks extend usable context far beyond what naive attention could handle efficiently. All of that builds on top of — not instead of — the mechanism you just implemented.

Same core idea from lesson one — predict the next token — scaled up with everything in between: real gradients, real attention, real training. That's the whole arc of this course, and it's the same arc every one of these models was built on.