Capstone: From GPT-2 to Modern LLMs

Recap, in one line each: text becomes tokens, tokens become embeddings, position gets injected, self-attention lets tokens gather context from each other, a feed-forward layer lets each token process what it gathered, residuals and norms keep it all trainable, gradient descent on cross-entropy loss teaches the whole stack, and temperature/top-k sampling turns its output into actual generated text.

That's the whole recipe. Every large language model you've heard of runs on some version of it.

Here's the payoff: every piece above, composed into one small program that actually runs. Tokenize (already done, our "tokens" are just characters), embed, attend, feed-forward, score, sample, repeat. Untrained weights mean the output is gibberish, but the mechanism is completely real, this is architecturally a GPT, just tiny and never trained:

That code isn't just printing text, it's actually running. Here's the same untrained network, live: pick a starting prompt and generate one token at a time, watching it flow through embed → attend → feed-forward → sample → get fed straight back in as the input for the next token.

Zoom out from this toy version to real systems, and most of what changes is just the numbers getting enormous: more layers stacked, wider vectors, many attention "heads" running in parallel instead of one, vastly more training text, vastly more compute.

One structural change is genuinely new, not just "the same thing, bigger": Mixture of Experts (MoE). Models like Mixtral and DeepSeek's larger releases replace the single feed-forward layer inside each block with several feed-forward layers ("experts") plus a small learned router. For every token, the router scores every expert and activates only the top few, the rest sit completely idle for that token, contributing zero compute.

Both, in different senses, and that's exactly why MoE is attractive. The model stores all 8 experts' worth of parameters, so its total capacity to have learned different things is closer to the 8-expert number. But it only runs 2 experts per token, so its compute cost per token is closer to the 2-expert number. That decoupling, huge parameter count without a proportionally huge compute cost per token, is the one real exception to "scaling up is structurally the same thing, just bigger": a dense model and a sparse MoE model with the same total parameter count are not doing the same amount of work per token.

What separates a raw next-token predictor from something like a modern assistant is mostly what happens after this core training: instruction tuning and RLHF, covered in the previous two lessons, nudge the model to follow instructions and prefer helpful, safe responses. On top of that: tool use lets it call external systems (search, code execution, calculators), and architectural tricks extend usable context far beyond what naive attention could handle efficiently. All of that builds on top of, not instead of, the mechanism you just implemented.

Same core idea from lesson one, predict the next token, scaled up with everything in between: real gradients, real attention, real training. That's the whole arc of this course, and it's the same arc every one of these models was built on.