A Real Transformer, Live

Every earlier lesson built these pieces with small, hand-placed toy vectors so the math stayed visible. This lesson swaps in the real thing: the same tiny BERT model already powering the attention, embedding, and masked-word widgets earlier in this course (2 real layers, 128 real dimensions, ~4.4M genuine trained parameters). Type anything below and watch it flow through tokenization, embedding, position, attention, residuals, normalization, and feed-forward, each stage computed live from real pretrained weights, not illustration.

One more real thing to try. Everything above looked inside the model at a single pass. This last piece asks it to actually write, using nothing but its real weights, real predictions, real softmax, pushed into a loop it was never trained for.

If any stage above felt like a black box, that's exactly what the earlier lessons are for: Positional Encoding, Residuals, Normalization & the Feed-Forward Layer, and Multi-Head Attention all derive this exact same math by hand, on numbers small enough to track by eye, before scaling up to what you just watched run for real.