Training: From Logits to Loss
A stack of transformer blocks turns a token into a rich, context-aware vector, but that's still not a prediction. One more layer turns it into logits: one raw score per word in the vocabulary. Higher score means the model thinks that word is more likely to come next.
Right after initialization, those scores are close to random. Training nudges them toward reality using cross-entropy loss: given the correct next word, penalize the model for putting low probability on it. The gradient of this loss with respect to the logits has a beautifully clean form, softmax(logits) − one_hot(correct_word), the exact same gradient-descent loop from the derivatives lesson, just with a vector-valued output instead of a single number.
Predict before you look
Before you scrub or hit play: the correct next word is "mat." Roughly what do you expect P('mat') to look like at step 0 versus step 30, and will the loss curve fall smoothly, or in fits and starts?
Scrub through training and watch two things happen together: the loss curve falls, and the predicted probability on the correct word ("mat", highlighted) climbs toward 1 while every other candidate gets squeezed toward 0. That's all training is: millions of tiny nudges just like this one, repeated across every position in every sentence in the training set.
That's the loss signal driving every gradient step during training. Once training is done, generating text needs a different idea: how do you turn a probability distribution into a single chosen word? That's next.