Self-Attention: Letting Words Talk to Each Other

An embedding gives a token a general meaning, but words shift meaning depending on context. Take the word "bank" in:

"I sat by the river bank." "I deposited money at the bank."

Same token, same starting embedding — completely different meaning. The model needs a mechanism to let context reshape each token's representation. That mechanism is self-attention.

For every token, self-attention asks: "Which other tokens in this sentence should I pay attention to, and how much?"

It computes an attention weight between the current token and every other token in the sequence. High weight means "this token strongly influences my meaning here." The token then updates its representation by blending in information from the tokens it's attending to, weighted by those scores.

This happens for every token, all at once, which is what lets the model capture relationships regardless of how far apart words are in a sentence.

Click different tokens above and notice how the attention pattern changes — a verb might attend strongly to its subject, while "bank" attends strongly to whichever nearby word disambiguates it ("river" or "money").

Stack many of these attention layers together, interleaved with small neural networks, and you get a transformer — the architecture behind virtually every modern LLM. That's where we're headed next.

Everything above used small, hand-placed 2D embeddings so the math stayed visible. Here's the same computation — literally the same formula, Q·K/√d → softmax → weighted sum of V — run with real pretrained weights from an actual trained model instead of invented numbers. Type any sentence and watch real self-attention happen.

That widget isn't calling any API — it fetches the exact same real weight file the page loaded and runs the exact same forward pass, right here, in real Python:

That walkthrough showed one query at a time. But a model computes this for every token simultaneously — producing a full attention pattern. Here's the whole thing at once: every key against every query, the way 3Blue1Brown visualizes it in his attention series.

Each column is a query, each row is a key, and each dot's size is the attention weight between them — exactly the same softmax output as the widget above, just laid out for all ten tokens at once instead of one. Click a column to pick a different query and watch the right panel update: the top few V vectors it draws from, blended by those weights, into the green output vector.

Now let's implement the entire thing in plain Python — no PyTorch, no NumPy, just nested loops — using the exact same numbers as the visualizations above, so the printed output matches what you just saw:

That's the entire mechanism: project to Q/K/V with a matrix multiply, score every query against every key, softmax each row, then multiply by V to blend. Everything else in a real transformer — multiple attention "heads" running in parallel, higher-dimensional vectors, many stacked layers — is this exact same computation, repeated and scaled up.

One more wrinkle. Everything so far let every token see every other token — including tokens later in the sentence. That's fine for reading a finished sentence, but a model that generates text one word at a time can't be allowed to peek at words it hasn't produced yet — that would be like grading an exam by reading the answer key.

The fix is a causal mask: before the softmax, block every score where the key comes after the query, so a token can only attend to itself and what came before it.

Toggle the mask off and on and watch the upper-right triangle of dots appear and disappear — that's every "attend to a future token" connection being allowed or forbidden. With the mask on, each row's remaining weights still sum to 1; they just redistribute across a smaller set of allowed keys.