Embeddings: Turning Tokens Into Meaning

Once text is split into token IDs (plain integers), the model needs to turn each ID into something that carries meaning. It does this with an embedding: a list of numbers (a vector) assigned to each token.

You can picture each token as a point in space. Tokens with similar meaning end up near each other; unrelated tokens end up far apart. "Cat" and "dog" land close together. "Cat" and "bicycle" land far apart.

Crucially, these positions aren't hand-designed — they're learned during training. The model starts with random vectors for every token and gradually nudges them around as it sees which words tend to appear in similar contexts ("the cat sat" vs. "the dog sat"). Words used in similar ways drift toward similar positions.

Real embeddings have hundreds or thousands of dimensions, far more than we can visualize. The explorer below compresses them down to 2D so you can see the shape of the idea, even though real embedding space is much richer.

This is why embeddings are so powerful: relationships become geometry. Famously, in well-trained embedding spaces, king - man + woman ≈ queen — the direction between words captures a relationship, not just a category.

Now that each token is a meaningful vector, the model needs a way to let tokens influence each other based on context. That's the job of self-attention, next.

The explorer above used hand-placed 2D positions so the idea of clustering stayed visible. Here's the real thing: genuine 128-dimensional embeddings from an actual trained model, compressed down to 3D with real PCA (principal component analysis) instead of hand-placed — drag to rotate, and look up any word.