Positional Encoding: Absolute and Rotary

You now have every ingredient of self-attention itself: embeddings, Q/K/V, the softmax-weighted blend. Before assembling a full transformer block, there's one gap worth closing first, self-attention has no built-in sense of word order at all.

Self-attention computes a weighted sum of value vectors, and weighted sums don't care about order. "The dog bit the man" and "the man bit the dog" would produce identical attention math if word order weren't encoded somewhere. So before anything else happens, a positional encoding, a unique pattern of sine and cosine values per position, gets added directly onto each token's embedding.

Different frequencies per dimension mean nearby positions get similar-but-distinct patterns, while far-apart positions look very different, giving the model a consistent, learnable sense of relative distance, not just an arbitrary index.

That sinusoidal encoding is the classic version, and it's genuinely how the original transformer worked, but it's mostly not what modern LLMs use anymore. Llama, Mistral, Qwen, and most other current models use RoPE (Rotary Position Embedding) instead. The idea is different in an interesting way: instead of adding a position pattern onto the embedding, RoPE rotates the query and key vectors by an angle proportional to their position, before the attention dot product happens.

That invariance is the entire point. A dot product between two vectors depends on the angle between them (rotating both by the same amount doesn't change the angle between them, it just spins the whole picture). So the attention score between a query at position m and a key at position n ends up depending only on their relative offset n − m, never on m and n themselves. Sinusoidal encoding gives the model a way to infer relative position from two absolute patterns; RoPE builds relative position directly into the math, which tends to generalize better to sequence lengths longer than anything seen in training.

Either way, position is now injected. Two more pieces turn this into a trainable block: residual connections and a feed-forward layer, next.