Course
Zero to GPT
Build up an intuitive, from-scratch understanding of how large language models work, from derivatives and backpropagation through tokenization, embeddings, self-attention, and all the way to a working (if tiny) GPT.
Module: Tokenization
Lesson 4
Character-Level & Word-Level Tokenization
The two simplest ways to chop text into tokens, and why both break down in practice.
Lesson 5
Byte-Pair Encoding: Building It From Scratch
The middle ground between character-level and word-level tokenization that virtually every modern LLM actually uses.
Lesson 6
WordPiece, SentencePiece & Choosing a Real Tokenizer
Raw BPE isn't quite what production tokenizers ship with. The real variants, and a real one running on real vocabulary.
Module: Assembling the Transformer
Lesson 10
Positional Encoding: Absolute and Rotary
How a transformer knows word order, sinusoidal encoding, and the RoPE variant modern LLMs actually use.
Lesson 11
Residuals, Normalization & the Feed-Forward Layer
The remaining pieces that make a transformer block trainable, then assemble one from scratch.
Lesson 12
Multi-Head Attention
Why one attention head isn't enough, and how running several in parallel gives a transformer block its expressive power.
Module: Training and Sampling
Lesson 13
Training: From Logits to Loss
How raw prediction scores get turned into a trainable loss signal.
Lesson 14
Sampling Strategies: Temperature, Top-k & Top-p
How a trained model actually turns next-word probabilities into generated text.
Lesson 15
KV Caching: Making Generation Fast
The practical trick that keeps token-by-token generation from getting slower as the conversation grows.