Fine-Tuning Intuition (LoRA)

The tiny GPT from the capstone is a pretrained model: it already has weights (however small and untrained here). In practice, you almost never train a large model from scratch for a new task — you take one that already exists and fine-tune it.

The naive approach — full fine-tuning — just keeps training every single weight in the model on your new task's data. It works, but it's expensive: for a model with billions of parameters, you're updating billions of numbers, and you need a full copy of the entire model saved per task you fine-tune for.

LoRA (Low-Rank Adaptation) takes a different approach: freeze the pretrained weight matrix W completely, and instead learn a small update on top of it. That update isn't a full matrix of its own weights — it's the product of two much smaller matrices, A and B, so ΔW = A·B. The final weight used at inference is just W' = W + ΔW.

The trick is that A and B are skinny: if W is n×n, then A is n×r and B is r×n for some small rank r — often 4, 8, or 16, versus n in the thousands. Multiplying two skinny matrices together can only ever produce a low-rank update, which turns out to be enough to adapt a model's behavior meaningfully.

W (gray/blue heatmap) never changes — only A and B do. As you increase the rank, A and B grow wider, the patch A·B can represent more complex updates, and the parameter-count bar for LoRA creeps closer to full fine-tuning. At low rank, you're training a small fraction of the parameters a full fine-tune would touch.

This matters for more than just training speed:

• Storage: instead of saving a full copy of the model per task, you save one shared frozen base model plus a tiny A/B pair per task.
• Swappable adapters: switching tasks can mean swapping out a few megabytes of A/B weights instead of loading a whole new multi-gigabyte model.
• Cheaper experimentation: fewer trainable parameters means fewer gradients to compute and store, so fine-tuning runs faster and fits in less memory.

The whole idea in plain Python, matching the widget's rank-2 example:

Those are exactly the kind of matrices you've been computing with all along — the W_Q, W_K, W_V from the self-attention lessons are precisely what a real fine-tuning setup would freeze and patch with a LoRA adapter rather than retraining from scratch. LoRA is how to adapt a model's weights cheaply — next is what those adapted weights actually get trained on to turn a raw pretrained model into something that behaves like an assistant.