Fine-Tuning Intuition (LoRA)

The tiny GPT from the capstone is a pretrained model: it already has weights (however small and untrained here). In practice, you almost never train a large model from scratch for a new task, you take one that already exists and fine-tune it.

The naive approach, full fine-tuning, just keeps training every single weight in the model on your new task's data. It works, but it's expensive: for a model with billions of parameters, you're updating billions of numbers, and you need a full copy of the entire model saved per task you fine-tune for.

LoRA (Low-Rank Adaptation) takes a different approach: freeze the pretrained weight matrix W completely, and instead learn a small update on top of it. That update isn't a full matrix of its own weights, it's the product of two much smaller matrices, A and B, so ΔW = A·B. The final weight used at inference is just W' = W + ΔW.

The trick is that A and B are skinny: if W is n×n, then A is n×r and B is r×n for some small rank r, often 4, 8, or 16, versus n in the thousands. Multiplying two skinny matrices together can only ever produce a low-rank update, which turns out to be enough to adapt a model's behavior meaningfully.

W (gray/blue heatmap) never changes, only A and B do. As you increase the rank, A and B grow wider, the patch A·B can represent more complex updates, and the parameter-count bar for LoRA creeps closer to full fine-tuning. At low rank, you're training a small fraction of the parameters a full fine-tune would touch.

How does training actually work with a frozen matrix sitting right there in the forward pass? The forward pass uses the full W' = W + ΔW, so W still shapes every prediction. But when .backward() runs, gradients only get stored for tensors marked trainable: W is marked requires_grad=False, so no gradient is ever computed for it, while A and B are trainable and do receive ∂L/∂A and ∂L/∂B from the exact same backprop machinery built earlier in this course. Nothing about the chain rule changes, only which parameters are allowed to listen to it.

Real LoRA also scales the patch: ΔW = (α/r)·A·B, where α is a fixed hyperparameter you pick once. Try the alpha slider on the widget above. Without that (α/r) factor, doubling the rank would roughly double the patch's magnitude too, forcing you to re-tune the learning rate every time you change r. Dividing by r cancels that out, so a learning rate that works well at one rank keeps working reasonably well at another.

Why should a low-rank patch be enough to meaningfully change a model's behavior? Intuitively, adapting a pretrained model to a new task usually doesn't require rewriting everything it knows, it needs a comparatively small, focused nudge: emphasize some directions in its existing representations a bit more, suppress others a bit. Researchers call the space of updates actually needed for a given task its intrinsic dimension, and empirically it's often far smaller than the full parameter count. A low-rank matrix is exactly a structured way to represent "a small number of meaningful directions" instead of "every possible direction independently," which is why rank 4-16 is frequently enough even on matrices with thousands of rows and columns.

One more real-world decision LoRA requires: which weight matrices get an adapter. A transformer block has several: W_Q, W_K, W_V, and an output projection W_O inside attention, plus the feed-forward layer's weights. Adapting all of them costs more trainable parameters (and more memory for gradients); adapting fewer costs less but may adapt less of the model's behavior.

Notice how little of the attention block's parameters get touched even when all four projections are adapted, that's the LoRA trade in miniature: rank r=8 on a 4096×4096 matrix is under half a percent of that matrix's full parameter count. In practice, Q+V is a common minimal choice; modern setups like QLoRA often go further and adapt every linear layer in the block, since the extra trainable parameters are still tiny next to the frozen base model.

This matters for more than just training speed:

• Storage: instead of saving a full copy of the model per task, you save one shared frozen base model plus a tiny A/B pair per task.
• Swappable adapters: switching tasks can mean swapping out a few megabytes of A/B weights instead of loading a whole new multi-gigabyte model.
• Cheaper experimentation: fewer trainable parameters means fewer gradients to compute and store, so fine-tuning runs faster and fits in less memory.

The whole idea in plain Python, including the alpha scaling, matching the widget's rank-2 example:

Those are exactly the kind of matrices you've been computing with all along: the W_Q, W_K, W_V from the self-attention lessons are precisely what a real fine-tuning setup would freeze and patch with a LoRA adapter rather than retraining from scratch. LoRA is how to adapt a model's weights cheaply; next is what those adapted weights actually get trained on to turn a raw pretrained model into something that behaves like an assistant.