Lesson 17 of 20

From Completion to Chat: Instruction Tuning

Every lesson so far has trained a model on one objective: predict the next token, given raw text. Do that at massive scale on a huge slice of the internet, and you get something genuinely powerful, but not something that reliably answers questions or follows instructions. It just continues text in whatever pattern it's already in. That gap is real, and it's worth seeing directly before talking about how it gets closed.

Predict before you look

Before you look: given the prompt "Explain photosynthesis," do you expect a purely next-token-predicting model to just... explain photosynthesis? Or might it do something else entirely?

A model trained purely to continue internet text has seen plenty of text that starts with something like "Explain photosynthesis" (homework assignments, worksheets, quiz banks) and statistically, that phrase is often followed by more instructions, not a direct answer. The model isn't being unhelpful on purpose; it genuinely doesn't know the prompt is supposed to be treated as a command rather than a pattern to extend.

Instruction tuning (often called supervised fine-tuning, or SFT) fixes this the same way every fine-tuning lesson in this course already works: keep training the model, but on a new, curated dataset, this time, thousands of hand-written or hand-selected (instruction, response) pairs, where the response is a genuinely helpful, direct answer to the instruction.

One real technical detail worth knowing: during this training, the loss is typically only computed on the response portion of each example, not the instruction. The model isn't being taught to predict instructions, it already knows how to produce text; it's being taught what a good response to an instruction looks like.

That "only the response counts" detail is worth seeing directly, because it explains something that would otherwise be surprising: the instruction tokens are still right there in the model's input, feeding into every prediction that follows them, but the model is never graded on reproducing them. Toggle the mask below on and off and watch which tokens actually generate a gradient.

Predict before you look

Before you toggle it: if masking were left off and the loss counted every token equally, what would the model start learning to do with prompts like "Explain photosynthesis" itself, not just the answers to them?

With masking on, the green tokens are the only ones whose prediction error moves any weights this step. The instruction tokens ("Explain photosynthesis .") still get read and processed on the way there, position by position, contributing context, they just never appear as a target the model is scored against. That's the whole mechanism: instruction tuning doesn't use a different loss function than pretraining, cross-entropy on next-token prediction, it uses the exact same loss with most of the sequence's targets switched off.

The masking logic itself, in plain Python, on a toy vocabulary and toy logits:

Python

import math

instruction_tokens = ["Explain", "photosynthesis", "."]
response_tokens = ["Plants", "convert", "sunlight", "into", "energy", "."]
sequence = instruction_tokens + response_tokens

# 1 = this position's prediction counts toward the loss, 0 = masked out
loss_mask = [0] * len(instruction_tokens) + [1] * len(response_tokens)

def softmax(logits):
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    total = sum(exps)
    return [e / total for e in exps]

# toy per-position logits over a 4-word vocabulary; correct_index is which
# word should have come next at that position
vocab = ["Plants", "convert", "sunlight", "energy"]
per_position_logits = [
    [2.0, 0.5, 0.3, 0.1],  # predicting after "Explain"
    [0.2, 0.3, 2.5, 0.1],  # predicting after "photosynthesis"
    [1.8, 0.4, 0.2, 0.3],  # predicting after "."
    [3.0, 0.2, 0.1, 0.1],  # predicting after "Plants" -> should be "convert"
    [0.3, 3.0, 0.2, 0.1],  # predicting after "convert" -> should be "sunlight"
]
correct_indices = [0, 0, 0, 1, 2]  # only meaningful where loss_mask == 1

total_loss = 0.0
counted = 0
for position, (logits, target) in enumerate(zip(per_position_logits, correct_indices)):
    probs = softmax(logits)
    token_loss = -math.log(probs[target])
    if loss_mask[position] == 1:
        total_loss += token_loss
        counted += 1
    tag = "counts" if loss_mask[position] == 1 else "masked"
    print(f"position {position:>2} ({tag:>6}): loss = {token_loss:.3f}")

print()
print(f"average loss over the {counted} counted positions: {total_loss / counted:.3f}")

import math

instruction_tokens = ["Explain", "photosynthesis", "."]
response_tokens = ["Plants", "convert", "sunlight", "into", "energy", "."]
sequence = instruction_tokens + response_tokens

# 1 = this position's prediction counts toward the loss, 0 = masked out
loss_mask = [0] * len(instruction_tokens) + [1] * len(response_tokens)

def softmax(logits):
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    total = sum(exps)
    return [e / total for e in exps]

# toy per-position logits over a 4-word vocabulary; correct_index is which
# word should have come next at that position
vocab = ["Plants", "convert", "sunlight", "energy"]
per_position_logits = [
    [2.0, 0.5, 0.3, 0.1],  # predicting after "Explain"
    [0.2, 0.3, 2.5, 0.1],  # predicting after "photosynthesis"
    [1.8, 0.4, 0.2, 0.3],  # predicting after "."
    [3.0, 0.2, 0.1, 0.1],  # predicting after "Plants" -> should be "convert"
    [0.3, 3.0, 0.2, 0.1],  # predicting after "convert" -> should be "sunlight"
]
correct_indices = [0, 0, 0, 1, 2]  # only meaningful where loss_mask == 1

total_loss = 0.0
counted = 0
for position, (logits, target) in enumerate(zip(per_position_logits, correct_indices)):
    probs = softmax(logits)
    token_loss = -math.log(probs[target])
    if loss_mask[position] == 1:
        total_loss += token_loss
        counted += 1
    tag = "counts" if loss_mask[position] == 1 else "masked"
    print(f"position {position:>2} ({tag:>6}): loss = {token_loss:.3f}")

print()
print(f"average loss over the {counted} counted positions: {total_loss / counted:.3f}")

Your turn

Implement build_loss_mask(num_instruction_tokens, num_response_tokens): return a list of 0s and 1s the length of the full sequence, where instruction positions are 0 (masked) and response positions are 1 (counted).

Python

Instruction tuning teaches the model the right shape of behavior (respond, don't continue) using examples with one clearly correct answer per prompt. But plenty of real questions don't have one obviously correct response; they have several plausible ones that differ in tone, helpfulness, or safety. Capturing which one people actually prefer needs a different kind of training signal entirely, that's next.

Why does a purely pretrained (not instruction-tuned) model often fail to directly answer a prompt like "Explain photosynthesis"?

During instruction tuning, why is the loss usually masked so it only counts the response tokens?

← Fine-Tuning Intuition (LoRA)Aligning With Human Feedback: RLHF & DPO →