Lesson 14 of 15

Aligning With Human Feedback: RLHF & DPO

Instruction tuning teaches a model to respond instead of continue, using examples with one clear correct response per prompt. But most interesting questions don't have a single correct response — they have several plausible ones that differ in tone, thoroughness, or helpfulness. Capturing which one people actually prefer needs a different training signal: human preference data.

Be the human rater — this is literally how reward-model training data gets collected

Prompt: "How do I get better at chess?"

Comparison 1 of 3 — which response is better?

Real RLHF reward models are trained on exactly this kind of data — millions of pairwise human comparisons like the ones you just made — until the model can predict which response a human would prefer without needing a human in the loop for every judgment. That predicted preference is then used as a reward signal to further train the assistant toward the kind of response that tends to win.

That's genuinely how the raw data behind RLHF gets collected — at a much larger scale, with many more raters and many more response pairs, but the same basic action: given two candidate responses, pick the better one. Do that millions of times across thousands of prompts, and you have a dataset that captures human preference far more richly than any single "correct answer" ever could.

RLHF (Reinforcement Learning from Human Feedback) uses that preference data in two stages:

Train a reward model — a separate model that takes a prompt and a response and predicts a single score: roughly, "how much would a human like this?" It's trained directly on the pairwise comparisons, learning to score the preferred response higher than the rejected one.
Optimize the assistant against that reward model using reinforcement learning (typically an algorithm called PPO) — generate a response, score it with the reward model, and nudge the assistant's weights to make higher-scoring responses more likely. A penalty term keeps the assistant from drifting too far from its instruction-tuned starting point, so it doesn't degenerate into just telling the reward model whatever maximizes its score regardless of actual quality (a failure mode called reward hacking).

DPO (Direct Preference Optimization) is a newer, simpler alternative that skips both the separate reward model and the reinforcement-learning loop entirely. It uses a loss function, derived mathematically to have the same optimum RLHF is aiming for, that trains the policy directly on preference pairs — no reward model, no PPO, just a modified version of the training loop you've already seen throughout this course. That simplicity is why many recent open-weight models use DPO instead of full RLHF.

The real DPO loss formula, computed on toy numbers:

Python

import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def dpo_loss(logp_theta_chosen, logp_ref_chosen, logp_theta_rejected, logp_ref_rejected, beta=0.1):
    # how much more (in log-probability) the current policy favors the
    # chosen/rejected response compared to the frozen reference model
    chosen_shift = logp_theta_chosen - logp_ref_chosen
    rejected_shift = logp_theta_rejected - logp_ref_rejected
    return -math.log(sigmoid(beta * (chosen_shift - rejected_shift)))

# case 1: the policy hasn't shifted away from the reference model at all yet
loss_before = dpo_loss(
    logp_theta_chosen=-2.0, logp_ref_chosen=-2.0,
    logp_theta_rejected=-2.0, logp_ref_rejected=-2.0,
)

# case 2: training has shifted the policy toward the chosen response and
# away from the rejected one, relative to the reference model
loss_after = dpo_loss(
    logp_theta_chosen=-1.0, logp_ref_chosen=-2.0,
    logp_theta_rejected=-3.0, logp_ref_rejected=-2.0,
)

print(f"loss when policy hasn't shifted yet: {loss_before:.3f}")
print(f"loss after policy favors the chosen response: {loss_after:.3f}")

Notice the loss drops as the policy shifts probability toward the chosen response and away from the rejected one, relative to the reference model — not in absolute terms. That relative comparison is exactly what keeps DPO tethered to a reasonable starting point instead of letting the model drift arbitrarily, the same job RLHF's KL penalty does, just built directly into the loss function instead of added as a separate term.

That's the real arc from raw pretraining to something like a modern assistant: predict the next token at massive scale, teach it to respond instead of continue with instruction tuning, then shape which responses it prefers with human feedback. Every piece of that — the training loop, the loss function, the gradient updates — is the exact same machinery built from scratch earlier in this course, just pointed at different data.

What does DPO eliminate compared to classic RLHF?

← From Completion to Chat: Instruction Tuning Capstone: From GPT-2 to Modern LLMs →