Apply direct preference optimization carefully
Understand exactly what a chosen–rejected pair optimizes.
- Compare policy and reference log probabilities.
- Inspect the effect of beta on a toy pair.
- Evaluate behavior beyond the preference loss.
Decide what preference training should change
Standard DPO learns chosen/rejected preferences with a frozen reference, without separate reward-model fitting or PPO rollouts; see the original paper.
Start by naming the behavior you want to improve: evidence use, concise explanations or instruction following. Keep a saved supervised baseline. Do not use preference labels to disguise a missing task capability. For example, if both candidate answers consistently misread a table, improve demonstrations or task coverage before assuming a ranking loss will supply the missing skill.
Read the chosen/rejected objective carefully
For prompt x, y+ is chosen and y− rejected:
Use completion-token sequence log probabilities conditioned on x. Minimize mean pair loss.
Inspect pairs before training
Hugging Face’s DPO documentation describes explicit prompt, chosen and rejected fields and conversational formatting. Its reference scores may be precomputed when the reference and preprocessing remain fixed. Those saved scores must correspond to the actual tokenized sequences used in training.
As a practical audit, render several pairs after templating and truncation. Confirm that both completions answer the identical prompt and that the preferred answer retains the text that justified its label. Remove accidental duplicates and contradictory labels. Separate evaluation by prompt family. Examine difficult near-ties rather than filling the dataset with obviously broken rejected answers that teach little about your target behavior.
Separate beta, temperature and measured KL
Beta belongs to the preference objective, not the decoding sampler. In the underlying KL-regularized formulation, a larger beta represents stronger reference regularization; it also changes the scale and saturation of the sigmoid loss. It does not impose a measured KL target on every minibatch. Generation temperature controls sampling and is a separate setting.
For a concrete experiment, keep the reference, data order and evaluation sampler fixed while comparing a small beta sweep. Record held-out quality, actual policy/reference divergence and response length. Avoid changing beta and evaluation temperature together: that makes the source of an improvement ambiguous. Do not interpret a larger training preference margin as a calibrated increase in user satisfaction.
Choose a method with distribution limits in view
Offline pairs constrain behavior only where comparisons provide useful coverage. Xu and colleagues’ comparison study analyzes how DPO can favor responses outside that coverage and examines data/model mismatch. Its results motivate checking distribution shift; they do not establish a universal winner for every task.
Use DPO as a practical baseline when you already have relevant, trustworthy pairs and want a simpler optimization loop. Consider reward-model-based RLHF when you can score freshly generated responses and maintain a rollout pipeline. Neither choice eliminates feedback bias or reward shortcuts. When the policy changes substantially, review fresh outputs before deciding whether another data-collection round is justified.
Compare outcomes, not just training losses
Run an SFT baseline, a DPO candidate and, if feasible, a PPO candidate from the same starting checkpoint. Reuse the preference rubric and evaluation prompts. Report both a matched-compute comparison and each method’s best result under a disclosed budget, because equal update counts do not imply equal work.
Record annotation effort, preprocessing, training, generation and evaluation costs. Blind reviewers to model identity, randomize answer order and retain ties. Report task success, factuality, refusal errors and length alongside preference judgments. Test unfamiliar prompts separately. Select checkpoints using validation, then assess the untouched test set once.
Make the work reviewable.
- Chosen and rejected responses to the same prompt.
- A frozen reference with compatible tokenization.
- Behavioral evaluation separate from training loss.