Preference data and reward models

Design comparisons around a question#

Start with the behavior the comparison should reveal. Give annotators the same prompt, evidence and permissions that the intended user would have. Compare plausible alternatives rather than an excellent answer with an obviously broken one. Randomize answer positions and hide model identities so those cues do not determine the label.

Use an explicit rubric covering factual support, instruction following and relevant constraints. Record serious errors separately from stylistic preferences. When both answers are unsuitable, preserve that fact; choosing the less harmful answer must not turn it into an approved demonstration. Sample across tasks and languages before collecting a large volume of easy comparisons.

Preserve disagreement#

Allow ties, uncertainty and requests for specialist review. Distinguish ambiguous instructions, missing evidence and legitimate differences in preference from accidental labeling errors. Keep individual votes and short reasons where appropriate. A majority vote can provide a training label, but should not erase disagreement from the audit record.

This fictional provider-neutral annotation record is illustrative, not an Align API schema:

{
  "pair_id": "p-014",
  "prompt_family": "release_notes",
  "rubric": "support-v2",
  "votes": {"a": 2, "b": 1, "tie": 1}
}

Review a shared calibration batch before scaling collection. Define how ties and disputed cases enter training; never silently convert them into confident binary preferences.

Split by the unit that can leak#

Split related prompt families before generating or expanding comparisons. Keep paraphrases, responses to the same question and comparisons derived from one ranked group together. Where examples depend on shared source documents or conversations, inspect those relationships before assigning partitions.

Freeze a held-out evaluation set that neither label refinement nor checkpoint selection uses. Deduplicate across partitions and record dataset lineage. For repeated experiments, separate a development set used for decisions from a final assessment set. A random row split can exaggerate generalization when nearly identical questions or answer templates appear on both sides.

Fit and interpret the pairwise objective#

Christiano and colleagues fit human comparisons using a logistic preference model over trajectory segments. For text comparisons, let r(x,y) be a scalar score for answer y to prompt x. Under the Bradley–Terry model:

P(preferred beats other | x)
  = sigmoid(r(x, preferred) - r(x, other))

L = mean(-log(sigmoid(
    r(x, preferred) - r(x, other)
)))

This provider-neutral formula illustrates binary pairwise training, not an Align API. Only score differences determine these modeled probabilities; an absolute score is not a universal quality scale. The loss fits observed preferences under its assumptions, not factual truth. Check sign conventions on a batch, then use stable log-sigmoid operations in implementation.

Look for reward shortcuts#

Gao and colleagues show, in a synthetic setup, that further optimization of a proxy reward can reduce performance measured by a separate reference reward. Their result motivates checking outputs independently rather than accepting rising training reward as success.

Singhal and colleagues investigate response-length correlations in RLHF and identify length sensitivity in reward models. Build matched review pairs that vary verbosity while preserving factual content. Include polished but unsupported answers and short correct answers. Inspect results by length difference, task and language; neither rewarding nor penalizing length automatically establishes quality.

Validate the model and its intended use#

Evaluate held-out pairwise accuracy and log loss, then inspect confidently incorrect rankings. Review disagreement-heavy cases separately. If probabilities will guide a decision, assess calibration on representative data rather than assuming the logistic form guarantees it.

Before using the reward for policy optimization, test whether independently reviewed answer quality improves as optimization increases. Track critical errors and useful-answer regressions alongside preference wins. Stop or revise an experiment when reward rises while external review worsens. A reward model remains a fallible predictor of the collected judgments; it cannot resolve omitted requirements, unseen settings or conflicts among stakeholders.