Model alignment: from behavior specification to evidence

Specify behavior before choosing an algorithm#

Begin with a decision the system must make, rather than a broad label such as helpfulness. Specify the users, available evidence, permitted actions and competing instructions. Separate answer correctness from permission enforcement: a response can accurately repeat information the user was never allowed to receive. Define critical failures independently of average quality.

Wallace and colleagues study instruction hierarchy: instructions have different privileges, and lower-priority content should not override higher-priority constraints. For an application, turn that principle into explicit cases. A requested language change may be acceptable; a passage telling the assistant to reveal restricted records is document content, not authorization.

Write a behavior contract with three possible outcomes: answer with evidence, request clarification, or withhold an unsupported or unauthorized claim. Name who resolves conflicts in the rubric. Otherwise, two annotators can follow different policies while both appear internally consistent.

Define the data split and supervised baseline#

Supervised fine-tuning learns from desired responses. Stiennon and colleagues’ summarization study provides a concrete example of separating a supervised policy, a preference model and reinforcement learning. Preserve that separation when designing your experiment.

Use a split manifest keyed by document family, conversation or other unit that can leak. Put paraphrases and comparisons from the same family in one partition. Demonstrations belong to training; development data guides checkpoint selection; final assessment remains unopened until the selection rule is fixed. Similar rows on both sides are not evidence of transfer.

A rubric needs operational labels. For release documentation, mark invented availability as a critical error; score required conditions as present, incomplete or absent; and distinguish a necessary clarification from an unnecessary refusal. Keep evidence locators and individual judgments. Resolve factual mistakes, but retain legitimate disagreement about style.

For the supervised baseline, inspect exactly which tokens contribute to loss. In an assistant-response training design:

Record the prompt mask, padding mask and normalization rule. A decreasing loss is hard to interpret if a template change silently changes the target.

Connect preferences to a specific learning target#

Collect alternatives from a named policy revision under saved decoding settings. Ask reviewers which response better satisfies the rubric, not which sounds more confident. Record ties, missing context and specialist-review requests. For each chosen response, retain the rejected alternative and the reason it lost.

Choose an explicit reward model when the experiment needs a separately inspectable scoring component; choose a direct preference objective when that is the method being tested. Keep the comparison data fixed for the first comparison between methods. Otherwise, a gain may come from easier labels or different candidates rather than the optimization rule.

Before policy optimization, inspect disagreement-heavy pairs and counterexamples: a short correct response against an elaborate invention, or a justified refusal against an unnecessary one. A preference model that fails these cases is teaching the wrong distinction. More optimization makes that problem more consequential.

Make assurance independent of training success#

The alignment survey’s cycle distinguishes forward alignment, which trains toward requirements, from backward alignment, which gathers evidence and governs the system. Assurance operates throughout the lifecycle. This is the authors’ organizing framework, not a guarantee supplied by a training method.

Assign different questions to different reviews. Data review asks whether the labels express the intended policy. Training review asks whether the implementation optimizes the stated objective. Behavioral review asks whether the resulting system meets requirements on unseen cases. Deployment review checks the actual retrieval, permissions and tools surrounding that model.

Set release gates before reading candidate scores. Require critical-case review and record subgroup results by task, language and uncertainty type. Reviewers should see evidence and randomized outputs without model names. A model’s own favorable assessment is a diagnostic signal, not independent approval.

Define the training artifact contract#

A training artifact contract specifies what one stage must provide to the next. This provider-neutral illustrative record describes a fictional experiment, not an Align API:

{
  "run": "release-assistant-r2",
  "base": "base-checkpoint-A",
  "tokenizer": "tokenizer-A",
  "template": "conversation-v1",
  "corpus": "release-docs-v1",
  "split_manifest": "family-split-v1",
  "demonstrations": "demos-r2",
  "preferences": "pairs-r2",
  "rubric": "evidence-first-r2",
  "selection_rule": "critical-gates-then-quality",
  "evaluation": "sealed-family-test-v1"
}

Resolve every identifier to immutable files and hashes. Add the code revision, seed, optimizer settings, decoding configuration and masks to the run record. The reward stage must export its checkpoint, input format, score location and validation report. The policy stage must identify the frozen reference and reward revisions. Evaluation must retain prompts, evidence snapshots, outputs and adjudications.

Make handoffs testable: load one saved example through each stage and verify that the answer span and evidence are unchanged. Keep a complete failed run as well as the selected checkpoint; otherwise, reviewers cannot reconstruct the selection decision.

Walk through a fictional alignment experiment#

Consider a teaching exercise with 600 fictional document families and six questions per family. Assign 400 families to training, 100 to development and 100 to final assessment: 2,400, 600 and 600 questions. These are invented design choices, not measured Align results.

First, review demonstrations for the training families and fit the supervised baseline. Next, sample two alternatives for each of 600 training prompts and label those comparisons. Development review then finds a specific failure: an answer claims CSV export exists in release 3.2, while its supplied notes describe it as planned for 3.3.

Trace the failure backward. If the correct note was omitted, repair the evidence pipeline. If the note was supplied but the demonstration asserted availability, repair the demonstration. If the preferred response invented availability because it sounded decisive, revise the rubric and relabel affected training pairs. Do not copy the development question into training.

Create independently written training variations, deduplicate them against held-out families, and issue revision r2. Refit the affected stage, compare with the saved baseline, and inspect both unsupported claims and useful answers lost to refusal. Only after choosing a candidate by the declared development rule should the sealed assessment be opened. A critical failure blocks this exercise’s release regardless of preference wins.

Diagnose failures and limit the conclusion#

Concrete Problems in AI Safety distinguishes objective misspecification from problems such as distributional change. The following original diagnosis checklist makes that distinction actionable:

Successful checks establish evidence within a defined scope. They do not prove universal safety or agreement among everyone affected. Preserve unresolved objections, a rollback decision and a schedule tied to changes in models, evidence and permissions. The deliverable is a reproducible behavior claim with explicit limits.