Turn safety evaluation into release gates

Separate three objectives#

Begin with three scorecards: safety violations on disallowed requests, successful assistance on permitted tasks, and unnecessary refusal on benign requests. A system that declines everything can appear safe while providing little value. The XSTest paper examines exaggerated safety behavior using safe prompts and unsafe contrasts.

Define the intended audience, permitted actions and unacceptable outcomes before running a candidate. Pair sensitive but legitimate questions with disallowed requests on similar topics. Label partial answers, clarification and appropriate abstention separately. Assign severity to concrete consequences rather than to keywords or an answer’s tone.

Combine benchmarks with domain cases#

Public benchmarks provide a shared reference, but an application also needs tests for its own documents, permissions and workflows. HarmBench standardizes automated red-teaming evaluation and shows why evaluation settings matter when comparing attack success.

Keep benchmark results separate from domain results. Build a coverage matrix across task, language, user role, failure consequence and input channel. Mark untested cells explicitly. Split development and held-out cases by scenario family, keeping close paraphrases together. Freeze the release set, corpus and rubric before tuning; rotate an exposed holdout rather than quietly continuing to optimize against it.

Declare denominators and budgets#

For this harness, define attempt-level attack success rate as confirmed successes divided by eligible, fully scored attack attempts. Also publish scheduled attempts, completed attempts, unscored outputs and reasons for exclusion. A timeout or judge parse failure is unknown, not evidence of safety. Report evaluation coverage so a favorable score cannot hide missing runs.

Report task-level success separately: the fraction of target tasks breached at least once within a fixed attempt budget. Record retries, sampling settings, output limits and tool permissions. Compare candidates under identical budgets. Repeated attempts on one task are correlated; they do not create an equivalent number of independent test cases.

Calibrate the judge#

Use deterministic checks for observable boundaries and a written rubric for semantic judgments. The MT-Bench judge study documents position, verbosity and self-enhancement biases in model-based grading.

Before automated grading, have reviewers label a representative calibration sample, including borderline failures and acceptable refusals. Compare the judge’s false positives and false negatives with those labels. For pairwise judgments, swap answer order and inspect disagreements. Version the judge, prompt and rubric. Escalate ambiguous or critical cases to human review; an unparseable verdict must remain visible rather than becoming a passing score.

Write a release contract#

This provider-neutral JSON describes an illustrative application policy, not an Align API or a measured result. The harness must implement each rule.

{
  "suite": "support-assistant-heldout-v1",
  "report_separately": ["safety", "usefulness", "over_refusal"],
  "critical_failure": "block_release",
  "unscored_case": "requires_review",
  "comparison": "paired_against_baseline"
}

Set tolerances before viewing results. Name a decision owner and preserve failed examples with the candidate configuration. A passed aggregate threshold must not override a designated critical failure. Document any accepted limitation and its review date.

Keep a release manifest containing the model identifier, prompt revision, corpus snapshot, harness version and tool policy. If any item changes during the run, mark affected comparisons invalid. Review the full evidence bundle before approval; a dashboard screenshot is insufficient to reproduce the decision.

Report uncertainty and limits#

Publish counts alongside rates and a stated confidence-interval method. A Wilson interval can describe a binary rate under an independent-case assumption. For paired comparisons, bootstrap scenario-family clusters when those clusters are the independent sampling units. Report subgroup sizes and intervals, not just a single average. Zero observed failures does not establish zero risk.

These intervals describe sampling uncertainty under the test design. They do not account for omitted threats, incorrect labels or future distribution changes. State those gaps explicitly. Treat a release gate as a decision about the tested configuration and scope, then repeat evaluation when models, data, permissions or tools change.