Preference tuning: distinguish ranking examples from factual supervision

A preference pair says one response is preferred under a rubric. It does not automatically say the chosen response is factually true. Mixing these signals without labels makes the training objective and evaluation difficult to interpret.

Keep supervision types explicit

The adaptation lab stores one ranking record and one source-backed factual record.

from adaptation_cases import preference_case

result = preference_case()
assert result["preference_ids"] == ["P1"]
assert result["factual_ids"] == ["F1"]
assert result["ranking_claims_truth"] is False
assert result["mixed_without_label"] is False

The preferred summary is ranked for concision while the invoice value has evidence. These are fixture records; no preference model or tuning job ran.

Write the rubric first

Define whether reviewers compare correctness, support, relevance, clarity, harmlessness or style. Do not collapse conflicting dimensions into an unexplained “better.” Verify factual prerequisites against sources before asking a style preference, or allow “both unacceptable.” Randomize candidate order and blind model identity.

Record prompt, candidates, source evidence, rubric version, reviewer, confidence and adjudication. Measure agreement and analyze systematic disagreements by language and task. Protect personal data and reviewer welfare.

Maintain separate factual supervision, preference pairs, safety constraints and evaluation holdouts. Evaluate the tuned candidate for both intended preference movement and factual/safety regressions. A more persuasive answer can be more dangerous if evidence support drops.

The Generative & Agentic AI course connects preference data to human review and independent evaluation.

Exercise

Create ten response pairs with a two-axis rubric. Include “both wrong” and “tie,” measure reviewer agreement and prove chosen style labels never overwrite factual reference fields.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Distillation: define what the smaller model must preserve.
• Continue with Write a model adaptation experiment report.

Reference: NIST AI RMF Generative AI Profile.