Prepare instruction examples without duplicating evaluation data

Instruction examples and evaluation rows often come from the same documents, conversations or templates. A random row split can put near-duplicates on both sides and inflate the apparent gain from tuning.

Group the source first

The adaptation lab gives two rows to each synthetic source. A naive split leaks S2; the grouped split does not.

from adaptation_cases import instruction_split_case

result = instruction_split_case()
assert result["naive_overlap"] == ["S2"]
assert result["grouped_overlap"] == []
assert result["eval_ids"] == ["I7", "I8"]
assert result["eval_labels_hidden"] is True

Exact source separation is necessary but not sufficient. Translated, paraphrased or template-derived examples may still carry the same answer.

Preserve provenance

Give every example an immutable ID, source ID, collection method, consent/rights status, authoring method, language and label history. Normalize and hash exact text, then review semantic-near duplicates and shared source passages. Split by customer, document, incident or template before generating variations.

Keep visible development examples, known regressions and a sealed final holdout. Prompt authors and annotators can use the development set. Restrict holdout labels and log every exposure. If a holdout row is inspected to fix a failure, move it into regression and replace it from the same sampling frame.

Audit instruction/response pairs for conflicting policies, sensitive data, unsupported facts and style shortcuts. Balance task slices deliberately; multiplying one template does not add equal information. Retain rejected and corrected examples with reasons.

The Generative & Agentic AI course treats data provenance and evaluation integrity as part of adaptation engineering.

Exercise

Create twenty source-grouped instruction pairs, inject two paraphrased duplicates and one conflicting label, then prove the split and audit catch them before any training job.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Fine-tuning versus retrieval for a changing knowledge base.
• Continue with LoRA explained through trainable parameter scope.

Reference: OpenAI Evals API.