Synthetic training data: audit diversity and label quality

Generating thousands of rows does not create thousands of independent examples. Synthetic data can repeat one template, reproduce generator biases and attach confident but wrong labels.

Expose duplicate and conflict patterns

The adaptation lab audits six authored records.

from adaptation_cases import synthetic_audit_case

result = synthetic_audit_case()
assert result["duplicate_pairs"] == [
    ("S1", "S2"), ("S1", "S3"), ("S2", "S3")
]
assert result["conflicting_cancel_labels"] is True
assert result["coverage"]["technical:en"] == 1
assert result["generated_by_model"] is False

Three refund rows are exact categorical duplicates, and cancel examples conflict. The rows are manually authored fixtures; no generator ran.

Audit information, not volume

Record generator model/revision, prompt, seed/settings, source constraints, date, license and review state. Compare intents, languages, lengths, entities, styles and difficulty with the intended user distribution. Detect exact and semantic duplicates, template concentration and train/eval source overlap.

Validate labels with deterministic rules or blinded human review. Sample every slice and oversample high-risk or low-frequency categories for audit. Look for impossible facts, personal data, copied copyrighted text, policy conflicts and stereotyped associations. Keep generated content quarantined until it passes.

Evaluate a baseline with and without the synthetic rows on a sealed real or independently authored holdout. Report adverse slices and diminishing returns. Synthetic validation examples derived from the same generator can share the same blind spots and are not independent proof.

The Generative & Agentic AI course treats generation provenance and label review as model-development controls.

Exercise

Generate or manually create 100 safe fixture rows from five templates. Measure unique patterns and slice coverage, review a stratified sample, and reject the batch if conflict or duplicate thresholds are crossed.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure multilingual model quality by language.
• Continue with Distillation: define what the smaller model must preserve.

Reference: NIST AI RMF Generative AI Profile.