Distillation: define what the smaller model must preserve

Distillation is useful only when the smaller system preserves the behaviours the application needs. Define those behaviours before generating teacher outputs or comparing speed.

Apply non-compensating requirements

The adaptation lab evaluates two authored student records.

from adaptation_cases import distillation_case

result = distillation_case()
assert result["requirements"]["safety_pass"] == 1.0
assert result["students"][0]["task_success"] == 0.88
assert result["students"][0]["accepted"] is False
assert result["accepted"] == ["small-b"]
assert result["training_executed"] is False

Small-a exceeds the task threshold but fails the safety invariant. Teacher and student values are fixtures; no training ran.

Specify the preservation contract

List output schema, critical task acceptance, evidence support, abstention, tool authorization, safety, multilingual slices and calibration. Set minimums plus hard failures. Add memory, latency and cost goals, but evaluate them only among acceptable candidates.

Version teacher, prompt, generation settings, data manifest and student training configuration. Teacher output is supervision, not truth: verify labels, sources and policy compliance. Mix trusted human references where consequences justify them, and keep an independent holdout outside teacher generation.

Compare teacher, base student and distilled student on identical tasks. Examine where the student disagrees and whether it copies teacher errors. Test long-tail and adversarial cases, local hardware behaviour and artifact reproducibility.

The Generative & Agentic AI course connects compression goals to critical gates and experiment evidence.

Exercise

Define five must-preserve behaviours for a toy classifier, create teacher labels with one injected error and show the student cannot pass merely by matching that error.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Synthetic training data: audit diversity and label quality.
• Continue with Preference tuning: distinguish ranking examples from factual supervision.

Reference: NIST AI RMF Generative AI Profile.