Evaluate a fine-tuned model against its base model

An adapted model should earn release by improving the target behaviour without breaking important base capabilities. Evaluate both candidates on the same cases, prompts, tools and graders.

Preserve the regression

The adaptation lab contains four paired fixture outcomes.

from adaptation_cases import base_tuned_eval_case

result = base_tuned_eval_case()
assert result["base_pass"] == 0.50
assert result["tuned_pass"] == 0.75
assert result["regressions"] == ["B3"]
assert result["paired"] is True
assert result["fixture_outputs"] is True

The tuned aggregate rises, but safety case B3 regresses. No models ran, so the outcome illustrates comparison logic rather than adaptation performance.

Freeze the measurement contract

Pin base checkpoint, adapter and merged form, prompt, decoding, tools, retrieval snapshot, evaluation set and graders. Run identical task IDs with deterministic settings where supported; repeat variable runs and report their distribution. Blind human reviewers to candidate identity.

Separate target slices, general capability, safety, multilingual and out-of-domain cases. Report paired wins, losses and ties plus acceptance by slice. A mean score can hide one prohibited action. Critical invariants remain hard gates.

Compare operational evidence too: peak memory, warm/cold latency, throughput, artifact size and cost per accepted task on the intended hardware. Record failures and excluded runs. Use a sealed holdout only after development choices are fixed.

The Generative & Agentic AI course connects paired evaluation to release gates and experiment reports.

Exercise

Create a paired result table for twenty cases. Inject one critical regression and prove the release decision remains blocked even when the adapted aggregate improves by ten points.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in QLoRA trade-offs: memory savings and evaluation requirements.
• Continue with Detect overfitting in a small instruction dataset.

Reference: OpenAI Evals API.