Compare models using task quality, latency and cost

Model selection is a constrained decision. The highest task score may violate latency or cost limits; the cheapest model may fail critical cases. Define gates and a selection rule before running candidates.

Apply gates in order

The offline foundations lab uses authored comparison values.

from foundation_cases import model_comparison_case

result = model_comparison_case()
assert result["quality_gate"] == 80
assert result["latency_gate_ms"] == 500
assert result["eligible"] == ["small"]
assert result["selected"] == "small"
print(result["scope"])

The small fixture passes 84/100 tasks at p95 420 ms and costs 2.4 units per 1,000 tasks. The large fixture passes 91/100 but has p95 980 ms, above the 500 ms gate. No API ran; none of these values is a benchmark for a real model.

Measure the complete workflow

Use identical cases, output contracts, tools and retrieval. Quality should include deterministic invariants, human-reviewed cases and high-impact slices. Measure end-to-end latency with warm/cold conditions and realistic concurrency, not only provider-reported generation time.

Calculate cost per successful task. Include input/output tokens, retrieval, reranking, tool calls, retries, human review and failed tasks. A model with a lower call price can cost more if it needs repeated repairs.

Keep the decision current

Record exact model snapshots, regions, dates, limits and pricing sources. Provider names and prices change, so rerun rather than copying an old comparison. Use uncertainty or repeated runs for variable outcomes. Preserve per-case transitions; a higher average cannot excuse a regression on authorization or unsupported-answer gates.

Consider routing after establishing baselines: a smaller model may handle routine cases and escalate defined hard cases. Evaluate the router and total workflow, including misroutes.

The Generative & Agentic AI course connects model comparison to fixed eval sets, tracing and release decisions without promising one universally best provider.

Exercise

Compare two current model snapshots on 50 cases. Predeclare quality, critical-invariant and p95 gates. Report cost per successful task and date-stamped source prices, then write a decision record with conditions for reevaluation.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Design an abstention response for missing evidence.
• Continue with Build an offline fixture for an LLM integration.

Reference: OpenAI latency optimization guidance.