Compare model routing policies on quality-cost frontiers

A router should earn its complexity with a better quality-cost trade-off. Comparing quality in one report and cost in another makes that judgment easy to manipulate.

Define the measurable control

Run each policy on the same labelled cases. Include routing errors, retries and fallbacks in both quality and cost. Plot quality against cost and mark dominated policies: an option is dominated when another is at least as good and no more expensive. Choose a frontier point from business constraints.

The AI reliability lab makes the decision reproducible with authored data:

from reliability_cases import routing_frontier_case

result = routing_frontier_case()
assert result["frontier"] == ["strong_all", "small_all", "routed"]
assert result["dominated"] == ["wasteful"]
assert result["selected"] == "routed"
assert result["live_models_called"] is False

The fixture removes the wasteful policy because routed is higher quality and cheaper. Strong-all, small-all and routed remain on the frontier. Routed is selected for the authored requirements, and no live model was called; these numbers teach the comparison method only.

Challenge the result

Average quality may hide a critical slice routed to the weak model. Apply per-slice floors and test uncertain or adversarial requests. Count router latency and spend, and keep an explicit fallback when routing confidence is low.

Keep a reviewable evidence pack:

• paired dataset and policy definitions
• quality-cost frontier with slice floors
• routing confusion review
• fallback and model-version record

This work aligns with the evaluation, security, cloud operations, reliability and FinOps sequence in the FDE for Professionals course. The linked course describes the learning pathway; this article’s numbers are synthetic and do not report a model, client, audit or production result.

Practice task

Add a fifth policy that costs 55 and scores 0.91. Determine whether it belongs on the frontier and explain the decision mathematically.

Continue learning

This article is part of the Advanced AI reliability and assurance sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Load-test an AI workflow with realistic failure rates.
• Continue with Build a per-client AI cost attribution model.

Reference: OpenAI evaluation guidance.