Build a release gate that blocks a known regression

An aggregate pass rate can improve while a critical case regresses. Release policy must distinguish negotiable quality movement from failures that violate authorization, privacy, safety or core task contracts.

Block the critical case

The LLMOps and security lab runs three authored outcomes. Two pass, but critical regression R-2 fails.

from llmops_cases import release_gate_case

result = release_gate_case()
assert round(result["aggregate_pass_rate"], 3) == 0.667
assert result["critical_failures"] == ["R-2"]
assert result["known_regression"] == "R-2"
assert result["decision"] == "block"

The gate does not average away R-2. This demonstrates the rule; it is not a model evaluation because the outcomes are fixtures.

Encode policy as data

Each test needs a stable ID, owner, severity, task slice, expected behaviour and evidence source. Store hard gates for unauthorized writes, cross-tenant access, secret disclosure, invalid required schemas and known severe incidents. Add statistical thresholds for broader task quality, abstention, latency and cost.

Pin the full release manifest and evaluation-set version. Fail closed when required results are missing, timed out or executed on the wrong bundle. Verify the evaluator itself with deliberately passing and failing controls. A green dashboard generated from stale cached evidence should not release new code.

Allow overrides only through an explicit, authenticated record with approver, reason, expiry and compensating control. Do not edit a failing row or lower severity to make a build pass. Quarantine invalid data separately and preserve the original result.

Feed incidents back into regression

Every confirmed material failure should become the smallest reproducible case plus variants. Keep a sealed holdout as well; regression suites are necessary but increasingly familiar to developers and prompts.

The Generative & Agentic AI course combines hard gates with representative evals, shadow tests and rollback.

Reproducible exercise

Create a command-line gate that reads a JSON result file. Prove a 99% aggregate still exits non-zero when one critical case fails, when a required result is missing and when the release digest differs.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Write an AI incident postmortem from trace evidence.
• Continue with Separate offline evaluation from production success metrics.

Reference: OpenAI Evals API.