Release an AI feature using shadow evaluation

Offline evaluation cannot reproduce every production input shape, latency condition or integration path. Shadow evaluation adds evidence by running a candidate beside the current system while keeping candidate output away from users and external actions.

Compare paired fixture outcomes

The LLMOps and security lab evaluates four authored tasks on current and candidate routes.

from llmops_cases import shadow_release_case

result = shadow_release_case()
assert len(result["rows"]) == 4
assert result["candidate_pass_rate"] == 0.75
assert result["regressions"] == ["S2"]
assert result["candidate_served_to_user"] is False
assert result["fixture_outputs"] is True

S2 regresses even though the candidate improves S3. The four rows are too small for a release claim and no production traffic or model executed.

Isolate the shadow path

Copy only traffic covered by consent, purpose and data-handling policy. Remove or tokenize fields the candidate does not need. Run with separate credentials and enforce read-only tools or stub all effects. The candidate must never send a message, create a ticket or alter memory. Label and delete shadow artifacts under a defined retention rule.

Use a shared task ID to compare current and candidate outputs on the same eligible request. Record both release manifests, inputs’ data class, latency, cost and acceptance results. Exclude cases consistently and report the excluded volume. Candidate timeouts still count as outcomes.

Start with a small sample and capacity guardrails. Shadow work consumes provider quota and can affect the live path if it shares queues, connections or rate limits. Add a kill switch and stop when error, cost or privacy thresholds are crossed.

Decide after investigation

Review regressions by criticality and slice. A high average cannot override an authorization or sensitive-data failure. Shadow success supports a later canary; it does not itself expose users to the candidate or establish causal business impact.

The Generative & Agentic AI course links shadow evidence to release gates, canaries and rollback rules.

Reproducible exercise

Replay fifty scrubbed local requests to two deterministic adapters. Pair results by task ID, inject one critical regression and prove the shadow runner cannot call its write stub.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Version prompts, models, tools and evaluation sets together.
• Continue with Define a rollback rule after a model change.

References: OpenTelemetry GenAI conventions and NIST AI RMF Generative AI Profile.