Evaluate an agent trajectory as well as its final answer

An agent can reach a correct answer after unauthorized, wasteful or fragile steps. Final-answer scoring alone misses unnecessary calls, leaked data and invalid approvals.

Preserve a failing trajectory

The agent controls lab records three authored steps.

from agent_cases import trajectory_case

result = trajectory_case()
assert result["final_answer_pass"] is True
assert result["all_steps_allowed"] is True
assert result["unnecessary_steps"] == 1
assert result["trajectory_pass"] is False

The answer passes and every tool is allowed, but a duplicate read is unnecessary, so the trajectory gate fails. No agent ran.

Define step-level invariants

Check tool authorization, argument validation, evidence use, call order, retries, approvals, idempotency, budget and termination. Label mandatory, optional and forbidden actions. Some tasks permit multiple valid paths, so avoid demanding one exact trace when invariants suffice.

Measure task success beside calls, latency, cost, duplicate work and recoverability. Inspect whether a correct answer came from supplied evidence or luck. Keep traces privacy-minimized and link opaque receipts rather than secrets.

Add adversarial paths

Test prompt injection, stale approval, tool error, timeout, cancellation and misleading successful output. A model judge can assist after calibration, but deterministic permission and side-effect checks remain hard gates.

The Generative & Agentic AI course evaluates trajectory evidence in addition to final outputs.

Exercise

Label 20 traces for allowed, useful and ordered steps. Create a scorer that accepts multiple paths but rejects one unauthorized or duplicated critical action, then calibrate human disagreement.

Continue learning

This article is part of the Agent workflows and state sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Bound delegation depth and tool-call volume.
• Continue with Prevent a retrieved document from authorizing tool use.

Reference: OpenAI trace grading guidance.