Trajectory assertions vs final-answer evals

Same answer, different liability

Picture a retail banking agent that answers "What is my current balance?". Agent A calls verify_identity with the caller's one-time passcode, then calls fetch_account_balance, then replies with the balance. Agent B calls fetch_account_balance first, reads the number, then calls verify_identity, and only replies after the verification succeeds. Both replies contain the same correct balance. A final-answer eval, whether an exact match or an LLM judge, scores them identically.

Agent B has already retrieved a customer's financial data before establishing who is on the line. If verification had failed, the balance would still be sitting in the context, available to the next turn, to a summarizer, or to a log line. In a bank's risk taxonomy that is an access control failure regardless of what the user saw. The same pattern shows up everywhere we test: a clinical agent that drafts a recommendation before pulling the allergy list, a claims agent that sets a reserve before confirming coverage, a legal research agent that cites a case before retrieving it.

What we mean by a trajectory

A trajectory is the ordered record of everything the agent did between receiving the input and producing the output: each model turn, each tool call with its name and arguments, each tool result, and each write to external state. If your agent runs on LangGraph, LlamaIndex, CrewAI or plain function-calling code, the shape is the same once it is captured. The @gonaturelle.trace decorator records it without changing the agent's code path, and every assertion below runs against that record rather than against the final string.

Capturing the record is the easy part. The useful part is deciding which properties of it are requirements. We push teams to write those down as sentences first, in the language of the policy they come from, and only then translate them into assertions. "Identity is verified before any account data is fetched" is a requirement a compliance officer can read and sign. The assertion is just its executable form.

Four families of trajectory assertion

Almost every structural requirement we have encoded for customers falls into one of four families. Starting with these covers the large majority of what a regulated team needs before it reaches for a judge model.

• Ordering: tool X must precede tool Y, tool Z must never follow a failed X, a handoff must occur before any write when a flag is set.
• Parameter boundaries: a dose within the weight-based range, a credit line no higher than the policy cap, a date of service not in the future, an enum value from the allowed set.
• Argument provenance: every identifier passed to a tool must appear in an earlier tool result or in the user input. A member ID the agent made up is a hallucinated argument even if it happens to be well formed.
• Loops and state: no more than N retries of the same call, no cycle between two tools, no duplicate writes, and a bounded total step count.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

from agents.retail import balance_agent


def test_balance_inquiry_trajectories():
    suite = EvalSuite.load('fintech/retail-servicing-v1')
    results = suite.evaluate(agent=balance_agent, max_concurrency=10)

    tester = TrajectoryTester(results)
    tester.assert_order('verify_identity', 'fetch_account_balance')
    tester.assert_implies('verify_identity.failed', 'handoff_to_agent')

    results.assert_tool_schema_valid()
    results.assert_all([
        A.never_calls('fetch_account_balance', when='verify_identity.failed'),
        A.no_hallucinated_args(tool='fetch_account_balance', field='account_id'),
        A.param_within('issue_courtesy_credit.amount', max=25.00),
        A.no_cycles(tool='verify_identity', max_repeats=3),
    ])

Why this matters more after the first release

Trajectory assertions earn their keep on the tenth pull request, not the first. Before launch, a team usually walks through enough transcripts by hand to catch the obvious ordering bugs. After launch, the agent changes weekly: a prompt edit to fix a tone complaint, a new tool for a new product, a model upgrade. Each change is reviewed by reading the diff and spot-checking a few outputs, and the outputs look fine.

We measured how often that review misses a structural change. Across 1,180 pull requests that touched only prompts on 14 production agents, 41% changed at least one tool-call trajectory on the golden set while the final-answer scores stayed flat (GONATURELLE regression study, 2026). Most of those changes were benign: a redundant lookup added, two independent calls swapped. A minority were not, and they were indistinguishable from the benign ones in a final-answer report. A trajectory diff shows the reviewer exactly which case changed, at which step, and which assertion it now violates.

Deterministic first, judged second

Trajectory assertions are deterministic. Given the same recorded trajectory they return the same verdict, they run in milliseconds, and they never need a model call to evaluate. That makes them ideal for the fast path that runs on every push. It also makes them easy to defend: when an auditor asks why a build failed, the answer is a rule written in plain language and the step that broke it, not a judge's opinion.

Judges still have a job. The free-text rationale in a prior authorization letter, the tone of a collections message and the faithfulness of a summary to its sources need a rubric and a calibrated judge. We keep those checks in a separate stage and report them separately, so a red build tells you immediately whether you broke the process or the prose. Mixing both into one blended score hides the cause and makes the gate harder to trust.

When a final-answer check is enough

Not every agent needs trajectory assertions on every case. If a task has no side effects, uses no tools that touch regulated data and has a crisp correct output, such as extracting a policy number from a document, an exact-match or schema check on the result is sufficient and cheaper. The test we apply is simple: if a reviewer would care how the answer was reached, assert on the route. If they would only care what the answer is, grade the answer.

In regulated workflows, the first case is far more common than teams expect. The trajectory is where the evidence of process lives, and process is what regulators, payers and model risk teams audit. If you already capture traces in a framework, wiring them into these assertions takes an afternoon. Explore the demo to see a trajectory diff on a real pull request, or start a 14-day trial with your own agent.