How to test multi-turn AI agents end to end

Why agents need a different kind of test

A chat model takes a prompt and returns text. An agent takes a goal, decides which tools to call, reads their results, calls more tools, talks to a user for several turns and eventually changes something in the world: a prior authorization is submitted, a payment is scheduled, a claim is reserved. The text it prints at the end is the least informative part of that process.

Most evaluation practice was built for the first shape. You hold out a set of prompts, generate answers, and compare them with references using exact match, a similarity metric or a model-graded score. Applied to an agent, that method inspects one step out of fifteen. It cannot tell you that the agent fetched a member's benefits before verifying who the member was, that it passed units='2 vials' to a field that expects an integer between 1 and 12, or that it called the same lookup nine times before giving up. Each of those is a compliance incident in healthcare or banking, and each can sit behind a perfectly reasonable final message.

There is a second difference. Agents are run many times on similar tasks by real users, and a single good run proves little. A prompt edit that makes the agent more concise can leave the average score unchanged while flipping a handful of trajectories from compliant to non-compliant. In GONATURELLE's regression study (2026), 41% of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat (1,180 PRs across 14 production agents). If your tests only look at final answers, those changes ship.

This guide describes the testing approach we use with teams building clinical, lending, legal and claims agents. It is opinionated about order: deterministic checks first, judges only where a rubric needs expertise, consistency measured explicitly, and everything wired into CI so that tests gate merges instead of decorating dashboards.

Define the unit of evaluation

Before writing a single case, decide what one test is. For agents the useful unit is a task: an initial state (database rows, documents, a patient chart), a user goal, the tools available, the policy the agent must follow, and the expected outcome. The outcome has three parts that you will check separately.

The trajectory is the ordered list of steps the agent took: messages, tool calls with arguments, tool results. The end state is what changed in the systems the tools touch: rows written, status fields, messages queued. The final response is what the user saw. A task passes only when all three are acceptable. tau-bench (Yao et al., 2024) made the end-state comparison central for exactly this reason: two very different conversations can be equally correct if they leave the database in the same goal state, and a fluent conversation that leaves the wrong state is a failure.

Write the expected outcome as constraints rather than a single golden transcript. A golden transcript is brittle: the agent may legitimately ask questions in a different order or phrase a denial differently. Constraints survive that variation. 'verify_member happens before fetch_benefits', 'submit_prior_auth is called exactly once with cpt_code 73721', 'no message contains a member's date of birth', 'the final status is PENDED with reason NETWORK_EXCEPTION' are all checkable and all stable across harmless rewording.

• Initial state: fixtures for every system the agent's tools read, with fictional people and synthetic records.
• Goal and persona: what the simulated user wants and how they behave, including what they withhold until asked.
• Policy: the rules the agent must follow, written down, because the expected outcome derives from them.
• Expected outcome: trajectory constraints, end-state assertions and a rubric for the final response.

Build the golden set with the people who know the domain

A golden dataset is only as good as its labels, and in regulated domains engineers cannot write the labels alone. Whether a knee MRI request meets medical-necessity criteria, whether a stated adverse action reason is specific enough, whether a clause in a redline changes indemnity exposure: these need a physician, a fair-lending specialist or an attorney. Our packs are drafted from de-identified real patterns and regulatory edge-traps, then labelled independently by at least two credentialed reviewers, with disagreements sent to adjudication.

Measure agreement before you trust the set. Cohen's kappa for two reviewers corrects raw agreement for chance; Landis and Koch (1977) read 0.61 to 0.80 as substantial and 0.81 to 1.00 as almost perfect. If two physicians only reach 0.5 on a rubric, the rubric is ambiguous, and no automated judge can be calibrated to a target the experts themselves do not share. Fix the rubric, relabel, and measure again. For safety rubrics where violations are rare, report Gwet's AC1 beside kappa, because kappa can be misleadingly low when almost every case is a pass.

Over-sample the edges deliberately. Production traffic is dominated by easy cases; failures concentrate at boundaries: pediatric weights near a dosing cut-off, applicants with thin credit files, claims filed one day before a state deadline, documents with conflicting governing-law clauses. One real edge case can be expanded into many variants by changing the irrelevant details and keeping the hard part, but every expanded case still needs its expected label checked by a human.

Finally, version the set. Use semantic versioning: patch releases fix labels, minor releases add cases, major releases change the rubric or task definition. Record the version in every result. A score is meaningless a year later unless you can say which cases and which rubric produced it.

Deterministic checks first: schemas, bounds and provenance

The cheapest, most reliable tests are the ones that need no judgment. Every tool your agent can call has a contract: parameter names, types, required fields, enumerations, numeric ranges and string patterns. Validate every call in every trajectory against that contract. Then add the domain bounds that JSON Schema cannot express, such as a formulary's maximum daily dose for a patient's weight or a lending policy's maximum loan-to-value ratio.

Schema problems are not a minor category. In the GONATURELLE Failure Modes Catalog analysis (2026), 32.8% of catalogued multi-step agent failures were tool-argument schema or parameter-boundary violations rather than reasoning breakdowns. They are also the failures prompt edits introduce most often, because a model asked to be more concise will happily shorten '2' into '2 vials' or drop an optional field your downstream system actually needs.

The subtler deterministic check is provenance. A hallucinated argument is a value no upstream step produced: an account number the user never gave, a claim ID that no search returned, a citation not present in any retrieved document. It validates against the schema and still fails the task. Check it by tracing each argument value back to a source in the trajectory: the user's messages, a prior tool result, or a constant the policy allows.

You do not need any platform to start. The test below runs with plain pytest and the jsonschema package against traces your agent already emits.

import json
from pathlib import Path

import pytest
from jsonschema import Draft202012Validator

SCHEMAS = {
    'submit_prior_auth': {
        'type': 'object',
        'required': ['member_id', 'cpt_code', 'units'],
        'properties': {
            'member_id': {'type': 'string', 'pattern': '^M[0-9]{8}$'},
            'cpt_code': {'type': 'string', 'pattern': '^[0-9]{5}$'},
            'units': {'type': 'integer', 'minimum': 1, 'maximum': 12},
        },
        'additionalProperties': False,
    },
}

TRACES = sorted(Path('traces').glob('*.json'))


def tool_calls(trace):
    return [s for s in trace['steps'] if s['type'] == 'tool_call']


@pytest.mark.parametrize('path', TRACES, ids=lambda p: p.stem)
def test_arguments_match_schema(path):
    trace = json.loads(path.read_text())
    for call in tool_calls(trace):
        schema = SCHEMAS.get(call['name'])
        if schema is None:
            continue
        errors = [e.message for e in Draft202012Validator(schema).iter_errors(call['args'])]
        assert not errors, f"{call['name']} at step {call['index']}: {errors}"


@pytest.mark.parametrize('path', TRACES, ids=lambda p: p.stem)
def test_member_id_was_never_invented(path):
    trace = json.loads(path.read_text())
    seen = ''  # text the agent had read before each call
    for step in trace['steps']:
        if step['type'] == 'tool_call':
            member_id = step['args'].get('member_id')
            if member_id:
                assert member_id in seen, f'member_id {member_id} has no upstream source'
        else:
            seen += json.dumps(step.get('content') or step.get('result') or '')

Trajectory assertions: order, required and forbidden steps

Once individual calls are valid, check the shape of the path. Most regulated workflows have ordering rules that come straight from policy: verify identity before reading a balance, verify the member before fetching benefits, check allergies before recommending a medication, gather the customer's investment profile before making a recommendation. Express each as an ordering assertion over tool calls. Pair them with required steps (an escalation tool must be called when red-flag symptoms appear), forbidden steps (no funds transfer without a confirmation turn) and limits (no more than six tool calls per turn, no identical call repeated three times).

These assertions are deterministic, so they fail the same way every time, which is what you want from a merge gate. They are also readable by the people who own the policy. A compliance officer can review assert_order('verify_identity', 'fetch_account_balance') without knowing Python, and can tell you when the rule is wrong.

The GONATURELLE SDK wraps the patterns above so that the same assertions run locally, in CI and against production traces you export later. The suite id pins a dataset version; the decorator captures trajectories from your existing agent code without a rewrite.

import gonaturelle
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

from agent import run_prior_auth  # your existing LangGraph, LlamaIndex or custom loop


@gonaturelle.trace(agent='prior-auth')
def my_clinical_agent(case):
    return run_prior_auth(case.input, tools=case.tools)


def test_prior_auth_suite():
    suite = EvalSuite.load('healthcare/clinical-prior-auth-v2')  # pinned to 2.3.0 in gonaturelle.lock
    results = suite.evaluate(agent=my_clinical_agent, max_concurrency=10)

    results.assert_tool_schema_valid()
    results.assert_zero_contraindications()

    t = TrajectoryTester(results)
    t.assert_order('verify_member', 'fetch_benefits')
    t.assert_required('escalate_to_clinician', when=A.case_tag('red_flag'))
    t.assert_never('submit_prior_auth', unless=A.after('check_criteria'))
    t.assert_max_calls(per_turn=6)
    t.assert_no_repeat(max_identical=2)

    results.assert_all(A.no_pattern_in_tool_args(r'\b\d{3}-\d{2}-\d{4}\b'))  # SSN-shaped values
    results.assert_all(A.judge(rubric='prior-auth-physician-v3', threshold=0.9))

Multi-turn tests need a user, an end state and a seed

Single-turn tests miss the failures that only appear in conversation: the agent forgets a constraint the user stated three turns ago, accepts a correction that contradicts the record, or acts before the user has confirmed. To test those you need someone on the other side of the conversation. A user simulator is a model or script that plays the patient, applicant or adjuster, pursuing a hidden goal and revealing information only when asked.

Simulators add their own variance, so pin them down. Give them explicit instructions about what they know and when to reveal it, fix their sampling settings, and record their messages as part of the trace so a failure can be replayed exactly. For high-stakes paths, replace the simulator with a scripted user whose turns are fixed; you lose some realism and gain a test that only fails when the agent changes.

Then check the end state. After the conversation, compare the fixture databases with the expected goal state: was the right authorization created with the right codes, was the account left untouched when identity verification failed, was the claim pended rather than denied. End-state checks catch the agent that says 'your request has been submitted' without submitting it, and the one that submits twice.

pass@k versus pass^k: measure consistency, not luck

Run a stochastic agent on the same task several times and it will not always do the same thing. Two metrics summarize those repeated trials, and they answer opposite questions.

pass@k is the probability that at least one of k attempts succeeds. It comes from code generation, where a developer can generate k candidates and keep the one that passes the tests. If a single attempt succeeds with probability p, pass@k is 1 minus (1 minus p) to the power k, which rises quickly toward 1 as k grows. It is the right metric when a human selects among candidates.

pass^k, introduced with tau-bench, is the probability that all k attempts succeed: p to the power k. It falls as k grows, and it is the right metric when the agent acts once, unsupervised, on a real customer each time. In tau-bench's retail domain the best function-calling agent succeeded on under 50% of tasks in a single trial, and its pass^8 was below 25% (Yao et al., 2024). An agent that gets a prior authorization right 90% of the time has a pass^8 of about 43%: across eight members with the same request, the chance that every one is handled correctly is less than half.

For regulated agents, report pass^k with k of at least 3 on your multi-turn tasks, and gate on it. Use enough trials that a single flaky run shows up. The GONATURELLE Vertical AI Reliability Index reports trajectory completion as pass^3 for this reason, and the gap between per-call schema adherence and pass^3 completion is the most consistent finding across its 48 models.