Assert on agent reasoning, tool calls, and state transitions.
The right final answer can hide an unverified read and an invented account id. Toggle the prompt version and watch.
- 1fetch_account_balance(account_id="AC-44710")arg no upstream source
- 2verify_identity(customer_id="C-20417", otp="******")state: session.verified = true
- 3fetch_account_balance(account_id="AC-58810")arg from step 2 result
- 4respond("Happy to help! Your balance is $2,418.06.")
t.assert_order("verify_identity", "fetch_account_balance")fetch_account_balance at step 1 precedes verify_identity at step 2t.assert_no_hallucinated_args(tools=["fetch_account_balance"])account_id="AC-44710" at step 1 was produced by no prior stept.assert_state_before("session.verified", True, tool="fetch_account_balance")t.assert_max_calls("fetch_account_balance", 1)limit12 calls exceed limit 1t.assert_never_calls("transfer_funds")transfer_funds not invokedSeven steps in every trajectory. Each one can fail on its own.
What GONATURELLE records at each step, how that step typically breaks in regulated workflows, and the assertion that catches it.
| # | Step | Captured | Typical failure | Caught by |
|---|---|---|---|---|
| 1 | User intent | Raw input, channel, session metadata | Ambiguous or adversarial request accepted at face value | Judge: intent classification rubric |
| 2 | Agent plan | Plan or reasoning summary the framework exposes | Required step absent from the plan | assert_plan_includes() |
| 3 | Tool selection | Tool name, position in sequence | Wrong tool, forbidden tool, verification skipped | assert_order(), assert_never_calls() |
| 4 | Parameter extraction | Arguments with provenance per value | Out-of-bound dose, invented account id, wrong type | assert_param(within=), assert_no_hallucinated_args() |
| 5 | Tool execution | Result payload, latency, error | Error swallowed, retry storm | assert_handles_error(), assert_max_calls() |
| 6 | State mutation | Writes to memory, session or external state | Record updated before a check passed | assert_state_before() |
| 7 | Response synthesis | Final message and citations | Claim unsupported by any tool result | A.cites_only(), calibrated judge |
A deterministic assertion framework, written like pytest.
Four families of checks that never need a judge. They run first, fail fast and explain themselves.
Schema conformance and parameter boundaries
Every tool call is validated against its JSON schema and against domain bounds you attach: dose ranges from a formulary, loan amounts from a rate sheet, unit counts from a fee schedule. Type drift like units="2 vials" fails in milliseconds.
results.assert_tool_schema_valid()
t.assert_param('compute_dose', 'mg_per_kg_day', within=(80, 90))
t.assert_param('submit_prior_auth', 'units', within=(1, 12))Sequence constraints
Declare which calls must happen before others, which must never happen, and which must happen at least once. verify_identity() before fetch_account_balance() is one line.
t.assert_order('verify_identity', 'fetch_account_balance')
t.assert_never_calls('transfer_funds')
t.assert_called('check_allergies')Hallucinated argument detection
GONATURELLE traces each argument value to the user input, a prior tool result or a declared constant. A value with no upstream source, like an account id the model invented, is flagged with the step that introduced it.
t.assert_no_hallucinated_args(
tools=['fetch_account_balance', 'submit_claim'],
allow_constants=['USD'],
)Cycle detection and runaway loops
Repeated identical calls, ping-pong between two tools and unbounded retries are caught by a cycle detector with limits per tool and per turn, before they become a token bill or a duplicate submission.
t.assert_no_cycles(max_repeats=2)
t.assert_max_calls('check_coverage', 2)
t.assert_max_total_calls(12)Final-answer scores stay flat while trajectories move.
- of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat.
- 41%
- of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat.
- GONATURELLE regression study, 2026 (1,180 PRs across 14 production agents)
- of multi-step agent failures in the catalog were tool-argument schema or parameter-boundary violations, not reasoning breakdowns.
- 32.8%
- of multi-step agent failures in the catalog were tool-argument schema or parameter-boundary violations, not reasoning breakdowns.
- GONATURELLE Failure Modes Catalog analysis, 2026
- pass^8 for the best function-calling agent in the tau-bench retail domain: the chance it solves the same task correctly in all 8 trials. Single-trial success was under 50%.
- under 25%
- pass^8 for the best function-calling agent in the tau-bench retail domain: the chance it solves the same task correctly in all 8 trials. Single-trial success was under 50%.
- Yao et al., tau-bench, arXiv:2406.12045 (2024)
See exactly how the path changed between prompt versions.
The trajectory diff aligns the same case across two runs and marks steps that were added, removed or produced different output.
- verify_identity
- fetch_account_balance
- respond
- fetch_account_balanceunverified read
- verify_identity
- fetch_account_balance
- respondtone only
Order and provenance regressions. Final answer correct in both.
One test file per workflow, checked into the repo.
Assertions live next to your agent code, are reviewed in the same pull request as the prompt change, and run locally with the CLI before CI ever sees them.
gonaturelle run --suite fintech/servicing-balances --mode fast-path
import gonaturelle
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
from app.agent import balance_agent
suite = EvalSuite.load('fintech/servicing-balances')
def test_balance_trajectories():
results = suite.evaluate(agent=balance_agent, max_concurrency=10, trials=3)
results.assert_tool_schema_valid()
t = TrajectoryTester(results)
t.assert_order('verify_identity', 'fetch_account_balance')
t.assert_state_before('session.verified', True, tool='fetch_account_balance')
t.assert_no_hallucinated_args()
t.assert_never_calls('transfer_funds')
t.assert_no_cycles(max_repeats=2)
results.assert_pass_hat_k(k=3, min_rate=0.98)
results.check(A.no_pattern_in_tool_args(r'\b\d{3}-\d{2}-\d{4}\b'))Frequently asked questions
Which frameworks does trajectory capture support?
LangChain, LangGraph, LlamaIndex, CrewAI, AutoGen, the OpenAI Agents SDK, Semantic Kernel and DSPy through adapters, plus raw OpenAI, Anthropic and AWS Bedrock tool-calling loops. Custom loops use the trace decorator and step context manager. See the integrations catalog for setup per framework.
How does hallucinated argument detection work?
During capture, GONATURELLE records the user input and every tool result. For each argument value in a later call it searches those sources, with normalization for case, whitespace and common formats such as dates and currency. Values that match no source and no declared constant are reported as unsourced, with the step that introduced them.
Are trajectory assertions deterministic?
Yes. Order, schema, bound, provenance, state and cycle assertions are pure functions over the recorded trajectory. They return the same verdict on every replay, which is why they run before any judge and can gate a merge without a confidence threshold.
Agents are non-deterministic. How do you avoid flaky gates?
Sandboxes pin model versions and replay tool fixtures, and you can run each case k times and gate on pass^k, the probability that all k trials pass. Assertions that must hold every time use pass^k; softer rubric items use the ensemble mean with a tolerance band.
Test the path your agent took, not just where it landed.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.