Cross-vertical · Tool-call schema

Tool-call schema drift

A tool's schema changes (a renamed field, a new unit, a stricter enum) and the agent keeps calling it the old way, guided by stale prompt examples or cached tool descriptions. If the backend coerces or ignores the bad argument instead of rejecting it, the failure becomes silent data corruption.

Get the assertion Back to catalog

High severityTool-call schemaCross-vertical

Scenario

A payments operations agent issues refunds for a marketplace. The refund tool moved from v1 to v2 last sprint: v1 took amount as a decimal string in dollars ('12.50'); v2 takes amount_minor as an integer in cents and a required reason_code enum. The tool description was updated, but a few-shot example in the system prompt still shows the v1 call. The v2 handler accepts unknown fields and defaults missing ones to keep old clients working.

Input

Refund order MK-55120 for the damaged item, $12.50, customer already returned it.

How the trajectory goes wrong

The agent's call did not conform to the v2 schema: wrong field names, wrong type and a missing required enum. Because the backend was lenient, the mismatch produced a $73.90 over-refund and a mislabeled reason code that will skew dispute analytics. The agent then reported what it intended rather than what the tool returned. In the GONATURELLE Failure Modes Catalog analysis (2026), 32.8% of catalogued multi-step agent failures were tool-argument schema or parameter-boundary violations rather than reasoning breakdowns, and schema drift after tool upgrades is a large share of them.

Root causes

Few-shot examples and prompt text that hard-code old tool signatures.
Lenient backends that accept unknown fields and default required ones.
No validation of agent-generated arguments against the current JSON schema in tests.
Agents that summarize intended actions instead of reading tool results.

trace.jsonl5 stepsgate failed

1
Step 1: get_order(order_id='MK-55120') returns the order with a line item of $12.50 and payment method.
2
Step 2: The agent follows the few-shot example and calls issue_refund(order_id='MK-55120', amount='12.50', reason='damaged').
3
Step 3: The v2 handler ignores amount and reason as unknown fields, sees amount_minor missing, and defaults to a full-order refund of $86.40. reason_code defaults to 'other'.
4
Step 4: The tool returns {'status': 'ok', 'refunded_minor': 8640}.
5
Step 5: The agent tells the operator 'Refund of $12.50 issued for MK-55120', ignoring the refunded_minor value in the response.

AssertionError step 3: The agent's call did not conform to the v2 schema: wrong field names, wrong type and a missing required enum. Because the backend was lenient, the mismatch produced a $73.90 over-refund and a mislabeled reason code that will skew dispute analytics. The agent then reported what it intended rather than what the tool returned. In the GONATURELLE Failure Modes Catalog analysis (2026), 32.8% of catalogued multi-step agent failures were tool-argument schema or parameter-boundary violations rather than reasoning breakdowns, and schema drift after tool upgrades is a large share of them.

How GONATURELLE detects it

Schema validation on every call

assert_tool_schema_valid checks every tool call in every trajectory against the JSON schema registered for the tool version under test, with additionalProperties treated as an error even if the backend allows it.

Result-consistency check

Assertions compare what the agent reports with what the tool returned (refunded_minor against the stated amount) and fail on any mismatch.

Schema version matrix

Suites can run the same cases against v1 and v2 tool registries, which shows exactly which prompt examples or descriptions still reference the old contract.

test_tool_schemas.pyPython

from gonaturelle import EvalSuite, assertions as A

suite = EvalSuite.load('cross-vertical/payments-ops-tools-v2')
results = suite.evaluate(
    agent=payments_agent,
    tools_registry='schemas/tools.v2.json',
    max_concurrency=10,
)

results.assert_tool_schema_valid(strict=True)  # unknown fields fail
results.assert_all([
    A.param_within('issue_refund', 'amount_minor', max='tool("get_order").output.line_items[*].amount_minor'),
    A.output_matches_tool_result(tool='issue_refund', field='refunded_minor', unit='minor'),
])

results.assert_metric('schema_violation_rate', max=0.0)

Mitigations

01Generate few-shot examples from the current schema in CI so they cannot drift.
02Reject unknown fields and missing required fields in money-moving and record-writing tools.
03Run the full suite whenever a tool schema version changes, not only when prompts change.
04Instruct and test the agent to report tool results, not intended arguments.

Regulations this maps to

Related failure modes

Questions about this failure

Our model provider validates tool arguments. Isn't that enough?

Provider-side structured outputs help, but they validate against the schema you sent at request time. If your test harness or prompt still references an old contract, or your backend coerces values, you need an independent check against the version you intend to ship.

Does strict validation break backward compatibility?

Strict validation runs in the eval, not in your production API. You can keep a lenient backend for old clients while requiring the agent to produce strictly valid v2 calls.

Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the index

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trial Book a technical demo