GONATURELLE for Anthropic
Capture tool_use and tool_result blocks from Anthropic Messages API agents, and run Claude models as cross-family judges in calibrated ensembles.
import os
import anthropic
from gonaturelle.integrations.anthropic import AnthropicToolLoop
from app.tools import TOOLS, call_tool # TOOLS: list of {'name', 'description', 'input_schema'}
client = anthropic.Anthropic()
MODEL = os.environ['CLAIMS_AGENT_MODEL']
def run_agent(user_input: str) -> str:
loop = AnthropicToolLoop(client, agent='fnol-intake')
messages = [{'role': 'user', 'content': user_input}]
for _ in range(8):
response = loop.create(model=MODEL, max_tokens=1024, tools=TOOLS, messages=messages)
messages.append({'role': 'assistant', 'content': response.content})
if response.stop_reason != 'tool_use':
return ''.join(b.text for b in response.content if b.type == 'text')
results = []
for block in response.content:
if block.type == 'tool_use':
ok, output = call_tool(block.name, block.input)
results.append(loop.tool_result(block.id, output, is_error=not ok))
messages.append({'role': 'user', 'content': results})
raise RuntimeError('tool loop exceeded 8 turns')With the Anthropic Messages API, tools are declared with a name, description and input_schema. When the model wants a tool, the response has stop_reason set to tool_use and contains tool_use content blocks, each with an id, the tool name and an input object. Your code runs the tool and replies with a user message containing tool_result blocks that reference the same tool_use_id. That explicit pairing makes Claude-based agents straightforward to evaluate step by step.
GONATURELLE's AnthropicToolLoop wraps client.messages.create and records every tool_use block with its input and every tool_result you send back, including whether you marked it is_error. Multiple tool_use blocks in one response are recorded as parallel steps. Text blocks become reasoning or response steps, and if extended thinking is enabled for the agent, you choose whether thinking content is stored with the run or dropped on the client.
Claude models are also available as judges. Because ensembles require a judge from a different model family than the agent under test, teams running agents on OpenAI or open-weight models often use a Claude model as the cross-family frontier judge, alongside rule checks and specialized small models. As with every provider, judge agreement with your expert labels is measured and published, and a judge is gate-eligible only above the kappa floor you set.
What you can test with Anthropic.
tool_use pairing
tool_use ids are matched to tool_result blocks, so a tool result that was never returned, or returned for the wrong id, is caught as a protocol error in the trajectory.
Error path coverage
tool_result blocks marked is_error are recorded, so suites can assert that the agent recovers from a failed eligibility lookup instead of guessing an answer.
Input schema validation
Each tool_use input is validated against the input_schema you declared, plus suite-level bounds such as allowed CPT code formats or maximum transfer amounts.
Claude as a calibrated judge
Use Claude models in judge ensembles for agents built on other model families, with Cohen's kappa and Gwet's AC1 reported against your expert panel.
Set up in 5 steps.
Most teams have a first suite running against Anthropic in under an hour, then add the CI gate.
Get an API key- 01
Install gonaturelle and the anthropic Python package (or @anthropic-ai/sdk for TypeScript).
- 02
Export ANTHROPIC_API_KEY and GONATURELLE_API_KEY, or configure keys in the tenant for VPC deployments.
- 03
Wrap your client with AnthropicToolLoop and call loop.create wherever you called client.messages.create.
- 04
Build tool_result blocks with loop.tool_result(block.id, output, is_error=...) so each result is recorded and paired.
- 05
Evaluate the agent function on a suite and add tool-order and error-recovery assertions before gating in CI.
Anthropic integration FAQ
Can we evaluate agents that use Claude through AWS Bedrock or Google Cloud?
Yes. If you call Claude through the Bedrock Converse API, use the AWS Bedrock integration; if you use the Anthropic SDK's Bedrock or Vertex clients, AnthropicToolLoop works the same way because the response shapes match.
Is thinking content stored?
Only if you enable it. By default the client drops thinking blocks before upload and keeps tool calls and text. Some teams store it for debugging on BYOD suites; it is never used as a grading input unless a rubric explicitly asks for it.
Which Claude model should we use as a judge?
The one that calibrates best against your experts on your rubric, which is an empirical question. The calibration engine measures each candidate on a hold-out set and recommends weights; you can also restrict candidates to models approved by your security team.
Related integrations
OpenAI Model providerEvaluate agents built directly on the Responses or Chat Completions API, and use OpenAI models as calibrated judges.
AWS Bedrock Model providerEvaluate Converse API agents inside your AWS account, with judges served from Bedrock in the same region.
LangChain Agent frameworkCallback handler that records every tool call, retriever hit and LLM step from agents and runnables.
LangGraph Agent frameworkNode-level trajectory capture for StateGraph apps, including checkpointed state diffs between steps.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.