GONATURELLE for AutoGen
Evaluate AutoGen AgentChat agents and teams from their message stream, including every tool call request and execution result.
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.conditions import TextMentionTermination, MaxMessageTermination
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_ext.models.openai import OpenAIChatCompletionClient
from gonaturelle import EvalSuite, TrajectoryTester
from gonaturelle.integrations.autogen import AutoGenAdapter
from app.tools import lookup_symptoms, escalate_to_clinician
def build_team() -> RoundRobinGroupChat:
client = OpenAIChatCompletionClient(model='gpt-4.1')
triage = AssistantAgent('triage', model_client=client, tools=[lookup_symptoms, escalate_to_clinician])
reviewer = AssistantAgent('reviewer', model_client=client, system_message='Reply APPROVED only if red flags were escalated.')
stop = TextMentionTermination('APPROVED') | MaxMessageTermination(12)
return RoundRobinGroupChat([triage, reviewer], termination_condition=stop)
agent = AutoGenAdapter(team_factory=build_team, agent='triage-team')
def test_triage_team():
results = EvalSuite.load('healthcare/triage-escalation').evaluate(agent=agent, max_concurrency=5)
t = TrajectoryTester(results)
t.assert_called('escalate_to_clinician', when={'tags': 'red-flag'})
t.assert_stop_reason_not('MaxMessageTermination')AutoGen's AgentChat API, introduced with the 0.4 rewrite, builds agents such as AssistantAgent and composes them into teams like RoundRobinGroupChat and SelectorGroupChat. Running a team with run_stream yields a sequence of typed messages and events: text messages from each agent, ToolCallRequestEvent when a model asks for a function, ToolCallExecutionEvent when the function returns, and a final TaskResult with the stop reason.
That stream is a complete trajectory, and GONATURELLE's adapter consumes it directly. Each tool request and execution pair becomes a step with the calling agent's name, the function name, the JSON arguments and the result; text messages become reasoning steps; the termination condition and stop reason are recorded so a run that ended on max_turns rather than on a completion signal can be failed explicitly. Teams that hand control between agents get per-agent attribution for free.
A note on the roadmap: Microsoft has announced Microsoft Agent Framework as the successor that brings AutoGen and Semantic Kernel concepts together, and AutoGen itself now receives mainly maintenance updates. Many production agents still run on AgentChat, and the GONATURELLE adapter will keep supporting it. Because suites are framework-neutral, the same golden cases and assertions carry over unchanged if you migrate, which makes a migration a regression test rather than a leap of faith.
What you can test with AutoGen.
Termination and runaway detection
The stop reason from TaskResult is recorded with every case. Suites can fail runs that hit max_turns or repeat the same tool call, which in group chats is the usual sign of two agents talking past each other.
Speaker selection checks
For SelectorGroupChat, the sequence of speakers is part of the trajectory, so you can assert that the compliance reviewer always speaks before the final answer is emitted.
Function argument validation
Arguments in each FunctionCall are parsed and validated against the tool's schema and your suite's bounds, catching dosage, amount and date errors at the step where they appear.
Migration regression suites
Run the same pack against the AgentChat build and a rewritten build on another framework, then compare trajectories case by case before switching traffic.
Set up in 5 steps.
Most teams have a first suite running against AutoGen in under an hour, then add the CI gate.
Get an API key- 01
Install gonaturelle with autogen-agentchat and autogen-ext[openai] (or the model client extension you use).
- 02
Export GONATURELLE_API_KEY, plus your model provider credentials.
- 03
Write a factory that builds a fresh team per case, since teams hold conversation state, and wrap it with AutoGenAdapter.
- 04
Load a suite and call evaluate; the adapter runs team.run_stream(task=case.input) and collects every message and event.
- 05
Add termination, order and argument assertions, then add the CI gate.
AutoGen integration FAQ
Do you support AutoGen 0.2 (the pyautogen API)?
Not through the native adapter. Agents on the 0.2 ConversableAgent API can be evaluated with the generic trace decorator, recording tool calls from registered functions. We recommend AgentChat for new work and for anything you plan to gate in CI.
Should we migrate to Microsoft Agent Framework first?
That is a product decision for your team. Whichever way you go, write the GONATURELLE suite against your current build first: it becomes the acceptance test for the migration, with trajectory diffs showing exactly where the new implementation behaves differently.
How are teams with code execution handled?
Code executors run inside the GONATURELLE sandbox with network access denied by default. Executed code and its output are recorded as steps, and you can forbid execution entirely for suites where the agent should never run code.
Related integrations
LangChain Agent frameworkCallback handler that records every tool call, retriever hit and LLM step from agents and runnables.
LangGraph Agent frameworkNode-level trajectory capture for StateGraph apps, including checkpointed state diffs between steps.
LlamaIndex Agent frameworkInstrumentation for AgentWorkflow and FunctionAgent runs, with retrieval nodes attached to each step.
CrewAI Agent frameworkPer-agent and per-task trajectories for Crew runs, so hand-offs between roles are assertable.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.