Skip to content
GONATURELLE
AutoGen logoAgent framework

GONATURELLE for AutoGen

Evaluate AutoGen AgentChat agents and teams from their message stream, including every tool call request and execution result.

team.run_stream()
tests/test_triage_team.py
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.conditions import TextMentionTermination, MaxMessageTermination
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_ext.models.openai import OpenAIChatCompletionClient
from gonaturelle import EvalSuite, TrajectoryTester
from gonaturelle.integrations.autogen import AutoGenAdapter

from app.tools import lookup_symptoms, escalate_to_clinician

def build_team() -> RoundRobinGroupChat:
    client = OpenAIChatCompletionClient(model='gpt-4.1')
    triage = AssistantAgent('triage', model_client=client, tools=[lookup_symptoms, escalate_to_clinician])
    reviewer = AssistantAgent('reviewer', model_client=client, system_message='Reply APPROVED only if red flags were escalated.')
    stop = TextMentionTermination('APPROVED') | MaxMessageTermination(12)
    return RoundRobinGroupChat([triage, reviewer], termination_condition=stop)

agent = AutoGenAdapter(team_factory=build_team, agent='triage-team')

def test_triage_team():
    results = EvalSuite.load('healthcare/triage-escalation').evaluate(agent=agent, max_concurrency=5)
    t = TrajectoryTester(results)
    t.assert_called('escalate_to_clinician', when={'tags': 'red-flag'})
    t.assert_stop_reason_not('MaxMessageTermination')

AutoGen's AgentChat API, introduced with the 0.4 rewrite, builds agents such as AssistantAgent and composes them into teams like RoundRobinGroupChat and SelectorGroupChat. Running a team with run_stream yields a sequence of typed messages and events: text messages from each agent, ToolCallRequestEvent when a model asks for a function, ToolCallExecutionEvent when the function returns, and a final TaskResult with the stop reason.

That stream is a complete trajectory, and GONATURELLE's adapter consumes it directly. Each tool request and execution pair becomes a step with the calling agent's name, the function name, the JSON arguments and the result; text messages become reasoning steps; the termination condition and stop reason are recorded so a run that ended on max_turns rather than on a completion signal can be failed explicitly. Teams that hand control between agents get per-agent attribution for free.

A note on the roadmap: Microsoft has announced Microsoft Agent Framework as the successor that brings AutoGen and Semantic Kernel concepts together, and AutoGen itself now receives mainly maintenance updates. Many production agents still run on AgentChat, and the GONATURELLE adapter will keep supporting it. Because suites are framework-neutral, the same golden cases and assertions carry over unchanged if you migrate, which makes a migration a regression test rather than a leap of faith.

What you can test with AutoGen.

Termination and runaway detection

The stop reason from TaskResult is recorded with every case. Suites can fail runs that hit max_turns or repeat the same tool call, which in group chats is the usual sign of two agents talking past each other.

Speaker selection checks

For SelectorGroupChat, the sequence of speakers is part of the trajectory, so you can assert that the compliance reviewer always speaks before the final answer is emitted.

Function argument validation

Arguments in each FunctionCall are parsed and validated against the tool's schema and your suite's bounds, catching dosage, amount and date errors at the step where they appear.

Migration regression suites

Run the same pack against the AgentChat build and a rewritten build on another framework, then compare trajectories case by case before switching traffic.

Set up in 5 steps.

Most teams have a first suite running against AutoGen in under an hour, then add the CI gate.

Get an API key
  1. 01

    Install gonaturelle with autogen-agentchat and autogen-ext[openai] (or the model client extension you use).

  2. 02

    Export GONATURELLE_API_KEY, plus your model provider credentials.

  3. 03

    Write a factory that builds a fresh team per case, since teams hold conversation state, and wrap it with AutoGenAdapter.

  4. 04

    Load a suite and call evaluate; the adapter runs team.run_stream(task=case.input) and collects every message and event.

  5. 05

    Add termination, order and argument assertions, then add the CI gate.

AutoGen integration FAQ

Do you support AutoGen 0.2 (the pyautogen API)?

Not through the native adapter. Agents on the 0.2 ConversableAgent API can be evaluated with the generic trace decorator, recording tool calls from registered functions. We recommend AgentChat for new work and for anything you plan to gate in CI.

Should we migrate to Microsoft Agent Framework first?

That is a product decision for your team. Whichever way you go, write the GONATURELLE suite against your current build first: it becomes the acceptance test for the migration, with trajectory diffs showing exactly where the new implementation behaves differently.

How are teams with code execution handled?

Code executors run inside the GONATURELLE sandbox with network access denied by default. Executed code and its output are recorded as steps, and you can forbid execution entirely for suites where the agent should never run code.

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.