Skip to content
GONATURELLE
CrewAI logoAgent framework

GONATURELLE for CrewAI

Score CrewAI crews per agent and per task, with each role's tool calls and hand-offs captured through step and task callbacks.

Start 14-day trialSetup steps
step_callback / task_callback
tests/test_underwriting_crew.py
from crewai import Agent, Crew, Process, Task
from gonaturelle import EvalSuite, TrajectoryTester
from gonaturelle.integrations.crewai import CrewAICapture

from app.tools import bureau_pull, score_application, draft_notice

def build_crew(capture: CrewAICapture) -> Crew:
    analyst = Agent(role='Credit analyst', goal='Decide the application under policy', backstory='Follows the credit policy manual.', tools=[bureau_pull, score_application])
    writer = Agent(role='Notice writer', goal='Draft an accurate adverse action notice', backstory='Writes Regulation B notices.', tools=[draft_notice])
    decide = Task(description='Decide application {app_id}.', expected_output='Decision with principal reasons', agent=analyst)
    notice = Task(description='Draft the notice for {app_id}.', expected_output='Notice text', agent=writer)
    return Crew(agents=[analyst, writer], tasks=[decide, notice], process=Process.sequential,
                step_callback=capture.on_step, task_callback=capture.on_task)

def run_case(case):
    capture = CrewAICapture(agent='underwriting-crew', case=case)
    return build_crew(capture).kickoff(inputs=case.inputs).raw

def test_crew():
    results = EvalSuite.load('fintech/adverse-action-reg-b').evaluate(agent=run_case, max_concurrency=4)
    t = TrajectoryTester(results)
    t.assert_only_agent_calls('score_application', role='Credit analyst')
    t.assert_order('score_application', 'draft_notice')

CrewAI organizes work as a Crew of role-based Agents that complete Tasks, either sequentially or under a manager in a hierarchical process. In regulated workflows the roles often mirror real separation of duties: an intake agent gathers facts, an analyst agent applies rules, a reviewer agent drafts the decision. Testing the crew as a black box hides the most important question, which is whether each role stayed inside its job.

GONATURELLE attaches to the Crew through its step_callback and task_callback parameters. Every agent step is recorded with the acting agent's role, the tool it chose, the tool input and the observation it received, and every completed task is recorded with its TaskOutput. The result is a trajectory in which each step is labelled by role, so assertions can say that only the underwriting analyst may call the pricing tool and that the reviewer must run after the analyst on every case.

Crews are kicked off with the case's inputs dictionary, which CrewAI interpolates into task descriptions, so the same crew definition you ship runs against hundreds of golden cases with no changes. GONATURELLE records CrewOutput token usage for cost tracking and gates. Flows can be evaluated by tracing the crews they kick off.

What you can test with CrewAI.

Role-scoped tool permissions

assert_only_agent_calls checks that sensitive tools are used only by the role that owns them, which catches prompt changes that let a drafting agent quietly approve its own output.

Task order and completeness

Task callbacks give an exact record of which tasks completed and in what order, so a sequential process that skips the compliance review task fails the case.

Delegation tracking

When allow_delegation is enabled, delegated work is recorded as a hand-off step with the delegating and receiving roles, so suites can bound or forbid delegation on high-risk cases.

Per-role judge scoring

Each task output can be scored by a different rubric: factual completeness for intake, rule application for the analyst, tone and disclosure for the customer-facing draft.

Set up in 5 steps.

Most teams have a first suite running against CrewAI in under an hour, then add the CI gate.

Get an API key
  1. 01

    Install gonaturelle in the same environment as crewai and your tools.

  2. 02

    Export GONATURELLE_API_KEY and keep your model provider keys configured as CrewAI expects.

  3. 03

    Pass the callbacks from CrewAICapture into Crew(step_callback=..., task_callback=...) in a factory function that builds a fresh crew per case.

  4. 04

    Point a suite at a function that calls crew.kickoff(inputs=case.inputs) and returns the raw final output.

  5. 05

    Add role and order assertions, run the fast path locally, then commit and enable the CI gate.

CrewAI integration FAQ

Why build a new crew per case?

Callbacks and memory are attached to the Crew instance. A fresh crew per case keeps trajectories separate and prevents memory from one case leaking into the next, which would make results order-dependent.

Does it work with hierarchical process and a manager agent?

Yes. Manager decisions and delegations appear as steps attributed to the manager role, so you can assert on which worker received each task as well as on the workers' own tool calls.

Can we evaluate CrewAI Flows?

Yes, by tracing the Flow's entry point with the GONATURELLE decorator and attaching CrewAICapture to each crew the Flow kicks off. Steps from all crews land in one trajectory, labelled by crew and role.

Related integrations

All integrations

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo