Skip to content
GONATURELLE
OpenAI logoModel provider

GONATURELLE for OpenAI

Evaluate agents built directly on the OpenAI Responses or Chat Completions API, capture function calls from the raw loop, and use OpenAI models as calibrated judges.

Start 14-day trialSetup steps
Responses API function calls
app/agent_loop.py
import json
from openai import OpenAI
from gonaturelle.integrations.openai import OpenAIToolLoop

from app.tools import TOOLS, call_tool  # TOOLS: list of {'type': 'function', 'name', 'parameters'}

client = OpenAI()

def run_agent(user_input: str, model: str = 'gpt-4.1') -> str:
    loop = OpenAIToolLoop(client, agent='kyc-triage')
    items = [{'role': 'user', 'content': user_input}]
    for _ in range(8):
        response = loop.create(model=model, input=items, tools=TOOLS)
        items += response.output
        calls = [o for o in response.output if o.type == 'function_call']
        if not calls:
            return response.output_text
        for call in calls:
            result = call_tool(call.name, json.loads(call.arguments))
            items.append(loop.tool_result(call.call_id, json.dumps(result)))
    raise RuntimeError('tool loop exceeded 8 turns')

Many regulated teams skip frameworks and write the tool loop themselves against the OpenAI API: send input with a tools list, read function_call items from the response, execute the function, append a function_call_output item with the matching call_id, and repeat until the model answers. The loop is short, auditable and easy to reason about, which is why it is popular with compliance reviewers.

GONATURELLE instruments that loop with a thin wrapper around the client. OpenAIToolLoop records each function_call item with its name, the parsed arguments JSON and the call_id, pairs it with the output you return, and keeps the model's text output as the response step. The same wrapper supports the Chat Completions API, where tool calls appear in message.tool_calls. Arguments that fail to parse as JSON, or that parse but violate the function's parameters schema, are recorded as schema failures rather than silently coerced.

OpenAI models can also serve as judges. Inside GONATURELLE ensembles, a judge from a different model family than the agent under test is required by default to limit self-preference bias, so an agent running on an OpenAI model is scored by a non-OpenAI frontier judge plus rule checks and specialized small models, and vice versa. GONATURELLE calls providers under zero data retention terms; customers can also bring their own API keys and organization settings.

What you can test with OpenAI.

Raw loop capture

function_call and function_call_output items are paired by call_id, so parallel function calls in one response are recorded as separate, correctly matched steps.

Strict schema checks

Each call is validated against the parameters schema you sent in the tools list. Teams using strict mode still benefit, because suite bounds catch values that are schema-valid but clinically or financially wrong.

Model version pinning

The model identifier returned with each response is recorded in the manifest, so a gate can fail when a dated snapshot changes underneath the agent.

Cross-family judging

When the agent runs on an OpenAI model, ensembles automatically include a judge from another family, with agreement reported per judge.

Set up in 5 steps.

Most teams have a first suite running against OpenAI in under an hour, then add the CI gate.

Get an API key
  1. 01

    Install gonaturelle and the openai Python package (or @gonaturelle/sdk and openai for TypeScript).

  2. 02

    Export OPENAI_API_KEY and GONATURELLE_API_KEY. For Enterprise VPC deployments, configure your own key in the tenant settings instead.

  3. 03

    Replace direct client.responses.create calls inside your loop with loop.create, which forwards every parameter unchanged and records the output items.

  4. 04

    Record each function result with loop.tool_result(call_id, output) before sending it back to the model.

  5. 05

    Load a suite, evaluate the loop function, and add schema and order assertions before enabling the CI gate.

OpenAI integration FAQ

Do you support the Chat Completions API as well as Responses?

Yes. OpenAIToolLoop detects which API you call and reads tool calls from message.tool_calls for Chat Completions or from function_call output items for Responses. The resulting trajectory is identical.

Does GONATURELLE send our data to OpenAI?

Only if your agent or your chosen judges call OpenAI. Agent calls go wherever your code sends them. Judge calls made by GONATURELLE use zero data retention terms, and you can restrict judges to providers your organization has approved.

How do you handle non-determinism across runs?

Pin a dated model snapshot, set temperature low for evaluation runs, and use k trials with pass^k for critical cases. The manifest records the returned model identifier so changes in the served snapshot are visible.

Related integrations

All integrations

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo