Production-grade evaluation for high-stakes AI agents.
One harness from trace capture to merge gate: sandboxed runs, expert-labelled suites, calibrated judges and signed evidence.
Five stages between a prompt change and a merge.
Click a stage to see what runs there and the payload it hands to the next one. The same pipeline runs on a laptop, in CI and inside your VPC.
Ingestion SDK
A decorator or callback handler records every model call, tool call, argument, tool result and state write as an ordered trajectory. Nothing in your agent code changes beyond the wrapper.
{
"run_id": "run_7f3c2a",
"agent": "prior-auth",
"step": 3,
"type": "tool_call",
"tool": "submit_prior_auth",
"args": { "cpt_code": "73721", "units": 1 },
"parent_step": 2,
"redactions": ["member_dob"]
}Four pillars, each walked through in three steps.
The parts you configure, in the order you configure them.
Pick a pack, add your own cases, pin a version.
Packs are scoped to a single regulated workflow, not a whole industry. Prior authorization is a different pack from SOAP-note summarization because the failure modes and the reviewers differ.
Why the architecture asserts on steps and calibrates its judges.
- of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat.
- 41%
- of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat.
- GONATURELLE regression study, 2026 (1,180 PRs across 14 production agents)
- of multi-step agent failures in the catalog were tool-argument schema or parameter-boundary violations, not reasoning breakdowns.
- 32.8%
- of multi-step agent failures in the catalog were tool-argument schema or parameter-boundary violations, not reasoning breakdowns.
- GONATURELLE Failure Modes Catalog analysis, 2026
- Cohen's kappa against physician, attorney and analyst consensus: calibrated GONATURELLE judge ensembles versus an uncalibrated single-prompt judge.
- 0.87 vs 0.42
- Cohen's kappa against physician, attorney and analyst consensus: calibrated GONATURELLE judge ensembles versus an uncalibrated single-prompt judge.
- GONATURELLE calibration study, 2026 (6,400 double-annotated cases)
- Best function-calling agent's pass^8 on tau-bench retail tasks.
- under 25%
- Best function-calling agent's pass^8 on tau-bench retail tasks.
- Yao et al., tau-bench, arXiv:2406.12045 (2024)
What GONATURELLE does not do.
Clear scope is part of the evidence. Reviewers trust a tool that says where it stops.
Not a runtime firewall or latency proxy
GONATURELLE runs offline, before deployment and on every change. It does not sit in your request path, inspect live traffic or block a production response in 500 ms. Pair it with your own runtime guardrails.
Not a substitute for clinical, legal or actuarial judgment
Credentialed reviewers label our packs and calibrate our judges, but a passing suite is evidence for your experts to review, not a replacement for their sign-off on a deployment.
Not a certifier
We do not issue government certifications, legal compliance certificates or FDA clearance. Reports document testing you performed; your organization remains accountable for deployment decisions and regulatory filings.
- Synthetic cases are generated for evaluation only and are not real medical or financial records. Nothing on this page is legal or medical advice.
Your framework stays. GONATURELLE attaches at the edges.
Framework adapters translate native callbacks and events into one trajectory format. In-house loops call the trace context manager directly.
LangChainCallback handler
LlamaIndexAgentWorkflow events
Semantic KernelFunction invocation filter
CrewAIStep and task callbacks
DSPyModule trajectories
import gonaturelle
@gonaturelle.trace(agent="claims-intake")
def handle(claim: dict) -> str:
with gonaturelle.step("tool_call", tool="lookup_policy") as s:
policy = lookup_policy(claim["policy_no"])
s.record(args={"policy_no": claim["policy_no"]}, result=policy)
with gonaturelle.step("tool_call", tool="open_fnol") as s:
fnol = open_fnol(policy_id=policy["id"], loss_date=claim["loss_date"])
s.record(args={"policy_id": policy["id"]}, result=fnol)
return summarize(fnol)Governance a model risk committee can audit.
Role-based access per suite
Owner, admin, engineer, reviewer and auditor roles. Thresholds and baselines can be locked so only a named reviewer group can change a gate.
Append-only audit trail
Who changed which threshold, which dataset version, which judge weight, and when. Every entry links to the run it affected.
Exportable compliance packages
Annex IV technical documentation sections, GMLP-aligned validation summaries and model-governance packs, built from signed run manifests.
SSO, SCIM and data residency
SAML SSO with Okta or Microsoft Entra ID, SCIM deprovisioning, US or India regions, single-tenant VPC or air-gapped Kubernetes.
Frequently asked questions
Does GONATURELLE run in our production request path?
No. GONATURELLE is an offline pre-production and regression evaluation engine. It runs on pull requests, merges, nightly schedules and on demand, against sandboxed copies of your agent. It adds no latency to production traffic, and it is not a runtime firewall.
Do we need to rewrite our agent to use it?
No. The Python and TypeScript SDKs wrap your existing entry point with a decorator or attach a framework callback (LangChain, LangGraph, LlamaIndex, CrewAI, Semantic Kernel, DSPy, OpenAI Agents SDK, AutoGen). Custom loops emit steps with a single trace context manager.
What happens to side-effecting tools during a run?
Tools that write, pay, submit or send are bound to recorded fixtures or stubs inside the sandbox. You choose per tool whether to replay a fixture, call a sandbox endpoint you host, or fail the case if the agent tries to invoke it.
Where does the data live?
In the multi-tenant US or India cloud, in a dedicated single-tenant VPC on AWS, GCP or Azure, or in a self-hosted air-gapped Kubernetes deployment. A HIPAA BAA is available on every plan and model providers we call operate under zero data retention terms.
Can we use GONATURELLE alongside LangSmith, Braintrust or Arize?
Yes, and many teams do. Keep your tracing and experiment tools for observability and iteration; use GONATURELLE for expert-validated vertical suites, deterministic trajectory assertions, calibrated judges and the merge gate. Traces can be promoted into GONATURELLE cases through OpenTelemetry or file export.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.