Skip to content
GONATURELLE
LangChain logoAgent framework

GONATURELLE for LangChain

Attach the GONATURELLE callback handler to any LangChain agent or runnable and every model call, tool call and retriever hit becomes an assertable trajectory step.

Start 14-day trialSetup steps
GonaturelleCallbackHandler
tests/test_prior_auth_langchain.py
from langchain.agents import create_agent
from gonaturelle import EvalSuite, TrajectoryTester
from gonaturelle.integrations.langchain import GonaturelleCallbackHandler

from app.tools import verify_member, fetch_benefits, submit_prior_auth

agent = create_agent(
    model='openai:gpt-4.1',
    tools=[verify_member, fetch_benefits, submit_prior_auth],
    system_prompt=open('prompts/prior_auth_system.md').read(),
)
handler = GonaturelleCallbackHandler(agent='prior-auth', redact=['member_dob'])

def run_case(case):
    out = agent.invoke(
        {'messages': [{'role': 'user', 'content': case.input}]},
        config={'callbacks': [handler], 'run_name': case.id},
    )
    return out['messages'][-1].content

def test_prior_auth():
    suite = EvalSuite.load('healthcare/clinical-prior-auth-v2')
    results = suite.evaluate(agent=run_case, max_concurrency=10)
    results.assert_tool_schema_valid()
    TrajectoryTester(results).assert_order('verify_member', 'fetch_benefits')

LangChain already exposes the hooks an evaluation harness needs: callback handlers fire on every chat model start and end, tool start and end, retriever query and chain boundary. GONATURELLE ships a handler built on langchain_core's BaseCallbackHandler that turns those events into ordered trajectory steps, so you test the agent you run in production instead of a reimplementation.

The handler is passed through the standard config argument when you call invoke, ainvoke or batch on an agent built with create_agent or any LCEL runnable. It records the tool name, the parsed input, the output, the parent run id and the latency of each step, and it redacts configured PHI and PII fields on the client before anything leaves the process. Because LangChain assigns a run id to every node in the call tree, GONATURELLE can reconstruct nested tool calls made by sub-chains as well as the top-level agent loop.

Teams that use LangSmith for tracing keep it. The GONATURELLE handler runs alongside the LangSmith tracer in the same callbacks list, and neither one changes prompts, tool arguments or control flow. The difference is what happens to the data: LangSmith is where you inspect and iterate, GONATURELLE is where expert-labelled vertical suites, deterministic trajectory assertions and calibrated judges decide whether a pull request can merge. Tested against langchain 1.x and langchain-core 1.x.

What you can test with LangChain.

Tool-call order and argument provenance

Each on_tool_start event is stored with its parsed input and the run that produced it, so assert_order and assert_no_hallucinated_args work on agents built with create_agent, legacy AgentExecutor graphs and custom LCEL tool loops alike.

Retriever evidence for grounding checks

Documents returned by retrievers are captured as sources. A.cites_only() then fails any answer that states a clinical, legal or policy fact that no retrieved document contains.

Structured output and schema validation

Tool inputs are validated against the args_schema of each tool, whether it was declared with the @tool decorator, a Pydantic model or a JSON schema, and boundary rules from your suite are applied on top.

Token and latency accounting per step

Usage metadata from chat model responses is attached to each generation step, which feeds the cost and latency drift checks in the CI gate without extra instrumentation.

Set up in 5 steps.

Most teams have a first suite running against LangChain in under an hour, then add the CI gate.

Get an API key
  1. 01

    Install the SDK in the same environment as your agent: pip install gonaturelle langchain langchain-openai (or the provider package you use).

  2. 02

    Create an API key in GONATURELLE under Settings, API keys, and export it as GONATURELLE_API_KEY in your shell and your CI secrets.

  3. 03

    Instantiate GonaturelleCallbackHandler with an agent name and pass it in the callbacks list of the config argument when the suite calls your agent.

  4. 04

    Write a suite adapter function that maps each case input to your agent's invoke payload, then load a pack with EvalSuite.load and call evaluate.

  5. 05

    Add trajectory assertions for the tools that matter, run gonaturelle run --mode fast-path locally, then commit the test file and add the CI gate.

LangChain integration FAQ

Does the handler work with async agents and streaming?

Yes. The handler implements both the sync and async callback interfaces, so ainvoke, astream and astream_events all produce the same trajectory. Steps are ordered by LangChain's run tree, not by arrival time, so concurrent tool calls are recorded in a stable order.

Can we run it next to the LangSmith tracer?

Yes. Put both in the callbacks list, or keep LANGSMITH_TRACING enabled in the environment and add the GONATURELLE handler in config. They observe the same events independently and do not interfere.

What if our tools are plain Python functions?

Functions wrapped with the @tool decorator carry a name and argument schema, which is all GONATURELLE needs. For functions without type hints, add a Pydantic args_schema so schema assertions have something precise to validate against.

Related integrations

All integrations

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo