Skip to content
GONATURELLE
DSPy logoAgent framework

GONATURELLE for DSPy

Evaluate compiled DSPy programs and ReAct agents against expert-labelled golden suites, using the trajectory DSPy records for each prediction.

Start 14-day trialSetup steps
dspy.ReAct trajectory
tests/test_claims_react.py
import dspy
from gonaturelle import EvalSuite, TrajectoryTester
from gonaturelle.integrations.dspy import DSPyAdapter

from app.tools import lookup_policy, check_exclusions

dspy.configure(lm=dspy.LM('openai/gpt-4.1-mini'))

react = dspy.ReAct('claim_text, policy_no -> decision, cited_clause', tools=[lookup_policy, check_exclusions])
react.load('artifacts/claims_react_mipro.json')

agent = DSPyAdapter(
    react,
    agent='claims-react',
    inputs=lambda case: {'claim_text': case.input, 'policy_no': case.meta['policy_no']},
    output='decision',
)

def test_claims_program():
    suite = EvalSuite.load('insurance/claims-adjudication')
    results = suite.evaluate(agent=agent, max_concurrency=8)
    t = TrajectoryTester(results)
    t.assert_order('lookup_policy', 'check_exclusions')
    t.assert_no_hallucinated_args()

DSPy treats prompts as parameters. You declare signatures such as "claim, policy -> decision, rationale", compose modules like ChainOfThought and ReAct, and let an optimizer such as MIPROv2 or BootstrapFewShot rewrite instructions and demonstrations against a metric. That makes evaluation central to DSPy, and it also creates a specific risk: an optimizer that maximizes a weak metric will happily learn to satisfy the metric instead of the regulation.

GONATURELLE fits in two places. First, as the held-out gate: after compiling, you run the optimized program against an GONATURELLE pack whose cases were never in the trainset or devset, with trajectory assertions and calibrated judges instead of a single exact-match metric. Second, as the metric itself: an GONATURELLE judge ensemble can be wrapped as a DSPy metric function, so optimization targets expert-calibrated scores rather than string overlap.

For dspy.ReAct, the returned Prediction includes a trajectory dictionary with the thought, tool name, tool arguments and observation for each iteration. The adapter converts that into standard GONATURELLE steps, so order, schema and hallucinated-argument assertions work exactly as they do for LangGraph or CrewAI agents. Program versions are hashed from the saved program state and recorded in the run manifest, so a compliance export names the exact compiled program that was tested.

What you can test with DSPy.

Held-out gating after optimization

GONATURELLE refuses to score a case whose hash appears in the trainset or devset you register, so the gate measures generalization rather than memorized demonstrations.

Calibrated judges as DSPy metrics

as_dspy_metric() turns a judge ensemble into a callable metric for dspy.Evaluate and optimizers, with the kappa of that ensemble recorded alongside the optimization run.

ReAct trajectory assertions

tool_name and tool_args from each ReAct iteration become ordered tool steps, checked for order, schema, bounds and argument provenance.

Program version lineage

Saved program state is hashed into the manifest, so you can compare two compiled versions case by case before promoting one.

Set up in 5 steps.

Most teams have a first suite running against DSPy in under an hour, then add the CI gate.

Get an API key
  1. 01

    Install gonaturelle and dspy, and configure your language model with dspy.configure(lm=dspy.LM(...)).

  2. 02

    Export GONATURELLE_API_KEY and register your DSPy trainset and devset with suite.exclude_examples so they never enter the gate.

  3. 03

    Load the compiled program from disk and wrap it with DSPyAdapter, mapping case fields to the signature's input fields.

  4. 04

    Optionally pass an GONATURELLE judge ensemble to your optimizer through as_dspy_metric.

  5. 05

    Run the held-out suite with trajectory assertions and add it to CI as the promotion gate for new compiled programs.

DSPy integration FAQ

Can GONATURELLE cases be used as DSPy training data?

Pack cases are licensed for evaluation, not for optimization or training, and the gate excludes any case it finds in your trainset. Use your own BYOD cases for optimization and keep GONATURELLE packs as the held-out test.

Does this work with modules other than ReAct?

Yes. Any module can be evaluated on its outputs with judges. For non-ReAct programs that call tools in custom forward() code, decorate those tools with gonaturelle.step to get trajectory assertions too.

How do judge-based metrics behave inside an optimizer?

They are slower and costlier than exact-match metrics, so most teams use a deterministic metric for the inner optimization loop and the calibrated ensemble for candidate selection and the final gate.

Related integrations

All integrations

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo