Skip to content
GONATURELLE
CircleCI logoCI/CD

GONATURELLE for CircleCI

Add a CircleCI config 2.1 job that runs GONATURELLE in fast-path or full mode, uploads test results and gates merges through branch filters and status checks.

Start 14-day trialSetup steps
config.yml version 2.1
.circleci/config.yml
version: 2.1

jobs:
  gonaturelle-gate:
    parameters:
      mode:
        type: enum
        enum: [fast-path, full]
        default: fast-path
    docker:
      - image: cimg/python:3.12
    steps:
      - checkout
      - run: pip install gonaturelle -r requirements.txt
      - run:
          name: Run GONATURELLE gate
          command: >
            gonaturelle run --suite gonaturelle.yml
            --mode << parameters.mode >>
            --baseline main
            --junit test-results/gonaturelle/results.xml
      - store_test_results:
          path: test-results

workflows:
  evaluate:
    jobs:
      - gonaturelle-gate:
          name: fast-path
          context: gonaturelle
          filters:
            branches:
              ignore: main
      - gonaturelle-gate:
          name: full-suite
          mode: full
          context: gonaturelle
          filters:
            branches:
              only: main

CircleCI's version 2.1 configuration supports parameterized jobs, which keeps the GONATURELLE setup to one job definition used twice in a workflow: once with mode set to fast-path for feature branches and once with mode set to full for the main branch or for pull request builds. Branch filters decide which instance runs, and the job's status is reported to GitHub or Bitbucket as a check.

The CLI writes JUnit XML to a directory that store_test_results uploads, so CircleCI's Tests tab lists each regressed case with its failing assertion, and test insights track how often specific cases fail over time. That history is useful for tuning the fast-path set: cases that never fail are candidates to move out, and cases that fail intermittently point to non-determinism worth pinning down with pass^k.

Store GONATURELLE_API_KEY in a CircleCI context restricted to the security groups that should run evaluations, and reference the context in the workflow. For agents that need private network access, run the job on a self-hosted runner resource class inside your environment; the GONATURELLE CLI only needs outbound HTTPS to your GONATURELLE endpoint.

What you can test with CircleCI.

Parameterized job

A single gonaturelle-gate job takes a mode parameter, so fast path and full suite share setup, caching and reporting while running different case sets.

Tests tab and insights

store_test_results surfaces failing cases and assertions in the Tests tab, and CircleCI's test insights show flaky or frequently failing cases across builds.

Contexts for secrets

The API key lives in a restricted context, so only approved projects and groups can run gates that spend eval runs or read results.

Self-hosted runners

Agents that call internal services run on self-hosted runners inside your network, with the same job definition.

Set up in 5 steps.

Most teams have a first suite running against CircleCI in under an hour, then add the CI gate.

Get an API key
  1. 01

    Create a CircleCI context such as gonaturelle with GONATURELLE_API_KEY, and restrict it to the right security group.

  2. 02

    Commit gonaturelle.yml with pinned suites, assertions and thresholds.

  3. 03

    Add the job and workflow below to .circleci/config.yml.

  4. 04

    Push a branch with a prompt change and confirm the fast-path job reports results in the Tests tab.

  5. 05

    Require the full-suite status check in your VCS branch protection so failures block merges.

CircleCI integration FAQ

How do we run the full suite on pull requests rather than only on main?

Add a third job instance with mode: full that runs on branches with an open pull request, using CircleCI's pipeline values or your VCS integration's pull request settings, and make that status check required for merging.

Can we schedule a nightly full run?

Yes. Create a scheduled pipeline in the project settings that triggers the evaluate workflow on main, or add a scheduled trigger with a full-mode job instance.

Does the job need Docker?

No. The example uses the cimg/python convenience image, but the CLI also runs on machine executors and self-hosted runners with Python 3.10 or later, or through the standalone binary.

Related integrations

All integrations

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo