Block failing prompt merges in GitHub Actions in under 90 seconds.
Pick a trigger and what changed. The planner picks fast path or full suite and drafts the PR comment.
Native to GitHub Actions, GitLab CI and CircleCI.
One workflow file per repository. Each integration emits JUnit XML for the CI system’s own test views, posts a pull or merge request comment, and sets a status check you can make required.
name: gonaturelle-gate
on:
push:
branches-ignore: [main]
pull_request:
branches: [main]
permissions:
contents: read
pull-requests: write
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install -r requirements.txt
- uses: gonaturelle/gate-action@v2
with:
suite: gonaturelle.yml
mode: ${{ github.event_name == 'pull_request' && 'full' || 'fast-path' }}
baseline: main
comment: true
api-key: ${{ secrets.GONATURELLE_API_KEY }}stages: [test]
gonaturelle-fast-path:
stage: test
image: python:3.12-slim
rules:
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
script:
- pip install gonaturelle -r requirements.txt
- gonaturelle run --suite gonaturelle.yml --mode fast-path --junit report.xml
artifacts:
when: always
reports:
junit: report.xml
gonaturelle-full:
extends: gonaturelle-fast-path
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
script:
- pip install gonaturelle -r requirements.txt
- gonaturelle run --suite gonaturelle.yml --mode full
--baseline $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
--junit report.xml --mr-noteversion: 2.1
jobs:
gonaturelle-gate:
parameters:
mode:
type: string
default: fast-path
docker:
- image: cimg/python:3.12
steps:
- checkout
- run: pip install gonaturelle -r requirements.txt
- run: >
gonaturelle run --suite gonaturelle.yml
--mode << parameters.mode >>
--junit test-results/gonaturelle.xml
- store_test_results:
path: test-results
workflows:
eval:
jobs:
- gonaturelle-gate:
name: fast-path
filters:
branches:
ignore: main
- gonaturelle-gate:
name: full-suite
mode: full
filters:
branches:
only: main| case | assertion | main | #482 |
|---|---|---|---|
| PA-0977 | order(verify_member, fetch_benefits) | pass | fail |
| PA-1203 | schema submit_prior_auth.units | pass | fail |
| all | ensemble judge mean | 0.95 | 0.95 |
| all | tokens per case | 1,912 | 1,404 (−27%) |
prompts/prior_auth_system.md - Always confirm member eligibility before checking benefits. + Be concise. Check benefits and submit when criteria are met.
Run run_7f3c2a · fast path, 20 cases, 3 judges · 71 s · manifest sha256:9f2c41e0
A PR comment that names the cases, not just the score.
The bot shows exactly which cases regressed and on which assertion, the prompt diff that introduced the change, judge score deltas with their tolerance, and the token cost impact. One comment per PR, updated in place on each push.
Branch-based evaluation against production performance.
Every branch is compared with the baseline you choose, usually main or the tag currently deployed. Each metric has its own threshold, and any failed threshold fails the check.
| Metric | Baseline main@a41e9c0 | Branch prompt-concise | Threshold | Result |
|---|---|---|---|---|
| Critical assertion pass rate | 100% | 90% | 100% required | fail |
| Ensemble judge mean | 0.95 | 0.95 | drop ≤ 0.02 | pass |
| pass^3 on high-risk cases | 0.98 | 0.97 | ≥ 0.95 | pass |
| Tokens per case | 1,912 | 1,404 | rise ≤ 15% | pass |
| p95 agent latency | 6.1 s | 5.4 s | rise ≤ 20% | pass |
| Estimated cost per 1,000 cases | $11.40 | $8.30 | informational | info |
Fast path on every commit. Full suite where it counts.
Fast path
20 critical cases, three judges, every push. Chosen for risk tier, failure history and coverage of every critical assertion.
Full suite
Every case in every pinned suite, k trials for pass^k on high-risk cases, synthetic stress variants and cost drift, on merge requests, model bumps and nightly.
3,240 cases across 16 sandboxed shards. Results stream to the PR as shards finish.
Frequently asked questions
How fast is the gate?
The fast path runs 20 critical cases with three judges on every push. The median wall-clock time in GitHub Actions is 74 s (GONATURELLE platform telemetry, Q3 2026). Full suites shard across sandboxed runners and typically finish in minutes rather than hours, depending on case count and your agent's own latency.
Which cases go into the fast path?
You choose, or GONATURELLE proposes them: the cases most likely to catch a regression, ranked by how often each case has failed historically, its risk tier, and coverage of each critical assertion. The fast-path set is versioned with the suite so it does not change silently.
Does the bot post patient data or secrets into PR comments?
No. Comments include case ids, assertion names, score deltas and cost figures, never raw inputs or outputs. Reviewers click through to the run in GONATURELLE, where access is governed by your SSO roles and the case's redaction settings.
Can we require the gate before merging?
Yes. Mark the gonaturelle/gate check as required in GitHub branch protection or rulesets, require the GitLab job in merge request pipelines, or require the CircleCI status check. Overrides are logged with the approver's name for the audit trail.
What counts against our eval run quota?
Each case evaluated once against one agent version counts as one run, including judge scoring. A fast-path gate of 20 cases with three trials uses 60 runs. Plan limits and overage pricing are on the pricing page.
Make every prompt change earn its merge.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.