Gate a prompt change before it reaches a patient, borrower or court.
Pick a vertical suite, choose the baseline or the pull request, and press run. Everything below runs in your browser.
Agent version under test
#482 Make determination letters more concise · s.rao
- Always confirm member eligibility with verify_member() before any benefits lookup.
+ Be concise. Skip steps that do not change the determination.
- Cite the exact policy section that supports a denial.
+ Summarize the policy basis in one sentence.
Press run to test this version
main vs baseline
| # | Case | Check | Latency | Baseline | This run |
|---|---|---|---|---|---|
| 1 | Judge | pass | pass0.96 | ||
| 2 | Judge | pass | pass0.94 | ||
| 3 | Trajectory | pass | pass | ||
| 4 | Trajectory | pass | pass | ||
| 5 | Schema | pass | pass | ||
| 6 | Privacy | pass | pass | ||
| 7 | Judge | pass | pass1.00 | ||
| 8 | Judge | pass | pass0.93 | ||
| 9 | Boundary | pass | pass | ||
| 10 | Boundary | pass | pass | ||
| 11 | Trajectory | pass | pass | ||
| 12 | Schema | pass | pass |
Gate decision
Mergeable
- Pass rate
- 100.0%
- Threshold
- 95%
- Critical fails
- 0
Judge calibration
Cohen’s kappa vs expert labels- Deterministic rules · w 0.400.93
- Clinical judge 8B · w 0.350.84
- Frontier judge, cross-family · w 0.250.78
- Weighted ensemble0.88
Ensemble 0.88 reads as almost perfect agreement (Landis & Koch, 1977). Order swapping and length normalization on every pairwise judgement.
PA-0418 · Lumbar MRI, under 6 weeks of conservative therapy
main · Judge check
- 214 msverify_member()member_id="M-204417", dob_hash="a91f..."
- 193 msfetch_benefits()plan_id="HMO-GOLD-24", service="72148"
- 216 mslookup_policy()policy_id="MRI-LSPINE-07", section="3.2"
- 195 mscheck_contraindications()rx=["warfarin", "fluconazole"]
- 218 mssubmit_prior_auth()cpt_code="72148", units=1, urgency="standard"
- 197 msdraft_determination_letter()decision="pend", cite="MRI-LSPINE-07 §3.2"
Audit event log
- 09:42:10.000run.queued run_914e8df1cc queued for main · prompt v14 on prior-auth-agent
- 09:42:10.640sandbox.provisioned runner us-east-1/r-07, network egress denied, ZDR model routes
- 09:42:10.910suite.loaded healthcare/clinical-prior-auth-v2@2.3.0, 12 critical-path cases of 1,840, sha256 5cedb1d64536
- 09:42:12.280case.completed PA-0418 pass (ensemble 0.96) in 1080 ms
- 09:42:13.490case.completed PA-0977 pass (ensemble 0.94) in 1210 ms
- 09:42:13.910case.completed PA-1203 pass in 420 ms
- 09:42:14.300case.completed PA-1340 pass in 390 ms
- 09:42:14.420case.completed PA-1388 pass in 120 ms
- 09:42:14.515case.completed PA-1452 pass in 95 ms
- 09:42:15.455case.completed PA-1517 pass (ensemble 1.00) in 940 ms
- 09:42:16.605case.completed PA-1602 pass (ensemble 0.93) in 1150 ms
- 09:42:16.665case.completed PA-1688 pass in 60 ms
- 09:42:16.805case.completed PA-1730 pass in 140 ms
- 09:42:17.185case.completed PA-1795 pass in 380 ms
- 09:42:17.255case.completed PA-1841 pass in 70 ms
- 09:42:17.475judges.calibration ensemble kappa 0.88 vs expert labels (rules 0.93, specialized 0.84, frontier 0.78)
- 09:42:17.515gate.decision mergeable: pass rate 100.0%, 0 critical failures
- 09:42:17.665manifest.signed sha256 ef5e10c082c42f43f0b346295a5fa28671a72c3b9886ac62902c90f39268ef8a
What you just watched, in production terms.
The sandbox shows 12 critical-path cases per suite. In CI, the fast path runs 20 critical cases on every push and the full pack runs on merge or nightly.
- Trajectory inspectorEvery tool call with its arguments. Missing steps, out-of-bounds parameters and hallucinated arguments are marked at the step where they happen.
- Judge breakdownPer-judge scores, ensemble weights and each judge's kappa against expert labels for the pack, so a reviewer can see why a case passed.
- Gate decisionPass-rate threshold plus zero critical failures. The same rule runs in gonaturelle/gate-action@v2 and posts to the pull request.
- Audit exportEvery event with a timestamp, the gate reasons and a SHA-256 manifest digest, exported as JSON for your model-risk or quality file.
The same gate, one workflow file away.
Add the action, point it at a suite and set your thresholds. The check fails the pull request on a critical failure or a pass rate below the line, and the bot comments with the diff you saw above.
name: gonaturelle-gate
on: [pull_request]
jobs:
fast-path:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: gonaturelle/gate-action@v2
with:
suite: healthcare/clinical-prior-auth-v2
mode: fast-path
pass-rate: "0.95"
block-on-critical: "true"
env:
GONATURELLE_API_KEY: ${{ secrets.GONATURELLE_API_KEY }}Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.