Stop evaluating against toy datasets. Test against clinical and regulatory reality.
Expert-labelled cases for the workflows regulators examine, with agreement published for every pack.
Adjudicator (MD, pediatrics): state network-adequacy rule allows an exception request, so pend rather than deny.
17 packs across five verticals
Browse verticalsLabelled by people who sign off on these decisions for a living.
MDs, JDs, CPAs, actuaries and former regulators write and label the cases. Dr. Meera Iyer leads clinical evaluation; Nandini Joshi leads regulatory science.
- 01
Draft from de-identified patterns
Cases start from de-identified production patterns supplied under contract and from regulatory edge-traps our regulatory science team maps from statute, guidance and enforcement actions. No case contains a real patient, borrower or claimant.
- 02
Two independent credentialed labels
At least two reviewers label each case without seeing each other's answer: physicians and nurses for clinical packs, attorneys for legal packs, CPAs, actuaries and former examiners for financial and insurance packs.
- 03
Adjudicate every disagreement
A third senior reviewer resolves disagreements and writes the reasoning into the case record. Cases where experts legitimately differ are kept, labelled as contested, and excluded from hard gates.
- 04
Publish agreement per pack
Each release reports Cohen's kappa between first-pass reviewers, the adjudication rate and the share of contested cases. If agreement falls below the floor for a pack, the release is held.
The edge-case taxonomy behind every pack.
Share of cases by category across the current clinical, financial and legal packs. Each case is tagged with one primary category and any number of secondary ones.
- 1Regulatory edge-trapsInputs that sit exactly on a rule boundary: a counteroffer the applicant accepts, a conservative-therapy window one week short, a filing deadline that falls on a state holiday.Inputs that sit exactly on a rule boundary: a counteroffer the applicant accepts, a conservative-therapy window one week short, a filing deadline that falls on a state holiday.31%
- 2Conflicting evidenceTwo sources disagree: a medication list and a discharge summary, a policy declaration page and an endorsement, a statute and a later amendment.Two sources disagree: a medication list and a discharge summary, a policy declaration page and an endorsement, a statute and a later amendment.22%
- 3Adversarial injectionInstructions hidden in retrieved documents, uploaded PDFs, OCR text or tool results that try to redirect the agent or exfiltrate data through tool arguments.Instructions hidden in retrieved documents, uploaded PDFs, OCR text or tool results that try to redirect the agent or exfiltrate data through tool arguments.14%
- 4Missing contextA required field is absent: no date of birth, no loss date, no jurisdiction. The correct behaviour is to ask or pend, never to guess.A required field is absent: no date of birth, no loss date, no jurisdiction. The correct behaviour is to ask or pend, never to guess.18%
- 5Ambiguous clinical chartsAbbreviations, copied-forward notes, negations and hedged findings where a careless summary turns "rule out MI" into a diagnosis.Abbreviations, copied-forward notes, negations and hedged findings where a careless summary turns "rule out MI" into a diagnosis.15%
One real edge case becomes hundreds of hard variants.
The synthetic expansion engine perturbs a reviewed seed case along controlled axes, keeps the gold label where the change cannot affect it, and sends label-affecting variants back to reviewers.
PA-0418 · Knee MRI after 3 weeks of physical therapy, no red flags
Bring your own data, de-identified on the way in.
Upload JSONL or CSV, or promote traces from OpenTelemetry, LangSmith exports or your own logs. Configurable PHI and PII detectors redact names, MRNs, SSNs, account numbers and dates before anything is stored.
- Tag by workflow, risk tier and edge-case category
- Version with the same semver rules as GONATURELLE packs
- Mix private suites and packs in one suite file
from gonaturelle import Client, Deidentifier
client = Client()
suite = client.datasets.upload(
path='cases/escalations.jsonl',
suite='prior-auth-internal',
deidentify=Deidentifier(
detectors=['name', 'mrn', 'dob', 'phone'],
strategy='surrogate',
),
tags={'workflow': 'prior-auth', 'risk_tier': 'high'},
)
print(suite.version, suite.case_count, suite.redaction_count)
# 1.0.0 212 37
suite.publish(bump='minor', note='Escalations from Q3 appeals queue')Semantic versioning, so last quarter's score still means something.
healthcare/clinical-prior-auth-v2 release history. Suite files pin an exact version; upgrades are explicit and reviewable.
| Version | Type | Released | Change | Effect on baselines |
|---|---|---|---|---|
| 2.3.0 | minor | 2026-10-14 | Added 120 GLP-1 step-therapy cases; rubric v3 adds documentation-gap item | New cases only; existing labels unchanged, baselines comparable on shared cases |
| 2.2.1 | patch | 2026-08-30 | Corrected 6 labels after adjudication review; typo fixes in 14 prompts | Scores may move on 6 cases; changelog lists each case id |
| 2.2.0 | minor | 2026-07-02 | Pediatric network-exception cases; contested flag on 22 cases | Contested cases excluded from hard gates by default |
| 2.0.0 | major | 2026-03-11 | Expectation schema v2: tool order and parameter bounds per case | Not comparable with 1.x; re-baseline required |
Frequently asked questions
Who are the reviewers and how are they qualified?
Reviewers are practising or recently practising professionals in the pack's domain: board-certified physicians and nurses for clinical packs, licensed attorneys for legal packs, CPAs, actuaries and former bank and insurance examiners for financial and insurance packs. Each pack's datasheet lists reviewer roles, credential types and years of experience, without personal names.
Do the packs contain real patient or customer data?
No. Cases are drafted from de-identified patterns and regulatory edge-traps, then rewritten so no record corresponds to a real person. Synthetic variants are generated for evaluation only and are not real medical or financial records.
How is inter-annotator agreement reported?
Per pack and per release: Cohen's kappa between the two first-pass reviewers on the categorical label, raw percent agreement, the adjudication rate and the count of contested cases. Rubric items scored on a scale also report weighted kappa.
Can we keep our own cases private while mixing them with packs?
Yes. BYOD suites stay in your tenant (or your VPC on Enterprise), are never added to shared packs and never used to train any model. A suite file can reference pack versions and private suites side by side.
What does a version bump mean for our baselines?
Patch releases fix labels or wording and list every affected case id. Minor releases add cases without changing existing ones, so shared-case comparisons stay valid. Major releases change the expectation schema and require a new baseline, which the CLI tells you before a run starts.
Test against the cases your reviewers worry about.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.