Skip to content
GONATURELLE

Research and field guides for regulated AI agents

Benchmarks, failure catalogs and regulatory testing guides written by the clinicians, attorneys and engineers who build GONATURELLE.

Explore the IndexSearch failure modes
Open Open Open Open Open

Research

Original measurement, published openly and versioned so you can cite it.

Vertical AI Reliability Index · November 202648 models, 12 regulated task domains, four metrics that predict production failure.
1Reasoning closed A · trajectory 61.277.1
2Frontier closed A · trajectory 58.276.6
3Reasoning closed B · trajectory 59.675.0
Median pass^3 trajectory completion across all models: 31.8%. GONATURELLE Vertical AI Reliability Index.Agent Eval GuideTesting multi-turn agents end to end.
  1. 04Deterministic checks first: schemas, bounds and provenance
  2. 05Trajectory assertions: order, required and forbidden steps
  3. 06Multi-turn tests need a user, an end state and a seed
  4. 07pass@k versus pass^k: measure consistency, not luck
  5. 08LLM judges: calibrate before you trust
Failure Modes Catalog155 ways regulated agents fail, each written as a test.
dose, citation, PHI, adverse action
32.8% of catalogued multi-step failures were tool-argument schema or boundary violations. GONATURELLE Failure Modes Catalog analysis, 2026.Glossary67 terms engineers, clinicians and compliance teams share.
Cohen's kappapass^kTrajectoryAnnex IVPCCPAdverse action noticeGwet's AC1

Regulatory testing guides

Show all 20
  1. 1EU AI ActEuropean Union
  2. 2FDA SaMDUnited States (federal)
  3. 3CFPB adverse actionUnited States (federal)
  4. 4FINRA 2111 & Reg BIUnited States (federal)
  5. 5HIPAAUnited States (federal)
  6. 6FTC Section 5United States (federal)

Failure modes

Open catalog
  1. 1Negative medication interaction omissionHealthcareCritical
  2. 2Adverse action reason misalignmentFinTechCritical
  3. 3Fabricated case citationLegalCritical
  4. 4Prompt injection via retrieved documentCross-verticalCritical

From the blog

All posts
Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the index
Monthly newsletter

The Regulated Agent

A monthly deep dive on AI evaluations, safety benchmarks and regulatory enforcement. One email, first Tuesday of the month.

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo