The 2026 Vertical AI Reliability Index
48 models, 12 regulated task domains, four metrics that predict production failure. Free to cite.
- 1Reasoning closed Aschema 92.3 · trajectory 61.277.1
- 2Frontier closed Aschema 92.6 · trajectory 58.276.6
- 3Reasoning closed Bschema 90.2 · trajectory 59.675.0
- 4Frontier closed Bschema 92.5 · trajectory 54.574.4
- 5Reasoning closed Cschema 92.4 · trajectory 53.574.1
Models write valid tool calls. They do not finish regulated tasks reliably.
- Median schema adherence across all 48 models: most individual tool calls are well formed.
- 87.4%
- Median schema adherence across all 48 models: most individual tool calls are well formed.
- GONATURELLE Vertical AI Reliability Index, November 2026 release
- Median trajectory completion (pass^3): the same multi-turn task finished correctly in all three trials.
- 31.8%
- Median trajectory completion (pass^3): the same multi-turn task finished correctly in all three trials.
- GONATURELLE Vertical AI Reliability Index, November 2026 release
- Best trajectory completion of any model. No model clears two thirds of regulated multi-turn tasks three times in a row.
- 61.2%
- Best trajectory completion of any model. No model clears two thirds of regulated multi-turn tasks three times in a row.
- GONATURELLE Vertical AI Reliability Index, November 2026 release
- pass^8 for the best function-calling agent on tau-bench retail, the public result the Index's pass^k design follows.
- under 25%
- pass^8 for the best function-calling agent on tau-bench retail, the public result the Index's pass^k design follows.
- Yao et al., tau-bench, arXiv:2406.12045 (2024)
Leaderboard
Filter by vertical, task domain and model class, then rank by any of the four metrics. Select a model for its per-domain breakdown.
| # | Model | Schema | Trajectory | Factuality | Robustness | Comp. |
|---|---|---|---|---|---|---|
| 1 | 92.3 | 61.2 | 82.5 | 72.4 | 77.1 | |
| 2 | 92.6 | 58.2 | 80.3 | 75.2 | 76.6 | |
| 3 | 90.2 | 59.6 | 81.8 | 68.4 | 75.0 | |
| 4 | 92.5 | 54.5 | 80.8 | 69.7 | 74.4 | |
| 5 | 92.4 | 53.5 | 80.0 | 70.8 | 74.1 | |
| 6 | 92.5 | 52.8 | 78.2 | 71.6 | 73.8 | |
| 7 | 91.3 | 50.3 | 76.2 | 75.6 | 73.3 | |
| 8 | 91.2 | 50.9 | 78.6 | 66.0 | 71.7 | |
| 9 | 88.9 | 52.4 | 78.6 | 66.0 | 71.4 | |
| 10 | 89.8 | 46.8 | 76.3 | 64.6 | 69.4 | |
| 11 | 89.6 | 48.6 | 75.9 | 61.4 | 68.9 | |
| 12 | 90.1 | 44.3 | 73.0 | 66.8 | 68.6 | |
| 13 | 88.0 | 47.6 | 78.5 | 60.1 | 68.5 | |
| 14 | 89.6 | 46.0 | 73.8 | 63.5 | 68.2 | |
| 15 | 90.1 | 41.2 | 74.0 | 63.3 | 67.1 |
What the November release shows
- A 55.6-point gap between valid calls and finished tasksMedian schema adherence sits near 87 while median pass^3 trajectory completion sits near 32. Final-answer and per-call metrics overstate how often an agent completes a regulated workflow end to end.
- Case Law Research is the hardest domainMedian trajectory completion of 29.3% across all models. Most failures are citations that no retrieval step returned and superseded authority cited as current.
- Domain fine-tunes gain about 6 factuality points at home70B clinical, finance and legal fine-tunes beat a general 70B instruct model on factuality in their own vertical, then lose ground on schema adherence and on every other vertical.
- Robustness tracks size less than you would expectSeveral mid-size closed models preserve behaviour under perturbation better than larger open-weight models. Prompt injection through retrieved documents causes the largest share of robustness failures.
Twelve regulated task domains
Each domain is a frozen suite with its own expert panel, rubric and agreement statistics.
Methodology
Designed by Karthik Reddy’s judge calibration team with Dr. Meera Iyer’s clinical panels and Nandini Joshi’s regulatory science group. Every number on this page can be traced to a frozen case set, a rubric version and a signed run manifest.
How judges are calibrated01Expert-labelled cases, double annotated
14,060 scored cases and 1,980 multi-turn tasks, each drafted from de-identified real patterns or regulatory edge-traps and labelled independently by two credentialed reviewers. Disagreements go to a third senior reviewer for adjudication. Cohen's kappa before adjudication is published per domain below.
02Deterministic checks first
Schema adherence and trajectory completion are scored without any judge: JSON schema validation, parameter bounds, required and forbidden calls, call order, loop limits and end-state comparison. Every model runs each multi-turn task three times at temperature 0 (or the provider's lowest setting) and must pass all three.
03Calibrated judge ensemble for factuality
Domain factuality uses a weighted ensemble of rules, a specialized grader and two frontier judges from different model families. Responses are order-swapped and length-normalized. The ensemble is calibrated against the adjudicated expert label on a held-out calibration split; judge kappa per domain is published below.
04Anonymized, then vendor review
Models appear by class (frontier closed, reasoning, open-weight by size, domain fine-tune). Vendor names are withheld pending vendor review: each provider receives its own rows and failing traces 30 days before any name is attached, and may submit a corrected configuration.
05Monthly refresh, frozen versions
The Index refreshes on the first of each month. Each release is frozen and versioned (this is 2026.11); case sets rotate 15% per quarter to limit contamination, and retired cases are published so anyone can audit drift. Next refresh: 1 December 2026.
Sample sizes and agreement by domain
Annotator kappa is Cohen’s kappa between the two independent expert labels before adjudication. Judge kappa compares the calibrated ensemble with the adjudicated label on the held-out split. On the Landis and Koch scale, 0.61 to 0.80 is substantial and 0.81 to 1.00 almost perfect.
| Domain | Vertical | Cases | Tasks | Expert panel | Annotator κ | Judge κ |
|---|---|---|---|---|---|---|
| Clinical Pharmacology | Healthcare | 1,420 | 180 | Clinical pharmacists, internists | 0.82 | 0.88 |
| Medical Coding | Healthcare | 1,860 | 140 | Certified professional coders (CPC), physicians | 0.79 | 0.86 |
| Prior Authorization | Healthcare | 1,240 | 220 | Utilization review physicians, nurses | 0.81 | 0.89 |
| Clinical Triage | Healthcare | 980 | 160 | Emergency physicians, triage nurses | 0.84 | 0.90 |
| Credit Decisioning | FinTech | 1,310 | 200 | Credit risk analysts, fair-lending counsel | 0.83 | 0.87 |
| Wealth Suitability | FinTech | 760 | 120 | Registered representatives, former FINRA examiners | 0.78 | 0.85 |
| KYC & AML | FinTech | 1,150 | 190 | AML investigators, CAMS-certified analysts | 0.80 | 0.86 |
| Contract Analysis | Legal | 1,090 | 110 | Commercial attorneys (JD) | 0.77 | 0.84 |
| Case Law Research | Legal | 1,530 | 130 | Litigation attorneys, law librarians | 0.85 | 0.91 |
| Insurance FNOL | Insurance | 870 | 170 | Claims adjusters, coverage counsel | 0.81 | 0.87 |
| Claims Adjudication | Insurance | 1,020 | 210 | Senior adjusters, actuaries | 0.79 | 0.86 |
| Pharmacovigilance | Life Sciences | 830 | 150 | Drug safety physicians, PV scientists | 0.80 | 0.88 |
| All domains | 5 verticals | 14,060 | 1,980 | Credentialed reviewers only | 0.81 | 0.87 |
Source: GONATURELLE Vertical AI Reliability Index, November 2026 release. Kappa bands: Landis & Koch, Biometrics 33(1), 1977.
Four metrics, four scoring rubrics
- Deterministic Schema Adherence% of tool calls
- Share of tool calls whose arguments validate against the domain JSON schema and every parameter bound (formulary dose ranges, rate tables, policy limits, enumerated codes).
- Binary per call, no judge involved. A call fails on a type error, a missing required field, an out-of-range value, or an argument value that no upstream step produced (hallucinated argument).
- Trajectory Completion Ratepass^3, % of tasks
- Share of multi-turn tasks completed in all three independent trials with every sequence constraint satisfied (for example verify_identity before fetch_account_balance) and the expected end state reached.
- Deterministic trajectory assertions on order, required steps, forbidden calls, loop limits and final database state. Reported as pass^3, so one flaky trial out of three fails the task.
- Domain Factuality Score0 to 100
- Rubric score for clinical, financial or legal correctness of the final output: contraindications named, reasons for adverse action specific and accurate, citations real and on point.
- Scored by a calibrated judge ensemble (rules, a specialized grader and two cross-family frontier judges, order-swapped and length-normalized) against credentialed expert rubrics. A single critical error caps the case at 40.
- Adversarial Robustness Index% behaviour preserved
- Share of perturbed variants (typos, code-mixed phrasing, hostile tone, distracting history, instructions injected through retrieved documents) on which the model keeps the behaviour it showed on the clean case.
- Each clean case expands into 12 perturbation variants. A variant counts as preserved when the trajectory assertions and the factuality verdict match the clean run; any executed injected instruction fails it.
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
suite = EvalSuite.load('index/prior-authorization-2026.11')
results = suite.evaluate(agent=candidate, trials=3, max_concurrency=16)
# Deterministic Schema Adherence: every tool call, no judge
dsa = results.tool_calls.share(lambda c: c.schema_valid and c.params_in_bounds)
# Trajectory Completion Rate: pass^3 over multi-turn tasks
t = TrajectoryTester(results)
t.assert_order('verify_member', 'fetch_benefits', 'submit_prior_auth')
t.assert_max_calls(per_turn=6)
tcr = t.pass_hat_k(k=3)
# Domain Factuality Score: calibrated ensemble, critical errors cap at 40
dfs = results.judge(rubric='prior-auth-physician-v3', ensemble='calibrated').mean(cap_on_critical=40)
# Adversarial Robustness Index: behaviour preserved on 12 perturbations per case
ari = results.perturbations.preserved_share()The composite is the unweighted mean of the four metrics. It is a ranking convenience, not a fifth measurement: teams in claims or coding usually weight schema adherence higher, clinical and legal teams weight factuality.
Free to cite under CC BY 4.0
Use the numbers in papers, model-risk memos, board decks and newsletters. Attribute GONATURELLE and the release version so readers can find the frozen data. Press and research questions go to press@gonaturelle.si.
CC BY 4.0 licenseGONATURELLE PRIVATE LIMITED (2026). 2026 Vertical AI Reliability Index (Version 2026.11, November 2026 release) [Data set]. https://gonaturelle.si/resources/vertical-benchmarks
@misc{gonaturelle2026index,
title = {2026 Vertical AI Reliability Index},
author = {{GONATURELLE PRIVATE LIMITED}},
year = {2026},
month = nov,
version = {2026.11},
howpublished = {\url{https://gonaturelle.si/resources/vertical-benchmarks}},
note = {48 models, 12 regulated task domains. CC BY 4.0}
}Frequently asked questions
Why are the models anonymized?
Vendor names are withheld pending vendor review. Providers see their own rows and failing traces before any name is attached and can submit a corrected configuration (system prompt format, tool-choice settings). Publishing names first would reward whichever vendor answers email fastest. Model class, access type, parameter count and context window are disclosed so teams can still reason about the trade-offs.
Can I cite the Index in a paper, a newsletter or a model-risk memo?
Yes. The Index and its methodology are published under Creative Commons Attribution 4.0 (CC BY 4.0). Cite the release version so readers can find the frozen numbers, for example "GONATURELLE Vertical AI Reliability Index, version 2026.11". The citation block above has APA and BibTeX formats.
Why is Trajectory Completion so much lower than Schema Adherence?
Schema adherence is scored per tool call; trajectory completion is scored per task across three trials, and every step must be in order with the right end state. A model can emit perfectly valid JSON on 90% of calls and still fail most tasks because it skipped a verification step once, looped, or stopped early in one of the three runs. This mirrors the gap tau-bench reported between single-trial success and pass^k.
Does a high score mean a model is safe to deploy in my product?
No. The Index measures base-model behaviour on standardized tasks with a reference tool set. Your agent adds prompts, retrieval, tools and guardrails that change the result in both directions. Use the Index to shortlist, then run your own agent against your own golden suite. The Index is not a certification, regulatory approval or clinical validation.
How do you keep the test cases out of training data?
The case sets are not published in full. 15% of cases rotate each quarter, retired cases are released with their labels, and every release includes a canary string so contamination can be detected. Scores on retired versus live cases are compared each month; a gap above three points triggers a review of that model's row.
Can we run the Index suites against our own agent?
Yes. Every domain is available as an GONATURELLE suite (index/<domain>-2026.11) on the Scale and Enterprise plans, so your agent is scored with the same assertions, judges and perturbations as the published rows.
The Regulated Agent
A monthly deep dive on AI evaluations, safety benchmarks and regulatory enforcement. One email, first Tuesday of the month.
Score your own agent on the Index suites.
Every Index domain ships as an GONATURELLE suite. Run your agent with the same assertions, judges and perturbations. 14-day free production trial, no card.