LLM-as-a-Judge calibrated to human expert consensus.
Every judge ships with its agreement against the experts who labelled the pack. Pick a judge and compare.
- Cohen’s κ vs MD panel
- 0.87almost perfect
- Gwet's AC1
- 0.90
- Precision
- 0.93
- Recall
- 0.91
- Gate eligible
- yes
General agreement is not domain agreement.
Strong LLM judges agree with crowd preferences on open-ended chat about as often as people agree with each other. That result does not transfer to whether a denial letter cites the right coverage criterion. On domain rubrics, an uncalibrated single-prompt judge lands in the moderate band.
- Agreement between a strong LLM judge and human preferences on open-ended chat, matching human-to-human agreement, alongside documented position, verbosity and self-enhancement biases.
- over 80%
- Agreement between a strong LLM judge and human preferences on open-ended chat, matching human-to-human agreement, alongside documented position, verbosity and self-enhancement biases.
- Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023
- Cohen's kappa against physician, attorney and analyst consensus: calibrated GONATURELLE judge ensembles versus an uncalibrated single-prompt judge.
- 0.87 vs 0.42
- Cohen's kappa against physician, attorney and analyst consensus: calibrated GONATURELLE judge ensembles versus an uncalibrated single-prompt judge.
- GONATURELLE calibration study, 2026 (6,400 double-annotated cases)
- Cohen's kappa band conventionally read as "almost perfect" agreement; 0.61 to 0.80 is "substantial".
- 0.81 to 1.00
- Cohen's kappa band conventionally read as "almost perfect" agreement; 0.61 to 0.80 is "substantial".
- Landis & Koch, Biometrics 33(1), 1977
- Position biasPairwise judges favour whichever answer appears first or second. GONATURELLE scores every pairwise comparison in both orders and counts a preference only when it survives the swap.
- Verbosity biasLonger answers look more thorough. Rubric items are scored on the presence of required elements, and holistic scores are length-normalized against the reference answer.
- Self-preference biasA judge tends to prefer outputs from its own model family. Ensembles always include a judge from a different family than the agent under test, and same-family judges are down-weighted.
The calibration engine.
Domain rubrics tuned against thousands of expert annotations, with agreement measured on data the judge never saw. Karthik Reddy leads judge calibration research.
- 01
Write the rubric with the reviewers
Rubric items mirror how the expert decides: the medical necessity criterion that applies, whether each adverse action reason is specific and accurate, whether every citation exists and supports the proposition.
- 02
Collect expert labels on a calibration set
Two credentialed reviewers label each calibration case independently, disagreements are adjudicated, and the set is split into fit and hold-out partitions stratified by edge-case category.
- 03
Fit judge prompts and ensemble weights
Judge prompts, thresholds and ensemble weights are tuned on the fit partition only. Agreement is then measured once on the hold-out set, which is the number we publish.
- 04
Monitor drift and re-calibrate
A slice of live evaluation results is re-labelled by experts each month. If a judge drops below its kappa floor, it is removed from gating until it is re-fit.
Agreement metrics for every judge, on every pack.
Judge report for healthcare/clinical-prior-auth-v2 2.3.0, measured on a 1,200-case physician-labelled hold-out set. Each pack page carries its own report.
| Judge | Type | Cohen’s κ | Gwet’s AC1 | Precision | Recall | Notes |
|---|---|---|---|---|---|---|
| rules.lcd-criteria | Deterministic | 0.93 | 0.95 | 0.97 | 0.88 | Covers 6 of 11 rubric items |
| slm.utilization-review-8b | Specialized SLM | 0.71 | 0.76 | 0.86 | 0.81 | Fast; weighted 0.25 |
| frontier.cross-family | Frontier model | 0.78 | 0.82 | 0.88 | 0.86 | Order-swapped pairwise |
| single-prompt baseline | Uncalibrated | 0.42 | 0.51 | 0.71 | 0.62 | Not gate eligible |
| ensemble.prior-auth.v3 | Weighted ensemble | 0.87 | 0.90 | 0.93 | 0.91 | Gate eligible (floor 0.80) |
Headline ensemble and baseline figures: GONATURELLE calibration study, 2026 (6,400 double-annotated cases).
Multi-judge consensus: rules first, models second, weights from measured reliability.
Deterministic rules settle what can be settled exactly. Specialized small models score rubric items cheaply. Frontier judges from a different model family handle open-ended items. Weights come from hold-out agreement, not intuition.
- 0.40rules weight
- 0.25SLM weight
- 0.35frontier weight
ensemble: prior-auth.v3
rubric: rubrics/utilization-review-v3.yml
kappa_floor: 0.80
judges:
- id: rules.lcd-criteria
type: deterministic
weight: 0.40
- id: slm.utilization-review-8b
type: model
weight: 0.25
- id: frontier.cross-family
type: model
weight: 0.35
exclude_family_of: agent_under_test
pairwise:
swap_order: true
length_normalize: true
aggregate: weighted_mean
threshold: 0.90
on_disagreement: route_to_reviewerFrequently asked questions
Why report Gwet's AC1 as well as Cohen's kappa?
Cohen's kappa can drop sharply when one class dominates, for example when 95% of cases are compliant, even if raters agree almost always. Gwet's AC1 is more stable under that prevalence imbalance. Reporting both, with raw agreement, lets reviewers see whether a low kappa reflects real disagreement or skewed labels.
Which models do you use as judges?
A mix: deterministic rule checks, small specialized models fine-tuned on expert-labelled rubric data, and frontier models from more than one provider. You can restrict judges to providers your organization has approved, or to models served inside your VPC.
Can we calibrate judges on our own expert labels?
Yes. Upload labels from your clinicians, counsel or analysts for a BYOD suite and the calibration engine fits prompts and ensemble weights to your panel. Enterprise plans include expert-curated calibration sets built with our reviewer network.
Does a calibrated judge replace expert review?
No. Calibration means the judge agrees with your experts often enough to triage at scale and gate obvious regressions. Contested cases, new failure categories and deployment decisions still go to qualified humans. GONATURELLE does not replace clinical, legal or actuarial judgment.
Put a number on how far you can trust your judge.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.