Calibrating LLM judges with Cohen's kappa

Why "the judge agrees with our doctors 90% of the time" is not enough

LLM-as-a-judge is now the default way to grade free text at scale, and for good reason. Zheng et al. ("Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023) showed that strong judges reach over 80% agreement with human preferences on open-ended chat, roughly the level at which humans agree with each other. That result is the reason most teams trust a judge at all. It is also routinely misapplied, because the setting in that paper (pairwise preferences on general chat) is not the setting of a clinical or credit rubric, where most outputs are acceptable and the cases that matter are the rare failures.

Suppose two raters, a physician and a judge model, each label 1,000 prior authorization rationales as pass or fail. They agree on 900. That sounds strong. Now suppose both of them mark about 85% of cases as pass. If each rater simply labelled at random with that 85% prevalence, they would still agree on 0.85 x 0.85 + 0.15 x 0.15 = 0.745 of cases. The judge's 90% agreement is only modestly better than a coin weighted to the base rate.

Cohen's kappa in one line

Cohen's kappa measures agreement beyond chance: kappa = (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is the agreement expected from each rater's label distribution. With the numbers above, kappa = (0.90 - 0.745) / (1 - 0.745), which is about 0.61. On the Landis and Koch (1977) scale that sits right at the boundary between moderate and substantial (0.61 to 0.80 is substantial, 0.81 to 1.00 almost perfect). That is a very different story from "90% agreement".

For ordinal rubrics, such as a 1 to 5 clinical completeness score, use a weighted kappa so that a judge scoring 4 against an expert's 5 is penalized less than a judge scoring 1. Quadratic weights are the common choice. Whatever you choose, state it, because weighted and unweighted kappa on the same data can differ by a lot.

from sklearn.metrics import cohen_kappa_score, precision_score, recall_score


def agreement_report(expert: list[int], judge: list[int], ordinal: bool = False) -> dict:
    weights = 'quadratic' if ordinal else None
    kappa = cohen_kappa_score(expert, judge, weights=weights)
    report = {'n': len(expert), 'kappa': round(kappa, 3)}
    if not ordinal:
        # Treat 'fail' (0) as the positive class: we care about catching failures.
        report['fail_precision'] = precision_score(expert, judge, pos_label=0)
        report['fail_recall'] = recall_score(expert, judge, pos_label=0)
        report['expert_pass_rate'] = sum(expert) / len(expert)
    return report

The kappa paradox and Gwet's AC1

Kappa has a well-known weakness, sometimes called the kappa paradox (Feinstein and Cicchetti, 1990): when prevalence is extreme, p_e approaches 1 and kappa can be low or even negative while raw agreement is very high. Regulated eval sets often look exactly like this. A mature clinical agent might pass 97% of cases, and the judge's job is to find the other 3%. Kappa on that set can swing widely with a handful of disagreements.

Gwet's AC1 estimates chance agreement differently and stays stable under skewed prevalence. For a binary label, AC1 uses p_e = 2 x pi x (1 - pi), where pi is the average proportion of passes across both raters. In the earlier example, pi = 0.85 gives p_e = 0.255 and AC1 = (0.90 - 0.255) / (1 - 0.255), about 0.87. We publish both statistics for every pack, with prevalence, because each one alone can mislead. If they disagree sharply, look at the confusion matrix before trusting either.

Agreement is also not the whole story for a gate. A judge that agrees with experts on passes but misses a third of the failures is dangerous even with decent kappa. We report precision and recall on the fail class per judge, and we set gate thresholds on fail recall first.

Measuring and controlling judge bias

Zheng et al. documented three biases that we still see in current judges. Position bias: in pairwise comparisons, the judge prefers whichever answer appears first (or second, depending on the model). Verbosity bias: longer answers win even when the extra length adds nothing. Self-enhancement bias: a judge tends to favour outputs from its own model family. Each one is measurable on your own data, and each one has a cheap control.

• Position: run every pairwise comparison twice with the order swapped, and count a win only when both orders agree. Report the swap-inconsistency rate as a judge health metric.
• Verbosity: include length-matched pairs in the calibration set, normalize scores by length bucket, and write rubrics that reward stated criteria rather than thoroughness.
• Self-preference: never let a judge grade outputs from its own model family alone. Use a cross-family ensemble, and check per-family score gaps on a fixed set of expert-labelled cases.

What calibration actually involves

Calibration is not prompt tuning until the numbers look good. It is a fitting procedure with a held-out test. We start from a double-annotated set: each case labelled independently by two credentialed reviewers, disagreements adjudicated, and expert-to-expert kappa recorded as the ceiling. We then fit the ensemble (deterministic rules, small specialized models and frontier models, weighted) on one split and report agreement on a holdout the weights never saw.

On 6,400 double-annotated cases across our packs, a calibrated ensemble reached kappa 0.87 against adjudicated expert labels, while an uncalibrated single-prompt judge on the same cases reached 0.42 (GONATURELLE calibration study, 2026). Most of the gain did not come from a bigger model. It came from splitting a vague rubric into checkable criteria, routing the deterministic ones to rules, and weighting each judge by its measured recall on the failures that matter for that rubric.

from gonaturelle import JudgeEnsemble, EvalSuite

suite = EvalSuite.load('healthcare/clinical-prior-auth-v2')
labels = suite.expert_labels(rubric='medical-necessity-v3', adjudicated=True)

ensemble = JudgeEnsemble.from_config('judges/medical_necessity.yaml')
fit = ensemble.calibrate(labels, holdout=0.3, objective='fail_recall', swap_order=True)

print(fit.report(metrics=['kappa', 'gwet_ac1', 'fail_precision', 'fail_recall']))
fit.assert_min(kappa=0.80, fail_recall=0.95)
fit.save('judges/medical_necessity.calibrated.json')

Keeping a calibrated judge calibrated

A judge's agreement with experts drifts. The judge model gets updated by its provider, your agent's outputs change style, and new failure modes appear that the calibration set never contained. We treat calibration as a versioned artifact tied to a specific judge model version and rubric version, and we re-check it on a rolling sample of fresh expert labels every release cycle. When kappa on the fresh sample drops below the published band, the gate stays on but its verdicts on that rubric are flagged for human review until recalibration.

Two practical rules round this out. First, publish expert-to-expert agreement next to judge-to-expert agreement; a judge cannot be held to a higher standard than the reviewers who define the truth. Second, never use a judge where a rule will do. If "the letter states the member ID" can be checked with a string match against the trajectory, it should be. Judges are for the criteria that genuinely need reading comprehension, and those are the ones that deserve the calibration effort.