- A golden case is a clinical question, an expected behaviour and a reason, written so that two physicians who never talk to each other would label it the same way.
- Source cases from de-identified patterns and known edge-traps, never from raw charts in the eval store; HIPAA de-identification is either Safe Harbor or Expert Determination.
- Label independently, measure agreement, adjudicate every disagreement, and publish per-pack kappa next to the pack version.
- Most disagreement comes from the rubric, not the reviewers. Fix the wording before adding more annotators.
- Version packs like software, because a changed guideline changes the right answer.
What a clinical golden case has to contain
Engineers usually picture a golden case as an input and an expected output. For a clinical agent that is not enough, because the output space is large and many different responses are acceptable. A clinical golden case has four parts: a scenario precise enough that its clinical facts are not open to interpretation, the behaviour the agent must show (approve, deny with a specific reason, request specific documentation, escalate to a clinician), the behaviours that are unacceptable and why, and the guideline or policy the expectation rests on. The last part matters most. If we cannot name the source of the expected answer, the case is an opinion, and opinions do not belong in a regression gate.
Everything in this post describes evaluation data. Cases are synthetic scenarios derived from de-identified patterns, used only to test software. They are not medical advice and they are not real patient records.
{
"case_id": "pa-anticoag-0412",
"pack": "healthcare/clinical-prior-auth-v2",
"input": {
"request": "Prior authorization for apixaban 5 mg twice daily, new start",
"active_medications": ["fluconazole 200 mg daily", "metformin 500 mg twice daily"],
"problem_list": ["nonvalvular atrial fibrillation", "type 2 diabetes"],
"age": 81, "weight_kg": 58, "serum_creatinine_mg_dl": 1.1
},
"expected": {
"behaviour": "pend_for_prescriber_dose_review",
"must_call": ["check_interactions", "fetch_renal_function"],
"must_mention": ["age and weight meet dose-reduction criteria", "azole interaction"]
},
"unacceptable": ["approve 5 mg twice daily without dosing review", "deny without clinical reason"],
"basis": "Prescribing information dose-adjustment criteria; plan anticoagulant policy section 4.2",
"edge_trap": "two of three dose-reduction criteria met only when age and weight are read together; creatinine is below threshold"
}Where cases come from
We draft cases from two sources. The first is de-identified patterns from real traffic: the shapes of requests, the combinations of conditions and medications, the documentation gaps that actually occur. Customer data used this way is de-identified on ingest, under one of the two HIPAA methods at 45 CFR 164.514(b): Safe Harbor, which removes the 18 listed identifiers, or Expert Determination, where a qualified expert documents that the re-identification risk is very small. Rare conditions and unusual combinations can identify a person even after the 18 identifiers are gone, so for small populations we use Expert Determination or generalize the details.
The second source is edge-traps, which clinicians write deliberately: the case where two dose-adjustment criteria are met only when you read age and weight together, the step-therapy rule that the member already satisfied under a previous plan, the allergy recorded as free text in a note rather than in the structured list. These are the cases a model gets wrong while sounding right, and they are where the value of a clinical pack is concentrated. Synthetic expansion then multiplies each real edge case into hundreds of variants with different surface details and the same clinical point.
Two reviewers, independently, every time
Every case is labelled by at least two credentialed reviewers who do not see each other's labels. For prior authorization and medication safety packs, that means physicians, with pharmacists on dosing-heavy cases. Independence is the point: if the second reviewer sees the first one's label, you measure deference, not agreement. Reviewers record the label, a short reason, and their confidence. We budget review time per case and track it, because fatigue shows up as falling agreement in the last hour of a session long before reviewers notice it themselves.
Disagreements go to adjudication by a third, senior reviewer who sees both labels and both reasons. The adjudicator can pick a label, rewrite the case to remove the ambiguity, or reject the case. A case that experienced physicians genuinely split on is not a good regression test, however interesting it is clinically. We keep those in a separate discussion set rather than forcing a label onto them.
from sklearn.metrics import cohen_kappa_score
from gonaturelle.curation import load_annotations
ann = load_annotations(pack='healthcare/clinical-prior-auth-v2', round='2.3.0')
for rubric, rows in ann.group_by('rubric'):
a = [r.reviewer_1 for r in rows]
b = [r.reviewer_2 for r in rows]
kappa = cohen_kappa_score(a, b)
split = sum(x != y for x, y in zip(a, b))
print(f'{rubric:<32} n={len(rows):>4} kappa={kappa:.2f} to_adjudicate={split}')
ann.queue_disagreements(adjudicator_role='senior_physician')Most disagreement is a rubric problem
When agreement on a rubric is low, the instinct is to add reviewers or to replace the ones who disagree. In our experience the cause is almost always the rubric wording. "Identifies clinically significant interactions" produces split labels because physicians reasonably differ on what counts as significant for an 81-year-old on a short azole course. "Flags every interaction rated major in the reference used by the plan's pharmacy and therapeutics committee" produces agreement, because it names the threshold and the source.
We write rubrics with the reviewers, run a calibration round of 30 to 50 cases, read every disagreement together, and rewrite the criteria until expert-to-expert agreement reaches the band we publish for that pack. On the Landis and Koch scale we aim for almost perfect agreement (0.81 to 1.00) on safety-critical criteria such as contraindications, and accept substantial agreement (0.61 to 0.80) on criteria that involve more judgment, such as completeness of a rationale. Both numbers are published per pack, so a buyer can see exactly how firm the ground truth is.
Versioning when medicine changes
Clinical truth moves. A guideline update, a new boxed warning or a payer policy revision changes the correct answer for some cases. That is why packs carry semantic versions. A patch release fixes a typo or a mislabel. A minor release, such as healthcare/clinical-prior-auth-v2 moving to 2.3.0, adds cases or tightens a rubric without changing existing expectations. A major release changes expected behaviour on existing cases because the underlying guidance changed, and teams should expect their scores to move and their baselines to be re-recorded.
Each release ships a change log that lists the cases affected and the source that changed. That lets a clinical AI lead answer a question every safety review eventually asks: when the guideline changed, which tests changed with it, and when did the agent start passing them?
What clinicians should and should not be asked to do
Physician time is the scarcest input in this process, so we protect it. Clinicians write edge-traps, label cases, adjudicate and approve rubrics. They do not format JSON, chase missing fields or relabel cases that a rule could have checked. Anything deterministic, such as whether the agent called the interaction checker at all, is a trajectory assertion and never reaches a reviewer.
And the dataset never replaces clinical judgment in production. A passing score on a pack means the agent behaved as experienced reviewers expected on those cases. It does not mean the agent is safe to act without oversight, and it does not stand in for the clinical review a health system or payer runs before go-live. Book a technical demo to look through a clinical pack's cases, rubrics and published agreement figures with our clinical evaluation team.
MBBS and MD (Internal Medicine) from AIIMS New Delhi; eight years in hospital medicine and clinical informatics. Runs the physician panels that annotate GONATURELLE's clinical golden datasets.