ML Research Engineer, Judge Calibration

About the role

An uncalibrated single-prompt judge reached a Cohen's kappa of 0.42 against expert consensus in our 2026 calibration study; our calibrated ensembles reached 0.87 on the same 6,400 double-annotated cases. That gap is the product. Closing it for each new rubric, each new vertical and each new judge model is this team's job.

You will work with Karthik Reddy's research group on the calibration engine: rubric decomposition, ensemble weighting across deterministic rules, small specialized models and frontier models, bias controls for position, verbosity and self-preference, and the agreement statistics (kappa, Gwet's AC1, per-class precision and recall) we report for every judge. You will also contribute to the Vertical AI Reliability Index methodology, which means writing code that other researchers will read critically.

The work sits between research and production. A result only counts when it survives contact with a customer's rubric, ships behind a version number, and still holds when the underlying judge model is upgraded by its provider.

What you will do

• Run calibration experiments against expert-annotated golden sets and turn the winners into versioned judge configurations
• Build and maintain bias probes: order swapping, length normalization, cross-family judging, and adversarial rubric attacks
• Detect judge drift when a provider updates a model, and design the re-calibration workflow customers see
• Contribute to the Reliability Index methodology and its public write-ups
• Pair with the clinical and regulatory teams to turn expert disagreement into better rubrics rather than noise

What you bring

• Master's or PhD in machine learning, statistics or a related field, or equivalent research experience
• Fluency with agreement statistics and their failure modes (prevalence and bias effects on kappa, when AC1 is preferable)
• Strong Python and experience running careful experiments with LLMs, including prompt and sampling controls
• Clear technical writing; you can explain a confidence interval to a compliance officer

Nice to have

• Published work on annotation, crowdsourcing, preference modelling or LLM-as-a-judge
• Experience fine-tuning small models for classification or reward modelling
• Background in a regulated domain such as medicine, law or credit