Why MMLU scores don't protect regulated agents

A benchmark answers someone else's question

Every time a new model ships, the first number in the launch post is a benchmark score. MMLU (Hendrycks et al., 2021) is a set of multiple-choice questions spread across 57 subjects, from clinical knowledge and professional law to high-school chemistry. GSM8K (Cobbe et al., 2021) is a set of grade-school math word problems that test multi-step arithmetic reasoning. Both were built to answer a research question: how much broad knowledge and reasoning does a language model carry? They answer it reasonably well, and they were never designed to answer the question a head of AI at a health plan or a lender has to sign off on.

That question is narrower and harder: when this specific agent, with this system prompt, these tools and this retrieval index, handles a prior authorization request or an adverse action notice, does it follow the process the regulator, the payer contract or the bank's policy requires? A model can know that warfarin interacts with many antibiotics, answer the MMLU clinical question correctly, and still let an agent approve a medication request without ever calling the interaction checker. Knowledge in the weights and correct behaviour in a workflow are different properties.

What a multiple-choice format cannot see

A multiple-choice item has four options and one key. The model reads a self-contained stem and emits a letter. Nothing in that format exercises the things that break agents in production. There is no tool to call, so there is no tool-call schema to violate. There is no sequence, so the model cannot fetch an account balance before verifying identity. There is no state, so it cannot write a claim reserve twice. There is no adversarial document in the context, so it cannot be steered by an instruction hidden in a faxed referral.

GSM8K adds multi-step reasoning, but the steps live inside a single response and are graded only by the final number. That grading choice is exactly the blind spot that hurts agents. When we ran our own regression analysis across production agents (GONATURELLE regression study, 2026), 41% of prompt-only pull requests changed at least one tool-call trajectory while final-answer scores stayed flat across 1,180 pull requests on 14 agents. A final-answer metric would have reported those changes as no change at all.

• No tool calls, so no argument validation, schema conformance or hallucinated parameters.
• No ordering, so no check that verification precedes disclosure or that eligibility precedes a benefits quote.
• No refusal grading: the benchmark rewards an answer even where the correct behaviour is to escalate to a human.
• No domain policy: payer medical policy, Regulation B reason codes and state claim-handling rules are not in the item bank.

Saturation and contamination make the number less informative every year

Two further problems erode what a public score tells you. The first is saturation: once frontier models cluster near the top of a benchmark, the remaining differences are inside the noise of prompt format and sampling settings, and the leaderboard stops separating candidates. The second is contamination. Benchmarks are public text, and public text ends up in pretraining corpora. A high score can reflect memorization of items, or of close paraphrases, rather than a transferable skill. Benchmark authors and model labs try to control for this, and the community has built held-out and refreshed variants, but no regulated team should treat a public score as evidence about unseen cases.

Your golden set is different by construction. It is private, versioned, drafted from de-identified patterns in your own traffic and your own regulatory edge-traps, and it changes when your policies change. It cannot leak into a model's pretraining data before you write it, and when it is refreshed you know exactly which cases moved.

The failure that actually ships

Here is the shape of a failure we see repeatedly in prior authorization agents. The team upgrades the underlying model to one with a better public score. Answers to the benchmark questions the team spot-checks look sharper. Two weeks later a utilization management nurse notices the agent has started pre-filling the CPT code from the referring note instead of calling the code-validation tool, because the new model is confident enough to skip the step. The codes are usually right. When they are wrong, the request goes to the payer with a procedure code nobody verified, and the denial lands days later.

Nothing about that failure is visible to MMLU, MedQA-style question sets or any final-answer metric run on the request summary. It is visible to a trajectory assertion that says the code passed to submit_request must come from the output of validate_cpt, and to a regression gate that compares the new model's trajectories against the baseline on the same golden cases before the upgrade merges.

What a vertical suite measures instead

A vertical suite starts from the workflow and the liability, not from a list of subjects. For a clinical prior authorization agent that means cases with contraindications buried in the medication list, members whose eligibility lapsed last month, step-therapy requirements the agent must check before recommending approval, and requests where the correct output is a handoff to a clinician. Each case carries deterministic assertions on the trajectory plus a judge rubric for the free-text rationale, calibrated against physician annotations.

The code below is close to what a team runs in its first week. It loads a versioned pack, runs the agent, and fails the build on any contraindication miss, any malformed tool call or any out-of-order step.

import gonaturelle
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

from agents.prior_auth import build_agent


@gonaturelle.trace(agent='prior-auth')
def my_clinical_agent(case):
    return build_agent(model='candidate').run(case.input)


def test_candidate_model_against_vertical_pack():
    suite = EvalSuite.load('healthcare/clinical-prior-auth-v2')
    results = suite.evaluate(agent=my_clinical_agent, max_concurrency=10)

    results.assert_zero_contraindications()
    results.assert_tool_schema_valid()
    TrajectoryTester(results).assert_order('verify_member', 'fetch_benefits')
    results.assert_all([
        A.no_hallucinated_args(tool='submit_request', field='cpt_code'),
        A.judge(rubric='medical-necessity-v3', threshold=0.90),
    ])

Use public benchmarks for what they are good at

None of this means ignoring public scores. They are a cheap first filter: a model that struggles on clinical knowledge questions is unlikely to become a good clinical agent, and a model that fails grade-school arithmetic should not be computing a premium. Use them to narrow five candidates to two. Then run both through your own suite, compare trajectories case by case, and let the vertical results decide.

The deeper point is about where evidence comes from. A compliance officer, a model risk validator or a notified body reviewer will ask how you know this agent behaves correctly for its intended purpose. A leaderboard rank does not answer that, because it was measured on someone else's task. A signed run manifest over your own golden set, with per-pack inter-annotator agreement and a regression history, does. If you want to see what that looks like on your own agent, run a free eval suite during the 14-day trial and compare it with the benchmark numbers you were using before.