Testing for citation hallucination in legal AI

The failure everyone knows, and the one they miss

In Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y.), attorneys filed an opposition brief citing judicial decisions that did not exist, generated by a chatbot, and then defended them when the court and opposing counsel could not locate them. On 22 June 2023 the court imposed $5,000 in sanctions. The case became shorthand for AI risk in law, and most legal AI teams now test, at least informally, whether their product invents cases.

Invented cases are the easy failure to detect, because a citation that resolves to nothing fails a lookup. The harder failure is a real case cited for something it does not say. Magesh et al. at Stanford RegLab and HAI ("Hallucination-Free?", 2024) evaluated leading AI legal research tools built on retrieval and found they hallucinated between 17% and 33% of the time on their preregistered query set. Their definition counted responses that were incorrect and responses that were misgrounded, where the cited source does not support the claim made. For a tool built to retrieve real authority, misgrounding is the dominant risk, and no existence check will catch it.

Five separate checks, not one score

A single "citation accuracy" score blends failures with very different causes and fixes. We decompose citation testing into five checks, each with its own assertion and its own reported rate, so that a regression points at the step that broke.

• Existence: the citation resolves to a real authority in a citator or the document store, with the right reporter, volume and page.
• Grounding: the authority was actually retrieved in this trajectory. A real case the model recalled from training, never retrieved, is ungrounded even if it is correct.
• Proposition support: the passage of the authority the agent relies on actually supports the proposition it is cited for.
• Quotation accuracy: any quoted language appears verbatim in the source, with the correct pin cite.
• Validity and jurisdiction: the authority has not been reversed or overruled on the relevant point, and it is binding or appropriately characterized as persuasive in the forum.

The deterministic layer

Existence, grounding and quotation can be checked without any model. Grounding is the most valuable of the three and the most often skipped. If the agent's trajectory shows that it retrieved documents with identifiers A, B and C, then every citation in the draft must resolve to A, B or C. A citation to D means the model wrote from memory, and memory is where fabricated and misremembered authority comes from. This one rule catches most Mata-style failures and a large share of subtler ones, and it runs in milliseconds.

Quotation checks are similarly mechanical: normalize whitespace and typographic quotes, then require the quoted span to appear in the retrieved text at the cited page. Paraphrases presented as quotations fail. So do quotations stitched together from two non-adjacent passages, which models produce more often than reviewers expect.

from gonaturelle import EvalSuite, assertions as A

from agents.research import research_agent


def test_memo_citations():
    suite = EvalSuite.load('legal/citation-verification')
    results = suite.evaluate(agent=research_agent, max_concurrency=8)

    results.assert_all([
        A.cites_only(source='retrieve_opinion.doc_id'),
        A.citations_resolve(citator='store', fields=['reporter', 'volume', 'page']),
        A.quotes_verbatim(source='retrieve_opinion.text', pin_cites=True),
        A.no_negative_treatment(source='check_citator.treatment'),
        A.judge(rubric='legal/holding-accuracy', threshold=0.93),
    ])
    results.export_report('reports/citation_run.pdf', group_by='check')

Proposition support needs a calibrated judge

Whether a passage supports a proposition is a legal judgment, and it is where a judge model earns its place. It is also where an uncalibrated judge does real damage, because a fluent model will often agree that a vaguely related holding supports a claim. We build the proposition rubric with practising attorneys: each golden case pairs a proposition with the cited passage, two attorneys label support independently, disagreements are adjudicated, and the judge ensemble is calibrated against the adjudicated labels with kappa and fail-class recall reported.

The rubric has to be specific about the distinctions attorneys care about. Holding versus dicta. A rule stated by the majority versus a concurrence or dissent. A case that states the rule but applies an exception on its facts. A statute section cited for language that appears in a different subsection. Written that way, attorneys agree with each other at high rates, and a calibrated judge can approach that agreement. Written as "does the source support the claim", agreement collapses and so does the judge.

Building the golden set

Legal golden sets need traps that mirror how misgrounding happens. We include questions where the leading case was overruled in part, so a model that relies on the older holding fails. Questions where the best-known case comes from another circuit, so jurisdictional characterization matters. Questions whose honest answer is that the law is unsettled, where confident synthesis is the failure. And questions with a near-miss authority: a real case with a similar caption or a similar fact pattern that stands for something else. Synthetic expansion varies the jurisdiction, date and phrasing around each trap to produce many test variants from one carefully drafted case.

Keep the retrieval corpus versioned with the golden set. Validity is time-dependent: a case good last year can be reversed this year. If the corpus and the expectations drift apart, you will see failures that are really stale expectations, and the team will learn to ignore the gate.

What testing does not change

Courts and bar regulators have been clear that the duty to verify authorities belongs to the lawyer who signs the filing, whatever tool produced the draft. A low hallucination rate on a test set is evidence that the product is less likely to mislead its users, which matters for product liability and for the trust of the firms that buy it. It is not a substitute for the attorney reading the case. Products should be designed so that verification is easy: every citation linked to the retrieved passage, quotations highlighted in context, negative treatment surfaced rather than hidden.

This note is not legal advice on professional responsibility obligations in any jurisdiction. If you build legal research or drafting agents and want to measure your own grounding and proposition-support rates, start a 14-day trial with the legal citation verification pack, or book a technical demo to walk through the rubric with our team.