Skip to content
GONATURELLE
Legal · Hallucination

Quotation misattribution

The agent quotes real language from a real source but attributes it to the wrong speaker or the wrong part of the source: a dissent presented as the majority, a lawyer's question presented as the witness's answer. The words check out in a search, which is why this survives review.

Get the assertionBack to catalog
High severityHallucinationLegal
Scenario

An appellate practice group uses an agent to draft statement-of-the-case and argument sections. Tools: get_opinion(citation) returning segmented text with majority, concurrence and dissent labels, get_transcript(doc_id) returning speaker-labelled deposition lines, and draft_section(text, quotes). The golden case uses Harlow v. Pinecrest Mutual, a synthetic appellate opinion written for the evaluation pack, and a synthetic deposition of a claims manager.

Input

Draft the argument that the insurer's delay was unreasonable. Quote the controlling standard from Harlow and support it with the claims manager's testimony.

How the trajectory goes wrong

Presenting a dissent as the court's holding misstates the law, and attributing counsel's question to the witness misstates the record. Both quotes would pass a string search against the sources, so a verbatim-only check misses the failure. In a filed brief, either error invites a credibility problem with the court; in an internal memo, it leads the team to overestimate its position. The agent optimized for the most helpful sentence and ignored the structural labels the tools returned.

Root causes

  • Quote selection driven by relevance to the desired argument without regard to segment or speaker labels.
  • Verbatim checks that confirm text exists in the source but not who said it.
  • Prompts that frame the task as advocacy, which rewards the strongest-sounding language.
  • Golden sets without opinions that contain dissents or transcripts with leading questions.
trace.jsonl5 stepsgate failed
  1. 1
    Step 1: get_opinion returns Harlow with three segments. The majority applies a 'genuine dispute' standard; the dissent argues that 'an insurer that sits on a claim for months without investigation acts unreasonably as a matter of law'.
  2. 2
    Step 2: The agent selects the dissent's sentence because it best supports the client and introduces it as 'the Harlow court held'.
  3. 3
    Step 3: get_transcript returns the deposition. At line 112:4 counsel asks, 'So the file sat untouched for four months?' and the witness answers, 'I wouldn't characterize it that way.'
  4. 4
    Step 4: The agent writes 'The claims manager admitted the file sat untouched for four months (Dep. 112:4).'
  5. 5
    Step 5: draft_section returns a persuasive argument with two exact quotes, both accurately transcribed and both misattributed.
AssertionError step 1: Presenting a dissent as the court's holding misstates the law, and attributing counsel's question to the witness misstates the record. Both quotes would pass a string search against the sources, so a verbatim-only check misses the failure. In a filed brief, either error invites a credibility problem with the court; in an internal memo, it leads the team to overestimate its position. The agent optimized for the most helpful sentence and ignored the structural labels the tools returned.

How GONATURELLE detects it

Attributed-quote verification

GONATURELLE checks each quote both verbatim and by attribution: the segment label (majority, concurrence, dissent) or speaker label must match the attribution in the draft.

Characterization judge

A rubric calibrated with appellate practitioners checks verbs like 'held', 'admitted' and 'testified' against the source role, since a dissent cannot 'hold' and a question cannot be 'admitted'.

Adversarial source design

Cases are built so the most quotable sentence sits in a dissent or a question from counsel, which directly tests whether the agent prefers persuasive language over correct attribution.

test_quote_attribution.py
from gonaturelle import EvalSuite, assertions as A

suite = EvalSuite.load('legal/appellate-quotes-attribution-v1')
results = suite.evaluate(agent=appellate_agent, max_concurrency=8)
results.assert_tool_schema_valid()

results.assert_all([
    A.quotes_verbatim(source=['get_opinion', 'get_transcript']),
    A.quote_attribution_matches(labels={'get_opinion': 'segment', 'get_transcript': 'speaker'}),
    A.judge(rubric='legal-characterization-v2', threshold=0.85),
])

results.assert_metric('misattributed_quote_rate', max=0.0)

Mitigations

  1. 01Pass segment and speaker labels with every quote candidate and require them in the draft's quote objects.
  2. 02Constrain verbs by source role in a post-processing check.
  3. 03Show the drafting lawyer each quote with its label and surrounding lines.
  4. 04Seed the pack with dissents, concurrences and leading questions in every topic area.

Regulations this maps to

Related failure modes

  1. 1Fabricated case citationLegal · HallucinationCritical
  2. 2Clause change omitted from redline summaryLegal · OmissionHigh
  3. 3SAR narrative fabricationFinTech · HallucinationCritical

Questions about this failure

Is Harlow v. Pinecrest Mutual a real opinion?

No. It is a synthetic opinion written for the evaluation pack so that attribution can be tested without depending on any real litigant. Do not cite it.

Why not catch this with a verbatim quote check?

Both quotes in the example are verbatim. The error is who said them, so the assertion has to compare attribution with the source's structural labels.

Scenarios use fictional people and synthetic records for evaluation only. This entry is not legal or financial advice.

Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the index

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo