Skip to content
GONATURELLE

Evaluation for legal research and contract agents that cite their sources.

Catch fabricated citations, altered quotations, wrong-forum authority and dropped clauses before a brief or a redline leaves the building.

Start free trialExplore demo
4,730 attorney-labelled cases across 3 packs
Golden suite · fast path
legal/citation-verification
2,480 cases · v2.2.0 · 5 critical shown
  1. 1All cited authorities resolvereporter, docket and research-log index4/5
  2. 2Quotations are verbatimcharacter-level comparisonfail
  3. 3Authority binds in N.D. Cal.forum and governing-law checkfail
  4. 4Negative history disclosedoverruled, reversed, disapproved1 gap
  5. 5No cross-matter retrievalprivilege wallpass
PR #219 · faster research mode · 4 regressions2/5
merge blocked
Hallucination rate measured in leading AI legal research tools on a preregistered query set.
17% to 33%
Hallucination rate measured in leading AI legal research tools on a preregistered query set.
Magesh et al., Stanford RegLab and HAI, "Hallucination-Free?" (2024)
Sanctions imposed on attorneys who filed a brief citing cases fabricated by a chatbot.
$5,000
Sanctions imposed on attorneys who filed a brief citing cases fabricated by a chatbot.
Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. June 22, 2023)
Judge agreement with attorney consensus: calibrated ensemble versus a single uncalibrated prompt.
0.87 vs 0.42
Judge agreement with attorney consensus: calibrated ensemble versus a single uncalibrated prompt.
GONATURELLE calibration study, 2026 (6,400 double-annotated cases)

Three packs for research, redlining and diligence.

Expand a pack to see the assertions it runs.

How packs are built
  1. 1Contract Review & Redlininglegal/contract-redline1,390 cases · v1.7.0 · κ 0.85

    Contract review and redlining agents measured against a negotiation playbook: clause omission (limitation of liability, indemnity carve-outs, assignment, data-protection terms), non-standard term flagging, and redlines that silently change meaning.

    Labelled by two commercial contracts attorneys; disagreements adjudicated.

    • assert_clause_present("limitation_of_liability")
    • assert_flags_deviation(playbook="saas-vendor-v4")
    • assert_redline_preserves(defined_terms=True)
  2. 2Case Law & Citation Verificationlegal/citation-verification2,480 cases · v2.2.0 · κ 0.90

    Statutory interpretation and case law research agents: every authority must exist, quotations must be verbatim, the holding must support the proposition, subsequent history must be disclosed, and the authority must bind in the forum.

    Labelled by two litigation attorneys admitted in the relevant jurisdiction; disagreements adjudicated.

    • assert_citations_exist()
    • assert_quotes_verbatim(tolerance=0)
    • assert_binding_in(forum="matter.forum")
  3. 3Multi-Document Consistencylegal/multi-doc-consistency860 cases · v1.1.4 · κ 0.81

    Diligence and drafting agents that read across a document set: defined terms, cap amounts, notice periods, governing law and order-of-precedence clauses must agree across MSA, SOW, DPA and order forms.

    Labelled by two transactional attorneys with a senior associate adjudicating; disagreements adjudicated.

    • assert_defined_terms_consistent(docs=matter.docs)
    • assert_numeric_agreement(field="liability_cap")
    • assert_precedence_respected()

Reviewer κ: Cohen’s kappa between the two independent credentialed reviewers on the current pack release, before adjudication. Published on each pack card.

Run five checks on every citation the agent writes.

A research memo from the PR under test. Press run, switch the forum, and click any authority to see why it failed. Authorities come from the matter's fixture index.

legal/citation-verification · matter LEG-2291
Research memo: enforceability of a non-compete against a California-based engineer
Authority cited by the agentExistsQuoteSupportsGood lawForum
[4] Brandt v. Corvex Systems
exists: No matching authority in the reporter index, docket search or the matter's research log. Fabricated citation.
quote, support, history and forum checks: not evaluable, authority does not exist.
Citation existence rate
80%
Clean citations
1/5
assert_citations_exist() · assert_quotes_verbatim() · assert_binding_in(forum="N.D. Cal.")

Contract review and redlining, measured clause by clause.

Clause omission detectionThe redline looked tidy. It deleted the data-breach carve-out from the liability cap.
12.1 Except for a party's indemnification obligations under Section 11, and liability arising from a breach of Section 8 (Data Protection), neither party's aggregate liability shall exceed the fees paid in the twelve (12) months preceding the claim.
assert_clause_present("liability_cap.carve_outs.data_protection") fail
Default gates
  • Clause omission recall≥ 0.97
  • Playbook deviation precision≥ 0.92
  • Meaning-changing edits flagged≥ 0.95
Thresholds from the default contract-review suite configuration; tune them per playbook.
Non-standard term flaggingPositions come from your playbook, not the model’s idea of market.
auto-renewal > 12 mounilateral price changeIP assignment of pre-existing workuncapped indemnityaudit rights without noticeexclusive venue abroad
Synthetic variantsOne negotiated clause, renumbered, reordered and restyled 120 ways.

Agents that only find “Limitation of Liability” under heading 12 break when it moves to Schedule B. Synthetic stress catches it.

Synthetic stress

Multi-document consistency verification.

Diligence agents read a document set as one record. GONATURELLE checks that what they report agrees with every document, and that they notice when the documents disagree with each other.

TermMSASOWDPAOrder formAgent must
Liability cap12 months' fees12 months' feesUncapped for data breach$250,000flag conflict
Notice period for termination30 days30 daysn/a30 daysreport as consistent
"Customer Data" definition§1.4, includes Personal DataReferences MSA §1.4§2.1, excludes aggregated datan/aflag conflict
Governing lawDelawareDelawareIreland (EU SCCs)Delawarereport as consistent
Order of precedenceDPA > MSA > SOWSOW > MSADPA prevails on data termsn/aflag conflict

Jurisdictional boundary checks.

  • Forum and governing lawThe authority must bind, or be clearly labelled persuasive, in the forum the matter record names. A Delaware Chancery holding answering a California employment question fails.
  • Temporal validityStatutes are checked against effective dates and amendments; superseded or repealed provisions fail even when quoted accurately.
  • Court hierarchy and historyNegative subsequent history (reversed, overruled, disapproved) must be disclosed when the agent relies on the authority.
  • Matter wallsRetrieval cannot reach documents from another client matter. Privilege leakage across matters is a hard failure, not a score.
tests/test_research_agent.py
from gonaturelle import EvalSuite, TrajectoryTester
from research.agent import memo_agent

suite = EvalSuite.load("legal/citation-verification", version="2.2.0")

def test_research_memo_citations():
    results = suite.evaluate(agent=memo_agent, max_concurrency=8)

    results.assert_citations_exist()  # reporter, docket, research log
    results.assert_quotes_verbatim(tolerance=0)
    results.assert_binding_in(forum="matter.forum", allow_persuasive=True)
    results.assert_history_disclosed(
        negative=["overruled", "reversed", "disapproved"],
    )

    # The agent must search before it drafts, and never read across matters
    TrajectoryTester(results).assert_order("search_case_law", "draft_memo")
    TrajectoryTester(results).assert_never_called(
        "fetch_document", where="matter_id != case.matter_id",
    )
Trajectory assertions

Go deeper on legal AI testing.

Regulatory testing guides

Failure modes in the catalog

Product capabilities used here

Golden DatasetsTrajectory EvalCalibrated JudgesSynthetic StressCI/CD Gates
Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the index

Frequently asked questions

How do you detect a fabricated citation?

Every authority the agent cites is resolved against a reporter and docket index plus the matter's own research log. Anything that does not resolve fails assert_citations_exist() deterministically, before any judge is involved. Quotations are then compared verbatim against the source text, and a calibrated judge checks whether the holding actually supports the stated proposition.

Why does jurisdiction need its own check?

Because the commonest research error is a real, accurately quoted case from the wrong court. Each golden case records the forum and governing law, and the assertion fails when the agent relies on authority that does not bind there without saying so.

Does GONATURELLE give legal advice or certify our product's output?

No. GONATURELLE measures how your agent performs on attorney-labelled test cases and records the evidence. It does not review live client matters, does not certify outputs, and does not replace a lawyer's review of work product.

How does this relate to ABA Formal Opinion 512?

Opinion 512 (July 2024) reminds lawyers that competence, confidentiality, communication and reasonable fees apply to generative AI use, including verifying output. Systematic regression tests on citation accuracy and confidentiality boundaries are one way vendors and firms can show their tools are reviewed, not trusted blindly.

Can we load our own playbooks and precedent?

Yes. Upload your clause library and negotiation playbook; GONATURELLE turns positions into deviation assertions and expands each real negotiation into synthetic variants (renumbered sections, reordered clauses, different defined-term styles).

Find the fabricated citation before opposing counsel does.

Run the citation and redline packs against your agent in minutes. 14-day free production trial, no card.

Start free trialBook a technical demo