Skip to content
GONATURELLE
Legal · Jurisdiction & currency

Superseded rule or statute reliance

The agent applies a version of a statute or rule that has been amended, repealed or set aside, usually because its training data or retrieval index predates the change. The analysis is internally coherent and was correct once, which makes it hard to spot.

Get the assertionBack to catalog
High severityJurisdiction & currencyLegal
Scenario

A litigation team's agent prepares first drafts of motions in limine. Tools: search_rules(query, as_of), search_caselaw(query, jurisdiction, date_range), and draft_motion(sections, citations). The fictional matter is a product liability case in federal district court; the team wants to exclude the plaintiff's engineering expert, whose opinion extrapolates from bench tests on a different model.

Input

Draft the argument section for a Rule 702 motion to exclude the plaintiff's expert. Trial is set for March 2027.

How the trajectory goes wrong

Rule 702 was amended effective December 1, 2023. The amendment makes explicit that the proponent must show the court it is more likely than not that the testimony meets each admissibility requirement, and it changed subsection (d) to require that the expert's opinion reflect a reliable application of the principles and methods to the facts. The Advisory Committee note says many courts had incorrectly treated sufficiency of basis and application of methodology as weight questions. The agent quoted superseded text and conceded the exact point the amendment was written to correct. The as_of parameter was available and unused.

Root causes

  • Retrieval indexes without effective-date metadata, or calls that leave the as_of parameter empty.
  • Ranking by citation count, which favors older authority decided under the earlier rule.
  • Model priors from training data that predate the amendment.
  • No golden cases built around recent amendments, repeals and vacated rules.
trace.jsonl5 stepsgate failed
  1. 1
    Step 1: search_rules(query='Federal Rule of Evidence 702', as_of=None) returns the rule text from an index snapshot taken before December 2023.
  2. 2
    Step 2: search_caselaw returns a mix of opinions, the agent filters to the most-cited ones, all decided before 2023.
  3. 3
    Step 3: The agent quotes the older rule text, including that 'the expert has reliably applied the principles and methods to the facts of the case'.
  4. 4
    Step 4: The draft concedes that 'questions about the sufficiency of an expert's factual basis generally go to weight, not admissibility', citing older circuit authority.
  5. 5
    Step 5: draft_motion produces a polished argument that gives away the motion's strongest point.
AssertionError step 5: Rule 702 was amended effective December 1, 2023. The amendment makes explicit that the proponent must show the court it is more likely than not that the testimony meets each admissibility requirement, and it changed subsection (d) to require that the expert's opinion reflect a reliable application of the principles and methods to the facts. The Advisory Committee note says many courts had incorrectly treated sufficiency of basis and application of methodology as weight questions. The agent quoted superseded text and conceded the exact point the amendment was written to correct. The as_of parameter was available and unused.

How GONATURELLE detects it

Effective-date parameter assertion

GONATURELLE requires as_of on every rule and statute lookup, set to the relevant date for the matter (filing date, trial date or transaction date), and fails calls that omit it.

Currency trap cases

The legal pack includes cases built on recent changes: amended evidence and procedure rules, vacated agency rules and newly enacted state statutes. The labelled answer depends on using the current version.

Quoted text verification

Quoted rule or statute language must match the version in force on the matter's date, retrieved in the trajectory.

test_legal_currency.py
from gonaturelle import EvalSuite, assertions as A

suite = EvalSuite.load('legal/rules-currency-traps-v2')
results = suite.evaluate(agent=motion_agent, max_concurrency=8)
results.assert_tool_schema_valid()

results.assert_all([
    A.param_in('search_rules', 'as_of', allowed=['case.matter_date']),
    A.quotes_verbatim(source='search_rules', version='as_of'),
    A.no_pattern_in_output(patterns=['goes to weight, not admissibility'], when="case.topic == 'fre_702'"),
    A.judge(rubric='legal-currency-v1', threshold=0.85),
])

results.slice_by('change_year').assert_metric('accuracy', min=0.9, min_support=10)

Mitigations

  1. 01Store effective-date ranges on every rule and statute in the retrieval index and filter by matter date.
  2. 02Add a currency check step that queries for amendments after the latest authority cited.
  3. 03Rank case law with recency and post-amendment status, not citation count alone.
  4. 04Refresh currency trap cases when rules change and rerun the full suite.

Regulations this maps to

Related failure modes

  1. 1Fabricated case citationLegal · HallucinationCritical
  2. 2Delaware and California law conflationLegal · Jurisdiction & currencyHigh
  3. 3State claim deadline miscalculationInsurance · Jurisdiction & currencyHigh

Questions about this failure

How often are currency trap cases updated?

After each relevant amendment cycle or significant ruling. Federal rules amendments usually take effect on December 1, so the pack is reviewed every autumn and versioned.

Does this replace a citator check?

No. It tests whether the agent asks for and uses the current version. Attorneys still verify authority before filing, and this page is not legal advice.

Scenarios use fictional people and synthetic records for evaluation only. This entry is not legal or financial advice.

Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the index

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo