How a legal research copilot stopped a broken release in week two and wired the gate in an afternoon
A venture-backed company sells a research copilot to litigation teams. Citation verification suites and a fast-path GitHub Actions gate caught a release that would have shipped unverifiable citations.
- Customer
- Legal research copilot company
- Published
- 9 Jul 2026
- Stack
- LlamaIndexOpenAIElasticsearchGitHub ActionsDeepEval (plugin assertions)
- when the gate blocked a release with a citation-verification regression; the CTO says the subscription paid for itself there
- Week 2
- when the gate blocked a release with a citation-verification regression; the CTO says the subscription paid for itself there
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- to add the gate to the existing GitHub Actions workflow, per the staff DevOps engineer
- 60 s
- to add the gate to the existing GitHub Actions workflow, per the staff DevOps engineer
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- fabricated citations in the citation-verification suite on every release since Day 30
- 0
- fabricated citations in the citation-verification suite on every release since Day 30
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- jurisdiction-boundary failure rate on the multi-document suite at Day 120, down from 11.4% at baseline
- 3.1%
- jurisdiction-boundary failure rate on the multi-document suite at Day 120, down from 11.4% at baseline
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
The problem
Before onboarding, as described by the team.
The copilot answers research questions by searching a case law index, retrieving opinions, and drafting a memo with pin cites. Its customers are litigators who have read about Mata v. Avianca, where attorneys were sanctioned $5,000 for filing a brief with chatbot-fabricated citations, and about the Stanford RegLab study measuring hallucination rates of 17% to 33% in leading legal research tools.
The team had a retrieval evaluation harness built on an open-source framework, measuring context precision and answer faithfulness. It did not check whether each cited case existed in the index, whether the pin cite pointed at the quoted passage, or whether the agent mixed jurisdictions when a question named one state.
Releases went out every Thursday. The staff DevOps engineer owned CI and was wary of adding anything that slowed it down.
Citation existence as a hard assertion
The legal/citation-verification pack (version 2.2.0, 2,480 cases) and the legal/multi-doc-consistency pack provided labelled research questions with the correct authorities, prepared by practising attorneys. The core assertion is deterministic: every citation in the memo must resolve to a document the agent actually retrieved in that run, and every quotation must appear in that document.
from gonaturelle import EvalSuite, assertions as A
suite = EvalSuite.load('legal/citation-verification')
results = suite.evaluate(agent=research_agent, max_concurrency=16)
results.assert(A.cites_only(source='retrieve_opinion.doc_id'))
results.assert(A.quotes_verbatim(source='retrieve_opinion.text'))
results.assert(A.jurisdiction_consistent(field='question.jurisdiction'))
results.assert(A.judge(rubric='legal/holding-accuracy', threshold=0.93))A gate small enough not to slow anyone down
The fast path runs 20 critical cases on every push and finished in a little over a minute in the team's runners. The full suite runs on the release branch every Wednesday night. Adding the gate meant one job in the existing workflow file.
gonaturelle:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: gonaturelle/gate-action@v2
with:
suite: legal/citation-verification
mode: fast-path
fail-on: regression
api-key: ${{ secrets.GONATURELLE_API_KEY }}Results at Day 60 and Day 120
- In week two the Wednesday full run failed on a retrieval change that truncated opinions to their first 4,000 tokens. The agent began citing cases it had only partly read, and 7% of pin cites pointed at passages outside the retrieved text. The release was held and fixed the same day.
- Since Day 30 every release has passed the citation-verification suite with zero fabricated citations. Jurisdiction-boundary failures on the multi-document suite fell from 11.4% at baseline to 3.1% at the Day 120 review.
- The team kept its retrieval metrics and runs them as plugin assertions inside the same gate. Figures are customer-reported against the pre-onboarding baseline.
“It paid for itself in week two. The nightly suite stopped a release that would have put unverifiable pin cites in front of litigators, which is the one thing our customers will not forgive.”
More customer stories
Show all- Series B clinical documentation companyHow a clinical documentation team gave its safety board the evidence to approve an ambient scribe agentHealth
- US regional bank, small-business underwriting agentHow a regional bank's model risk team turned CI runs into adverse action evidenceFinTech
- Top-20 US commercial insurerHow a top-20 US commercial insurer took prompt regressions in claims triage to zeroClaims
Figures are reported by the customer against a baseline captured before onboarding and approved for publication. GONATURELLE provides automated test evidence; it does not certify compliance, and the customer remains responsible for clinical, credit, legal and claims decisions and for regulatory filings.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.