How a top-20 US commercial insurer took prompt regressions in claims triage to zero
The claims AI team at a top-20 US commercial insurer runs an agent that triages first notice of loss, interprets coverage and routes claims. Trajectory gates cut manual prompt QA by 80% and kept production regressions at zero through Day 120.
- Customer
- Top-20 US commercial insurer
- Published
- 22 Sept 2026
- Stack
- CrewAIAWS BedrockGuidewire ClaimCenter integrationCircleCIMicrosoft Entra ID SSO
- prompt regressions reaching production between Day 60 and Day 120, from 5 in the prior two quarters
- 0
- prompt regressions reaching production between Day 60 and Day 120, from 5 in the prior two quarters
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- reduction in engineering hours spent on manual prompt QA
- 80%
- reduction in engineering hours spent on manual prompt QA
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- pack and BYOD claims cases running in the nightly suite
- 1,310 + 920
- pack and BYOD claims cases running in the nightly suite
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
The problem
Before onboarding, as described by the team.
The insurer's claims agent takes first notice of loss from brokers and policyholders, extracts the required fields, checks the policy for applicable coverage and exclusions, and routes the claim to the right adjuster queue. In the two quarters before onboarding, five prompt changes had caused production regressions, the worst being a flood exclusion applied to wind-driven rain damage that an endorsement explicitly restored.
Each release required a week of manual QA by engineers and two senior adjusters. The team of 22 engineers estimated that QA consumed about a fifth of every sprint. The adjusters' review covered the routing outcome but not the steps, so a run that reached the right queue by skipping the endorsement lookup counted as a pass.
The insurer's AI governance policy, written with the NAIC model bulletin on insurers' use of AI in mind, required documented testing for any change that could affect claim outcomes.
Coverage interpretation tested at the step that matters
The team combined the insurance/claims-adjudication and insurance/coverage-exclusions packs with 920 BYOD cases built from closed claims files, de-identified on ingest and labelled by the insurer's senior adjusters with GONATURELLE adjudication. The key assertion: any exclusion the agent applies must be preceded by a lookup of endorsements on the same policy, and the cited policy section must exist in the retrieved form.
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
suite = EvalSuite.load('insurance/claims-adjudication', extra=['insurance/coverage-exclusions', 'byod/closed-claims-2025'])
results = suite.evaluate(agent=claims_agent, max_concurrency=20)
results.assert_tool_schema_valid()
TrajectoryTester(results).assert_order('fetch_policy', 'fetch_endorsements', 'apply_exclusion')
results.assert(A.cites_only(source='fetch_policy.sections'))
results.assert(A.required_fields('create_fnol', ['loss_date', 'loss_location', 'peril', 'policy_number']))
results.assert(A.param_within('set_reserve.amount', max_from='fetch_policy.limits.per_occurrence'))Synthetic stress for broker phrasing
Broker emails vary widely in format and tone. Synthetic stress multiplied 60 real FNOL cases into about 2,400 variants with typos, forwarded threads, missing loss dates and conflicting addresses. Failures were clustered so engineers fixed the extraction prompt once per cluster rather than once per case.
Results at Day 60 and Day 120
- Between Day 60 and Day 120 no prompt regression reached production, compared with five in the two quarters before onboarding.
- Engineering hours on manual prompt QA fell by 80%. The senior adjusters now review only failure clusters and new BYOD cases.
- The governance committee receives the signed run manifest for each release as its documented test record. All figures are customer-reported at the Day 60 and Day 120 reviews.
“We took prompt regressions in production to zero and cut manual QA by 80%. More important, the adjusters stopped re-reading outcomes and started reviewing the failure clusters that actually needed them.”
More customer stories
Show all- Series B clinical documentation companyHow a clinical documentation team gave its safety board the evidence to approve an ambient scribe agentHealth
- US regional bank, small-business underwriting agentHow a regional bank's model risk team turned CI runs into adverse action evidenceFinTech
- Legal research copilot companyHow a legal research copilot stopped a broken release in week two and wired the gate in an afternoonLegal
Figures are reported by the customer against a baseline captured before onboarding and approved for publication. GONATURELLE provides automated test evidence; it does not certify compliance, and the customer remains responsible for clinical, credit, legal and claims decisions and for regulatory filings.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.