Regression testing for claims agents that read the policy before they decide.
FNOL intake, coverage and exclusion interpretation, and adjudication trajectories tested against state claims rules on every release.
- 1Sewer backup with endorsementendorsement restores coverage to sublimitfail
- 2Wind-created opening, interior rainwindstorm perilpass
- 3Slow leak over monthsseepage and wear-and-tear exclusionspass
- 4CA acknowledgement within 15 days10 CCR 2695.5(e)+6 days
- 5Denial cites policy provisionwritten basis for every denialpass
Three packs, from first notice to final letter.
Expand a pack to see the assertions it runs.
1
First Notice of Loss Intakeinsurance/fnol-intake1,240 cases · v1.5.0 · κ 0.881.5.0FirstFirst notice of loss agents across phone, chat and email: required fields captured (policy, date and location of loss, cause, injuries, other parties, police report), no invented values, and correct routing for injury, total loss and suspected fraud.
Labelled by two senior claims adjusters; disagreements adjudicated.
- assert_required_fields(schema="fnol_property_v3")
- assert_no_invented_values(fields=["date_of_loss", "policy_number"])
- assert_routes_to("bodily_injury_desk", when="injury_reported")
2
Coverage & Exclusion Interpretationinsurance/coverage-exclusions1,650 cases · v1.9.2 · κ 0.831.9.2CoverageCoverage interpretation against standard ISO-based homeowners, commercial property and general liability forms plus common endorsements: water, wind, earth movement, mold, ordinance or law, and concurrent causation.
Labelled by a coverage attorney and a senior property adjuster, with an actuary adjudicating; disagreements adjudicated.
- assert_determination_matches()
- assert_cites_policy_basis()
- assert_endorsements_applied()
3
Claims Adjudication Trajectoriesinsurance/claims-adjudication1,310 cases · v1.3.1 · κ 0.861.3.1ClaimsAdjudication agents run end to end: coverage verification before investigation, reserves within authority, state-specific acknowledgement and decision deadlines, and denial letters that state the policy provisions relied on.
Labelled by two claims managers licensed in the risk state; disagreements adjudicated.
- assert_order("verify_coverage", "set_reserve")
- assert_deadline(state="claim.risk_state")
- assert_no_denial_without_policy_basis()
Reviewer κ: Cohen’s kappa between the two independent credentialed reviewers on the current pack release, before adjudication. Published on each pack card.
FNOL intake validation, field by field.
An intake agent that writes a fluent summary can still drop the one field that routes a claim to the injury desk. GONATURELLE checks the structured record the agent emits against the conversation: required fields present, values traceable to something the claimant said, and no date of loss or policy number the agent made up.
- Required-field completeness by line of business
- Hallucinated values that no turn of the conversation contains
- Routing rules for injury, total loss and special investigations
- Code-mixed and noisy transcripts from synthetic stress
policy_numberHO-4471-2290date_of_loss2026-10-31loss_locationInsured premisescause_of_losswater: appliance dischargeinjuries_reportedmissingother_partiesnone statedmitigation_stepsmissingcontact_preferenceSMS
Coverage and exclusion interpretation, tested against adjudicated answers.
Cases follow the structure of the standard ISO-based homeowners special form, described in plain terms. Pick a loss, toggle the endorsement and the state, and compare main with the PR that merged two water exclusions.
During a storm the municipal sewer backs up through the basement floor drain; finished basement damaged.
Notice of claim received Mon, Nov 2, 2026. Denial letters must meet 10 CCR 2695.7(b)(1): written denial with the specific policy provisions relied on.
Flood and surface water exclusion.
Water backup endorsement restores coverage for water that backs up through sewers or drains, up to its sublimit.
- Agent acknowledgement due
- Mon, Nov 23, 2026
- Statutory deadline (10 CCR 2695.5(e))
- Tue, Nov 17, 2026
- Coverage determination matches adjudicated reference
- Determination cites the specific policy provision relied on
- Acknowledgement due within 15 calendar days of notice
- Flood referral given when flood exclusion applies
assert_determination_matches() · assert_cites_policy_basis() · assert_deadline(state="CA")
Claims adjudication trajectories, checked step by step.
State unfair claims settlement practices rules govern how a claim is handled, not only how it ends. The trajectory engine asserts on each stage the agent runs.
Trajectory assertions- 01
Verify coverage
Policy in force on the date of loss, named insured, location on the declarations, deductible and limits read from the policy system, not the conversation.
assert_order("verify_coverage", "open_investigation") - 02
Acknowledge and request
Acknowledgement and requests for information inside the state's clock; the agent's date arithmetic is tested across weekends, holidays and calendar versus business days.
assert_deadline(state="claim.risk_state", event="acknowledge") - 03
Investigate and reserve
Reserve set within the agent's delegated authority; anything above it routes to a human adjuster with the reasoning attached.
assert_reserve_within(authority="agent_tier_1") - 04
Decide and explain
Accept, deny or partially deny with the specific policy language relied on, and no reliance on unapproved data sources or protected characteristics.
assert_no_denial_without_policy_basis()
- A tribunal held a company responsible for what its customer-facing chatbot told a customer. Claims chat is no different.
- Liable
- A tribunal held a company responsible for what its customer-facing chatbot told a customer. Claims chat is no different.
- Moffatt v. Air Canada, 2024 BCCRT 149
- of multi-step agent failures in the catalog were tool-argument schema or parameter-boundary violations, not reasoning breakdowns.
- 32.8%
- of multi-step agent failures in the catalog were tool-argument schema or parameter-boundary violations, not reasoning breakdowns.
- GONATURELLE Failure Modes Catalog analysis, 2026
- of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat.
- 41%
- of prompt-only pull requests changed at least one tool-call trajectory in a multi-turn suite while the final-answer score stayed flat.
- GONATURELLE regression study, 2026 (1,180 PRs across 14 production agents)
Evidence for the NAIC AI Model Bulletin.
Adopted by the NAIC in December 2023 and issued by roughly half the states as of 2025, the bulletin expects insurers to govern, test and document AI systems that affect consumers. Here is where GONATURELLE output fits.
- Written AIS programSuite inventory, pack versions and owners exported as the testing section of your AI systems program.
- Unfair discrimination testingCounterfactual pairs and outcome ratios for claims and underwriting agents, with proxy reasoning flagged by calibrated judges.
- Documentation and audit trailEvery gated run produces a signed manifest: data, prompts, model versions, scores and the reviewer who approved the threshold.
- Third-party AI systemsRun the same packs against a vendor's claims model before contract signature and at each version they ship.
- Human oversightEscalation assertions prove that denials, large reserves and injury claims reach a licensed adjuster.
- Consumer data protectionsPII and PHI leakage checks on tool arguments, with de-identification on ingest and a BAA where health data is involved.
Go deeper on insurance AI testing.
Regulatory testing guides
- NAIC AI Model BulletinUnited States, state insurance regulators
- Colorado AI Act (SB 26-189) bias testingColorado, US, effective 1 January 2027
- EU AI Act high-risk testingEuropean Union
- NIST AI RMF Generative AI ProfileUnited States, NIST
- Bank model risk: SR 26-2, formerly SR 11-7United States, Federal Reserve, OCC and FDIC
- HIPAA PHI leakage preventionUnited States, HHS OCR
Failure modes in the catalog
- 1Coverage exclusion misread
- 2FNOL required field drop
- 3Claim denial without policy basis
- 4State deadline miscalculation
- 5Protected class proxy reasoning
- 6Tool call schema drift
Product capabilities used here
Frequently asked questions
Do you reproduce ISO policy forms?
No. ISO forms are copyrighted. Coverage cases describe the policy provisions in plain terms and are labelled by coverage attorneys and adjusters; customers who license ISO forms can load their own form library and endorsements into a private pack.
Which state claims deadlines do you test?
Claims adjudication cases carry the risk state, and assertions apply that state's unfair claims settlement practices rules, which are based in most states on the NAIC model act and regulation but set their own clocks. California's acknowledgement deadline, for instance, is 15 calendar days from notice (10 CCR 2695.5(e)) and its accept-or-deny deadline is 40 days from proof of claim (2695.7(b)). Rules are versioned and reviewed when states amend them.
What is the NAIC AI Model Bulletin and does it apply to us?
The NAIC adopted its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers in December 2023. It applies in states whose insurance department has issued it, roughly half the states plus the District of Columbia as of 2025, each with its own wording. It expects a written AI systems program with governance, risk management, testing and third-party oversight. Check your domiciliary and operating states.
Can the agent's coverage decisions go straight to policyholders?
That is your call and your regulator's. GONATURELLE tests the agent offline and gates releases; it does not sit in the live claims path, and it does not replace a licensed adjuster's judgment on contested claims.
How are concurrent causation and anti-concurrent clauses handled?
As their own case family. Wind-plus-flood and fire-plus-earth-movement losses are labelled under the policy's actual causation language, so an agent that applies one rule everywhere fails the cases where the form says otherwise.
Test the claims agent before it writes its first denial letter.
Run the FNOL and coverage packs against your agent in minutes. 14-day free production trial, no card.