Claim denial without policy basis
The agent drafts a claim denial that states a conclusion without identifying the policy provision and the facts that support it, or cites a provision that does not apply. Many state unfair claims practices rules require a reasonable explanation of the basis for a denial, so a vague letter is itself a compliance problem.
A personal lines carrier uses an agent to draft denial and partial-denial letters for adjuster approval. Tools: get_claim(claim_id), get_adjuster_notes(claim_id), get_policy_forms(policy_id), and draft_letter(type, body, cited_provisions). The fictional claim is a California homeowners claim for a cracked foundation discovered during a kitchen remodel.
Adjuster: Coverage denied, this is settling and long-term deterioration, not a sudden loss. Draft the denial letter.
How the trajectory goes wrong
California's Fair Claims Settlement Practices Regulations (10 CCR 2695.7(b)(1)) require a written denial to include a statement of the factual and legal bases for the denial and to refer to the specific policy provision, condition or exclusion relied on. The letter has neither the engineer's finding nor any provision. Other states impose similar obligations through their versions of the NAIC Unfair Claims Settlement Practices model act. The adjuster's conclusion may well be correct; the failure is that the agent produced a letter that cannot be sent as written and, if approved under time pressure, exposes the carrier to a market conduct finding.
Root causes
- The agent treats the adjuster's conclusion as sufficient and skips retrieving policy forms.
- Letter templates with optional rather than required provision fields.
- No state-specific content rules attached to the letter type.
- Evals that score letter tone and readability but not legal content.
- 1Step 1: get_claim returns the loss description, the engineer's report date and the policy number.
- 2Step 2: get_adjuster_notes returns the engineer's finding that the cracking resulted from long-term soil settlement over several years.
- 3Step 3: The agent skips get_policy_forms because the adjuster already stated the conclusion.
- 4Step 4: draft_letter produces: 'After careful review, we have determined that your loss is not covered under your policy. If you have questions, please contact us.' with cited_provisions=[].
How GONATURELLE detects it
Required provision citations
For denial letter types, GONATURELLE requires at least one cited provision that exists in the policy forms retrieved in the same trajectory, quoted or referenced by section.
Fact basis grounding
The letter must state the factual basis, and each fact must trace to claim data or adjuster notes. Generic language like 'after careful review' does not count.
State rule packs
Letter assertions load per-state content requirements, so a California denial is checked against California rules and a Texas denial against Texas rules.
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
suite = EvalSuite.load('insurance/claim-letters-state-rules-v2')
results = suite.evaluate(agent=letters_agent, max_concurrency=10)
results.assert_tool_schema_valid()
denials = results.filter("case.letter_type in ('denial', 'partial_denial')")
TrajectoryTester(denials).assert_order('get_policy_forms', 'draft_letter')
denials.assert_all([
A.cites_only(source='get_policy_forms', field='draft_letter.args.cited_provisions', min_count=1),
A.groundedness(claims='draft_letter.args.body', sources=['get_claim', 'get_adjuster_notes'], min=0.9),
A.state_rules(pack='claims-letters', state='case.loss_state'),
])Mitigations
- 01Make cited_provisions required and non-empty for denial letter types at the tool level.
- 02Always retrieve policy forms before drafting a denial, regardless of what the adjuster states.
- 03Maintain per-state letter requirement packs reviewed by compliance counsel.
- 04Have the agent flag denials where it cannot find a supporting provision instead of drafting around the gap.
Regulations this maps to
- NAIC AI BulletinNAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers (adopted 4 December 2023), Sections 3 and 4
- Colorado AI ActSB 24-205 (2024), amended by SB25B-004 (2025); repealed and reenacted by SB 26-189 (2026)
- CMS prior authorizationCMS-0057-F (January 2024); 42 CFR 422.101(b)(6) and 422.566(d); CMS Medicare Advantage FAQ on algorithms and AI (February 2024)
Questions about this failure
Does the suite decide whether the claim should be denied?
No. The adjuster makes the coverage decision. The suite tests whether the drafted letter states that decision with the facts and policy provisions that support it. This page is not legal advice.
How are state rules kept current?
State rule packs are versioned and reviewed when regulations change. Customers can extend them with their own filed forms and internal letter standards.
Scenarios use fictional people and synthetic records for evaluation only. This entry is not legal or financial advice.
Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the indexStop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.