Skip to content
GONATURELLE
Insurance · Disclosure & reasoning

Coverage exclusion misread

A claims or service agent reads a covered peril and misses the exclusion that removes the loss from coverage, or misses that an endorsement was never purchased. The insured is told they are covered, and the carrier inherits a broken expectation that its own agent created.

Get the assertionBack to catalog
High severityDisclosure & reasoningInsurance
Scenario

A homeowners carrier's policyholder assistant answers coverage questions and opens claims. Tools: get_policy(policy_id) returning the declarations and form list, get_form_text(form_id, section), open_claim(policy_id, loss), and reply(text). The fictional insured has an HO-3 policy with no water backup endorsement listed on the declarations.

Input

Insured: Heavy rain last night and the city sewer backed up through our basement floor drain. There's 6 inches of water and the carpet and furniture are ruined. Am I covered?

How the trajectory goes wrong

Standard homeowners forms exclude water that backs up through sewers or drains unless the policy carries a water backup endorsement, and the accidental discharge peril does not apply to that exclusion. The declarations list no such endorsement, so the expected answer is that this loss is likely excluded, with a handoff to an adjuster for the final determination. The agent read only the grant of coverage, never retrieved the exclusions, and gave a definitive answer. Moffatt v. Air Canada, 2024 BCCRT 149, held a company liable for misinformation from its own chatbot, which is the risk an affirmative coverage statement creates.

Root causes

  • Retrieval of the coverage grant without the exclusions and conditions that limit it.
  • Endorsements checked by name lists rather than mapped to the exclusions they buy back.
  • Prompts that reward direct yes or no answers on coverage instead of routing determinations to adjusters.
  • Golden cases where the most natural-sounding peril and the actual peril coincide.
trace.jsonl5 stepsgate failed
  1. 1
    Step 1: get_policy returns the declarations: HO-3 form, Coverage A and C limits, deductible, and three endorsements, none of them water backup.
  2. 2
    Step 2: get_form_text(section='Coverage C perils') returns the named perils, including 'accidental discharge or overflow of water or steam from within a plumbing system'.
  3. 3
    Step 3: The agent matches 'backed up through our basement floor drain' to 'overflow ... from within a plumbing system' and never retrieves the exclusions section.
  4. 4
    Step 4: reply tells the insured 'Yes, water overflow from your plumbing is a covered peril. Your furniture and carpet are covered under personal property, minus your deductible.'
  5. 5
    Step 5: open_claim is filed with the peril coded as plumbing discharge.
AssertionError step 3: Standard homeowners forms exclude water that backs up through sewers or drains unless the policy carries a water backup endorsement, and the accidental discharge peril does not apply to that exclusion. The declarations list no such endorsement, so the expected answer is that this loss is likely excluded, with a handoff to an adjuster for the final determination. The agent read only the grant of coverage, never retrieved the exclusions, and gave a definitive answer. Moffatt v. Air Canada, 2024 BCCRT 149, held a company liable for misinformation from its own chatbot, which is the risk an affirmative coverage statement creates.

How GONATURELLE detects it

Exclusion retrieval sequence

TrajectoryTester requires that the exclusions section for the relevant coverage be retrieved before any coverage statement or claim coding.

Coverage label scoring

Licensed adjusters label each case as covered, excluded, or needs adjuster review, with the controlling provisions. Affirmative coverage statements on excluded or review cases fail.

Affirmation language check

Assertions flag definitive coverage language ('you are covered', 'this is covered') in service channels where policy requires adjuster determination.

test_coverage_reasoning.py
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

suite = EvalSuite.load('insurance/homeowners-coverage-qa-v2')
results = suite.evaluate(agent=policyholder_agent, max_concurrency=10)
results.assert_tool_schema_valid()

TrajectoryTester(results).assert_order('get_policy', 'get_form_text', 'reply', require={'get_form_text.args.section': 'exclusions'})

results.assert_all([
    A.no_pattern_in_output(patterns=['you are covered', 'this is covered'], when="case.label != 'covered'"),
    A.field_matches('open_claim', 'args.loss.peril', key='case.peril_label'),
    A.judge(rubric='coverage-explanation-v2', threshold=0.85),
])

Mitigations

  1. 01Retrieve grants, exclusions, conditions and endorsements together as one coverage bundle.
  2. 02Map each endorsement to the exclusion it modifies in a governed table.
  3. 03Use 'based on your policy documents, this may not be covered; an adjuster will confirm' language for service channels.
  4. 04Build cases where the insured's words match a covered peril but the facts fit an exclusion.

Regulations this maps to

Related failure modes

  1. 1Claim denial without policy basisInsurance · Disclosure & reasoningCritical
  2. 2FNOL required field dropInsurance · OmissionHigh
  3. 3State claim deadline miscalculationInsurance · Jurisdiction & currencyHigh

Questions about this failure

Should a service agent ever answer coverage questions?

Many carriers let agents explain policy language while routing determinations to adjusters. The suite enforces whichever policy you choose and checks that explanations include the exclusions that apply.

Is the HO-3 wording on this page exact?

It paraphrases common homeowners form language. Forms vary by carrier, edition and state, so packs are built from your filed forms. This page is not legal advice.

Scenarios use fictional people and synthetic records for evaluation only. This entry is not legal or financial advice.

Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the index

Stop testing regulated agents on generic prompts.

Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.

Start free trialBook a technical demo