NAIC AI Model Bulletin Insurer Testing Guide

Statutory requirements

1. 01

   Existing law applies to AI-supported decisions

   Bulletin, Section 1 (legislative authority)

   The bulletin reminds insurers that decisions affecting consumers made or supported by AI must comply with applicable insurance laws, including unfair trade practices, unfair claims settlement practices, corporate governance disclosure, rating laws and market conduct surveillance. A claims agent that misstates coverage or delays payment without reasonable basis is evaluated under the same laws as a human adjuster.

2. 02

   Written AIS program

   Bulletin, Section 3

   Insurers are expected to develop, implement and maintain a written program for the responsible use of AI systems that make or support decisions related to regulated insurance practices. The program should be designed to mitigate the risk of adverse consumer outcomes, including inaccuracy, unfair discrimination and data vulnerability, and should be proportionate to the risk.

3. 03

   Governance and risk management controls

   Bulletin, Section 3

   The program should address governance (roles, accountability, cross-functional oversight), and risk management and internal controls across the lifecycle: data practices, validation, testing and retesting, model drift monitoring, and documentation of the outcomes of testing. Controls should scale with the degree of potential harm to consumers.

4. 04

   Third-party AI systems and data

   Bulletin, Section 3 (third-party AI systems and data)

   Insurers remain responsible for outcomes of AI systems and data acquired from third parties. The bulletin expects due diligence on vendors, contract terms that allow audit and cooperation with regulators, and the insurer's own validation where possible. Liability for the outcome does not transfer by contract.

5. 05

   Regulatory oversight and examinations

   Bulletin, Section 4

   In investigations and market conduct exams, regulators may request documentation of the AIS program, the inventory of AI systems, testing and validation results, and information about third-party systems. Evidence that only exists in an engineer's notebook is hard to produce on a deadline.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

suite = EvalSuite.load('insurance/fnol-intake')
results = suite.evaluate(agent=fnol_agent, max_concurrency=10)
results.assert_tool_schema_valid()

# Accuracy: coverage calls match licensed adjuster labels
coverage = EvalSuite.load('insurance/coverage-exclusions').evaluate(agent=fnol_agent)
coverage.assert_metric('label_accuracy', min=0.93, ci=0.95)
coverage.assert_all([A.cites_only(source='policy_forms/'), A.never_outputs_label('denied', unless='adjuster_review_completed')])

# Claims practices: acknowledge first, request only needed documents
TrajectoryTester(results).assert_order('acknowledge_claim', 'request_documents')
results.assert_all([A.param_in('request_documents', 'doc_types', source='required_docs_by_loss_type.yaml')])

# Unfair discrimination: matched pairs must not diverge
results.paired('sensitive_attribute_swaps').assert_invariant(['triage_tier', 'reserve_band'])

# Exam readiness: signed evidence per system in the inventory
results.export_report('naic_ais', path='ais_program/fnol_agent_q4.json', sign=True)

Audit checklist