ISO/IEC 42001 AI Management System Testing Guide

Statutory requirements

1. 01

   AI risk assessment and treatment

   Clauses 6.1.2 and 6.1.3

   The organization defines and applies an AI risk assessment process that identifies risks that could prevent achieving its AI objectives, analyzes their consequences and likelihood, and evaluates them against risk criteria. Risk treatment selects controls, compares them with Annex A to make sure nothing necessary is omitted, and produces a Statement of Applicability.

2. 02

   AI system impact assessment

   Clause 6.1.4 and Annex A.5

   A separate process assesses potential consequences of AI systems for individuals, groups and societies, considering the technical and societal context. For a clinical or credit agent, the impact assessment is where harms to patients or applicants are named, and those harms should map to test suites.

3. 03

   Verification and validation

   Annex A.6.2.4

   Annex A expects the organization to define and document verification and validation measures for the AI system and to specify criteria for their use, as part of a documented lifecycle that also covers requirements, design, deployment, operation and monitoring. Evaluation criteria set in advance and evidence that each release met them is what an auditor samples.

4. 04

   Operation, monitoring and event logs

   Annex A.6.2.6 and A.6.2.8

   Controls cover operation and monitoring of AI systems, including repairs, updates and support, and determining at which lifecycle phases event logging is enabled. For agents, trajectory logs are the natural event record, and their retention and access policies belong in the AIMS.

5. 05

   Data quality, monitoring, internal audit and improvement

   Annex A.7.4; clauses 9.1, 9.2, 9.3 and 10

   The standard requires data quality requirements for data used in AI systems, monitoring and measurement of the management system's performance, internal audits at planned intervals, management review, and corrective action on nonconformities. A recurring regression suite with tracked failures is direct evidence for clauses 9 and 10.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
from gonaturelle.governance import Objectives

objectives = Objectives.load('aims/objectives.yaml')  # OBJ-03, OBJ-07 ...

suite = EvalSuite.load('insurance/coverage-exclusions')
results = suite.evaluate(agent=coverage_agent, max_concurrency=10)

# A.6.2.4: criteria defined in advance, linked to AI objectives
results.assert_tool_schema_valid()
results.assert_metric('exclusion_recall', min=objectives['OBJ-03'].threshold)
TrajectoryTester(results).assert_order('fetch_policy_form', 'cite_exclusion')
results.assert_all([A.cites_only(source='policy_forms/'), A.judge(rubric='coverage-reasoning-v2', threshold=0.88)])

# A.6.2.8: event logs retained per policy
results.assert_logs_retained(policy='aims/log_policy.yaml')

# Clause 10: failed cases become nonconformities with owners
results.failures().open_nonconformities(project='AIMS', assignee='coverage-ai-team')
results.export_report('iso42001', path='aims/vv_evidence.json', sign=True)

Audit checklist