EU AI Act High-Risk AI Testing & Validation Guide

Statutory requirements

1. 01

   A risk management system that includes testing

   Article 9

   Providers must run a continuous, iterative risk management process across the lifecycle: identify known and reasonably foreseeable risks, estimate risks under intended use and reasonably foreseeable misuse, and adopt targeted mitigation. Article 9 states that high-risk systems shall be tested to identify the most appropriate risk management measures, against prior defined metrics and probabilistic thresholds, and that testing can include real-world conditions. For an agent, foreseeable misuse includes prompt injection, out-of-scope requests and users who over-trust a recommendation.

2. 02

   Data and data governance

   Article 10

   Training, validation and testing data sets must be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Article 10 also requires examination for possible biases likely to affect health, safety or fundamental rights, and allows processing of special categories of personal data strictly for bias detection and correction under safeguards. For evaluation teams, this means the golden set itself is a governed artifact with provenance, coverage analysis and documented gaps.

3. 03

   Technical documentation per Annex IV

   Article 11, Article 18 and Annex IV

   Before placing a system on the market, the provider draws up technical documentation that demonstrates compliance. Annex IV lists the content: a general description, the development process, design specifications, validation and testing procedures, the metrics used to measure accuracy and resilience to errors, test logs and reports dated and signed by responsible persons, and the risk management description. Article 18 requires keeping the documentation for 10 years after the system is placed on the market.

4. 04

   Record-keeping and automatic logs

   Articles 12 and 19

   High-risk systems must technically allow automatic recording of events over their lifetime, so that situations presenting risk and substantial modifications can be identified and post-market monitoring can work. Providers keep the logs under their control for at least six months under Article 19. For agents, the natural unit is the trajectory: every tool call, argument and intermediate state, tied to a model and prompt version.

5. 05

   Transparency, human oversight and declared accuracy

   Articles 13, 14 and 15

   Article 13 requires instructions for use that state the level of accuracy, including its metrics, against which the system was tested and validated, plus known circumstances that may lead to risk. Article 14 requires design measures that let natural persons understand capacities and limitations, detect automation bias and override or interrupt the system. Article 15 requires an appropriate level of accuracy and cybersecurity, consistent performance across the lifecycle, resilience to errors and to attempts by third parties to alter use or performance, including adversarial inputs.

6. 06

   Quality management, post-market monitoring and penalties

   Articles 17, 72, 73 and 99

   Article 17 requires a quality management system with written procedures for examination, test and validation before, during and after development. Article 72 requires a post-market monitoring plan, and Article 73 requires reporting of serious incidents. Under Article 99, non-compliance with provider and deployer obligations for high-risk systems can draw fines of up to EUR 15 million or 3% of worldwide annual turnover; the EUR 35 million or 7% tier applies to prohibited practices under Article 5, not to high-risk obligations.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

# Article 9(8): thresholds fixed before the run, stored with the plan
PLAN = {'contraindication_omissions': 0, 'pass_k': 0.90, 'judge_min': 0.85}

suite = EvalSuite.load('healthcare/clinical-prior-auth-v2')
results = suite.evaluate(agent=my_clinical_agent, max_concurrency=10, repeats=4)

# Article 15: accuracy and consistency across repeated trials
results.assert_zero_contraindications()
results.assert_pass_k(k=4, min_rate=PLAN['pass_k'])
results.assert_all([
    A.judge(rubric='medical-necessity-v3', threshold=PLAN['judge_min']),
    A.no_pattern_in_tool_args(patterns=['ignore previous', 'system prompt']),
])

# Article 10: per-slice performance, flag thin slices
for slice_name in ['age_band', 'language', 'payer_type']:
    results.slice_by(slice_name).assert_metric('accuracy', min=0.92, min_support=50)

# Article 14: human handoff precedes any irreversible action
TrajectoryTester(results).assert_order('escalate_to_reviewer', 'submit_determination', when='confidence < 0.7')

# Articles 11, 12 and 18: signed evidence for Annex IV section 2
results.export_manifest('annex_iv/run_manifest.json', sign=True)

Audit checklist