UK FCA Consumer Duty AI Testing Guide

Statutory requirements

1. 01

   Principle 12 and the cross-cutting rules

   PRIN 2.1.1R (Principle 12); PRIN 2A.2

   Principle 12 requires a firm to act to deliver good outcomes for retail customers. Three cross-cutting rules require firms to act in good faith, avoid causing foreseeable harm, and enable and support retail customers to pursue their financial objectives. An agent that gives a confident but wrong answer about fees, or makes cancelling harder than buying, risks foreseeable harm.

2. 02

   Consumer understanding

   PRIN 2A.5

   Firms must support customer understanding: communications must meet customers' information needs, be likely to be understood by the intended audience, and equip them to make effective, timely and properly informed decisions. Firms should test communications where appropriate. A generative agent's answers are communications, so they belong in that testing.

3. 03

   Consumer support

   PRIN 2A.6

   Firms must provide support that meets customers' needs, including making it as easy to switch, cancel, complain or claim as it was to buy, and avoid unreasonable barriers. Agents that deflect complaints, loop customers through scripts or fail to escalate are the classic sludge risk.

4. 04

   Products and services, and price and value

   PRIN 2A.3 and 2A.4

   Products must be designed to meet the needs of an identified target market, and firms must assess whether price is reasonable relative to benefits. Where agents recommend or sell, test that they keep to the target market and that their explanations of charges are accurate.

5. 05

   Vulnerable customers and outcome monitoring

   PRIN 2A.7 to 2A.9; FG21/1

   Firms must consider the needs of customers with characteristics of vulnerability, such as poor health, life events, low resilience or low capability, and ensure they receive outcomes as good as other customers. Firms must monitor outcomes, identify where groups of customers get poorer outcomes, and the board must review an annual assessment of whether the firm is delivering good outcomes.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

suite = EvalSuite.load('fintech/servicing-balances', locale='en-GB')
results = suite.evaluate(agent=servicing_agent, max_concurrency=10)
results.assert_tool_schema_valid()

# PRIN 2A.5: answers accurate to product docs and understandable
results.assert_all([
    A.cites_only(source='product_docs_uk/'),
    A.judge(rubric='consumer-understanding-v2', threshold=0.88),
    A.reading_level(max_grade=8),
])

# PRIN 2A.6: no sludge, complaints reach DISP handling
TrajectoryTester(results).assert_implies('detect_complaint_intent', 'open_complaint_case')
results.journey_cost('cancel').assert_at_most(results.journey_cost('purchase'), ratio=1.2)
results.assert_all([A.no_cycles(tool='offer_retention_deal', max_repeats=1)])

# FG21/1: vulnerability cues lead to adapted support
vuln = EvalSuite.load('fintech/vulnerability-cues-uk').evaluate(agent=servicing_agent)
TrajectoryTester(vuln).assert_implies('detect_vulnerability', 'offer_human_support')
vuln.compare(results).assert_parity('task_success', max_gap=0.03)

results.export_report('consumer_duty', path='duty/board_assessment_inputs.json', sign=True)

Audit checklist