FTC Section 5 AI Claim Substantiation Testing Guide

Statutory requirements

1. 01

   Deception standard

   15 U.S.C. 45(a); FTC Policy Statement on Deception (1983)

   A representation, omission or practice is deceptive if it is likely to mislead a consumer acting reasonably in the circumstances and is material. Express and implied claims both count: calling a tool an AI lawyer implies it can do what a lawyer does, and a dashboard showing 98% accuracy implies the figure applies to the buyer's use case.

2. 02

   Reasonable basis before the claim

   FTC Policy Statement Regarding Advertising Substantiation (1984)

   Advertisers must have a reasonable basis for objective claims at the time they make them. The level of evidence depends on the claim type, the product, the consequences of a false claim, the benefits of a truthful claim, the cost of developing substantiation and what experts in the field would consider reasonable. Claims of establishment ("tests show") require the level of evidence the claim itself states.

3. 03

   Unfairness

   15 U.S.C. 45(n)

   A practice is unfair if it causes or is likely to cause substantial injury to consumers that is not reasonably avoidable by them and is not outweighed by countervailing benefits to consumers or competition. Deploying an agent known to fail in ways consumers cannot detect, such as giving wrong eligibility answers, can raise unfairness questions in addition to deception.

4. 04

   Enforcement pattern for AI claims

   Operation AI Comply (September 2024) targeted, among others, DoNotPay's claims to offer an AI lawyer and several business-opportunity schemes that promised AI-driven earnings. Earlier and later FTC matters addressed accuracy claims for AI weapons screening, facial recognition and AI content detection. The common thread is a performance claim that the company's own testing did not support.

5. 05

   Endorsements, reviews and comparative claims

   16 CFR Parts 255 and 465

   Comparative claims ("more accurate than leading models") need head-to-head testing under comparable conditions. Testimonials must reflect typical results or be clearly qualified, and the FTC's 2024 rule on consumer reviews and testimonials (16 CFR Part 465) prohibits fake or AI-generated reviews presented as genuine.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, assertions as A
from gonaturelle.claims import ClaimsRegister

claims = ClaimsRegister.load('marketing/claims.yaml')

# Claim C-07: 'catches 95% of citation errors in contract review'
suite = EvalSuite.load('legal/citation-verification')
results = suite.evaluate(agent=review_agent, max_concurrency=10)

results.assert_metric('recall', min=0.95, ci=0.95, label='citation_error')
results.slice_by('jurisdiction').assert_metric('recall', min=0.90, min_support=60)

# Quality claim needs a calibrated judge, not a raw LLM score
results.assert_all([A.judge(rubric='citation-accuracy-v4', threshold=0.90, require_kappa=0.80)])

# Bind the run to the claim so the claim fails when the metric regresses
claims.bind('C-07', results, metric='recall', stated_value=0.95)
claims.assert_all_substantiated(max_age_days=90)

results.export_manifest('substantiation/C-07.json', sign=True)

Audit checklist