SR 11-7 Model Risk Management LLM Validation Guide

Statutory requirements

1. 01

   What SR 11-7 established

   SR 11-7 (April 2011), sections III to VI

   SR 11-7 defined a model as a quantitative method that processes input data into quantitative estimates, with input, processing and reporting components, and defined model risk as the potential for adverse consequences from decisions based on incorrect or misused model outputs. It required sound development and use, validation, and governance, with effective challenge by objective, competent parties. Many bank model risk policies still use this structure.

2. 02

   The three validation elements

   SR 11-7, section V

   SR 11-7 named three core elements of validation: evaluation of conceptual soundness (design, theory, data and assumptions), ongoing monitoring (process verification and benchmarking to confirm the model works as intended after implementation) and outcomes analysis (comparing outputs to actual outcomes, including back-testing). Each translates to agents: design review of prompts and tools, regression monitoring, and comparison of agent decisions to expert-labelled ground truth.

3. 03

   What SR 26-2 changed

   SR 26-2; OCC Bulletin 2026-13; FDIC FIL-15-2026

   The 2026 guidance is principles-based and organized around materiality, narrows the definition of a model (commentators note deterministic rule-based processes and simple calculations are out), and states it is guidance rather than an enforceable rule, while supervisory action remains possible for unsafe or unsound practices or violations of law. It also withdrew related guidance, including the Comptroller's Handbook model risk booklet and the 2021 BSA/AML model risk statement.

4. 04

   Generative and agentic AI outside the new framework

   SR 26-2, scope

   The agencies excluded generative and agentic AI from SR 26-2 as novel and rapidly evolving, directing banks to apply their broader risk management and governance practices, and said they would seek public input on AI model risk separately. Commentators differ on edge cases such as retrieval-augmented systems. Traditional machine learning models, such as fraud classifiers, remain in scope.

5. 05

   Obligations that still bind

   Excluding LLM agents from model risk guidance does not exclude them from law. Fair lending, UDAAP, adverse action, privacy, BSA/AML, third-party risk management guidance and safety-and-soundness standards still apply to whatever the agent does, and examiners still ask how the bank knows the agent works.

Evaluation methodologies

Testing artifacts to retain

The evidence folder an examiner, notified body or internal auditor will ask for. GONATURELLE compliance reports export most of it from signed runs.

GONATURELLE assertion rules

Each assertion maps to a requirement above. Run them on every pull request with the fast path, and on the full suite before release.

from gonaturelle import EvalSuite, TrajectoryTester, assertions as A

suite = EvalSuite.load('fintech/kyc-aml-triage')
champion = suite.evaluate(agent=aml_agent, max_concurrency=10, repeats=5)
challenger = suite.evaluate(agent=aml_agent_alt_model, max_concurrency=10, repeats=5)

# Conceptual soundness: design assumptions tested as assertions
champion.assert_tool_schema_valid()
TrajectoryTester(champion).assert_order('screen_sanctions', 'clear_alert')
champion.assert_all([
    A.never_calls('clear_alert', when="risk_tier == 'high' and not reviewer_assigned"),
    A.no_hallucinated_args(tools=['file_sar_draft']),
])

# Outcomes analysis against expert labels, with nondeterminism exposed
champion.assert_pass_k(k=5, min_rate=0.90)
champion.assert_metric('escalation_recall', min=0.97, ci=0.95)

# Benchmarking: flag slices where champion and challenger disagree
champion.compare(challenger).assert_disagreement(max_rate=0.08, by='alert_type')

champion.export_report('model_validation', path='mrm/kyc_agent_v3.json', sign=True)

Audit checklist