How a regional bank's model risk team turned CI runs into adverse action evidence
A US regional bank uses an agent to assemble small-business credit memos and draft adverse action reasons. Its model risk and compliance teams now review evidence packs exported from the same CI runs that gate each change.
- Customer
- US regional bank, small-business underwriting agent
- Published
- 4 Aug 2026
- Stack
- OpenAI Agents SDKAzure OpenAIInternal scoring serviceGitLab CISingle-tenant VPC on AWSOkta SSO
- to produce the adverse action reason-fidelity evidence pack for model risk review, previously assembled by hand
- 1 export
- to produce the adverse action reason-fidelity evidence pack for model risk review, previously assembled by hand
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- reason-misalignment failures caught before merge between onboarding and Day 120
- 41
- reason-misalignment failures caught before merge between onboarding and Day 120
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- model change validation cycle for prompt-only changes
- 6 weeks to 8 days
- model change validation cycle for prompt-only changes
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
The problem
Before onboarding, as described by the team.
The bank's underwriting agent reads tax returns, bank statements and bureau data, calls an internal scoring model, and drafts a credit memo plus the principal reasons for any decline. Under ECOA and Regulation B, and as the CFPB spelled out in Circular 2023-03 before withdrawing it in May 2025, the reasons on an adverse action notice must be specific and must reflect the factors that actually drove the decision.
The bank's model risk management function treated each prompt change as a model change under its model risk policy (then aligned to SR 11-7, since replaced by SR 26-2). Validation meant an analyst re-running a fixed set of files, comparing memos by hand and writing up the result. A prompt change took about six weeks to clear, so the AI team batched changes, which made each validation larger and riskier.
The specific failure the compliance team worried about was reason misalignment: the agent citing limited time in business while the scoring model's top factor was debt service coverage. Text similarity metrics scored those memos as near-perfect because the language was fluent and on-topic.
Asserting that stated reasons match model factors
The fintech/adverse-action-reg-b pack (version 2.1.0) supplied 1,960 cases labelled by former bank examiners and CPAs. The team added 380 BYOD cases from historical declined applications with personal data removed on ingest.
Instead of scoring the memo prose, the core assertion compares the reasons in the drafted notice with the ranked factors returned by the scoring tool call. Because the comparison reads the trajectory, it catches the case where the agent ignored or reordered the model output.
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
suite = EvalSuite.load('fintech/adverse-action-reg-b', extra=['byod/sb-declines-2025'])
results = suite.evaluate(agent=underwriting_agent, max_concurrency=8)
TrajectoryTester(results).assert_order('fetch_bureau_report', 'score_application', 'draft_notice')
results.assert(A.reasons_match_factors(
reasons_from='draft_notice.reasons',
factors_from='score_application.top_factors',
top_k=4,
))
results.assert(A.never_calls('approve_override'))
results.assert(A.judge(rubric='reg-b/specific-reasons', threshold=0.95))
results.export_report(format='model-governance', path='artifacts/mrm-pack')Evidence in the format model risk already used
Nandini Joshi's regulatory science team mapped the compliance report template to the sections the bank's validators already wrote: scope, data, conceptual soundness, outcomes analysis and ongoing monitoring. Each gated run exports a pack with the signed manifest, case-level results and judge agreement statistics, which validators review instead of rebuilding.
The bank runs GONATURELLE in a single-tenant VPC in its own AWS account, so applicant data never leaves the bank's environment.
Results at Day 60 and Day 120
- Between onboarding and Day 120 the gate blocked 41 pull requests or nightly runs for reason misalignment. In 29 of them the final memo scored within two points of baseline on the old similarity metric.
- Prompt-only changes now clear model risk review in about eight days, down from six weeks, because validators start from the exported pack rather than a blank document.
- The bank's compliance head describes the evidence pack as a one-click export from the CI run. The bank still makes its own validation and filing decisions; GONATURELLE supplies the test evidence.
- Figures are customer-reported at the Day 120 review.
“We generated the full adverse action evidence pack for our model risk committee in one click from the CI run that gated the change. Validators review it instead of rebuilding it.”
More customer stories
Show all- Series B clinical documentation companyHow a clinical documentation team gave its safety board the evidence to approve an ambient scribe agentHealth
- Legal research copilot companyHow a legal research copilot stopped a broken release in week two and wired the gate in an afternoonLegal
- Top-20 US commercial insurerHow a top-20 US commercial insurer took prompt regressions in claims triage to zeroClaims
Figures are reported by the customer against a baseline captured before onboarding and approved for publication. GONATURELLE provides automated test evidence; it does not certify compliance, and the customer remains responsible for clinical, credit, legal and claims decisions and for regulatory filings.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.