How a clinical documentation team gave its safety board the evidence to approve an ambient scribe agent
A Series B company selling ambient documentation to multi-specialty groups needed its clinical safety board to sign off on a new agent that drafts SOAP notes and reconciles medication lists. Golden datasets and trajectory assertions replaced a two-day manual review before every release.
- Customer
- Series B clinical documentation company
- Published
- 18 Jun 2026
- Stack
- LangGraphAnthropicAWS BedrockGitHub ActionsFHIR R4 EHR integrationOpenTelemetry
- fabricated-finding regressions reaching production between Day 60 and Day 120
- 0
- fabricated-finding regressions reaching production between Day 60 and Day 120
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- pre-release clinical review, now a full-suite nightly run plus a fast-path PR gate
- 2 days to 9 min
- pre-release clinical review, now a full-suite nightly run plus a fast-path PR gate
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- Cohen's kappa between the calibrated note judge and the company's physician reviewers
- 0.84
- Cohen's kappa between the calibrated note judge and the company's physician reviewers
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
- medication reconciliation tool-order bugs caught in CI before merge by Day 60
- 3
- medication reconciliation tool-order bugs caught in CI before merge by Day 60
- Customer-reported against pre-onboarding baseline (Day 60 / Day 120 reviews)
The problem
Before onboarding, as described by the team.
The company's agent listens to a visit, drafts a SOAP note, and calls a medication reconciliation tool against the practice's EHR before suggesting changes to the medication list. The engineering team of 34 shipped prompt and model changes weekly. Before each release, two physicians on the clinical staff read roughly 120 generated notes in a spreadsheet and flagged anything wrong.
That process had three problems. It sampled, so a regression in a rare specialty could pass. It looked only at the final note, so it could not see when the agent called reconcile_medications before it had parsed the allergy list. And it produced no artefact the clinical safety board could review: the board's question was not whether the notes looked right, but what had been tested, against whose labels, and how often the scorer agreed with a clinician.
The trigger was a near miss in staging. A conciseness change to the system prompt caused the agent to drop anticoagulants from the medication list in about one in forty long visits. Final-note readability scores improved, and the manual reviewers did not happen to sample an affected note.
Golden cases that matched the board's questions
The team started from the healthcare/soap-note-summarization pack (version 1.8.1) and added 640 BYOD cases drawn from de-identified visit transcripts. De-identification ran on ingest, and the company's own physicians double-labelled the BYOD cases using the same rubric as the pack, with GONATURELLE adjudicating disagreements.
Dr. Meera Iyer's clinical evaluation team worked with the company's CMO to map each board concern to a rubric dimension: fabricated findings, omitted active medications, allergy contraindications and wrong laterality. The board received the per-dimension agreement statistics before the first gated release.
Assertions on the reconciliation trajectory
The near miss was a path problem, so the team asserted on the path. Every case now checks that the allergy parser runs before reconciliation, that no medication appears in a tool argument unless it appeared in the transcript or the EHR response, and that anticoagulants present in the source survive into the note.
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
suite = EvalSuite.load('healthcare/soap-note-summarization', extra=['byod/scribe-visits-v4'])
results = suite.evaluate(agent=scribe_agent, max_concurrency=12)
results.assert_tool_schema_valid()
results.assert_zero_contraindications()
TrajectoryTester(results).assert_order('parse_allergies', 'reconcile_medications')
results.assert(A.no_hallucinated_args(tool='reconcile_medications', field='rxnorm_code'))
results.assert(A.judge(rubric='soap/fabricated-finding', threshold=0.98))
results.assert(A.preserves_entities(category='anticoagulant', source='transcript'))A gate the release process could depend on
The fast path runs 20 critical cases on every push through gonaturelle/gate-action@v2. The full suite of 2,850 cases runs nightly on main and on any pull request labelled clinical. The run manifest is attached to each release ticket, which is what the safety board now reviews.
Results at Day 60 and Day 120
- By Day 60 the fast path had blocked three pull requests that reordered or skipped the allergy parser. Each would have passed the old spreadsheet review because the final notes read correctly.
- By Day 120 the company reported no fabricated-finding regressions in production, and the clinical staff spent their review time on the failure clusters GONATURELLE surfaced instead of reading random samples.
- The calibrated note judge reached a kappa of 0.84 against the company's two physician reviewers on a held-out set of 400 notes, which the board accepted as the threshold for letting the judge gate merges on its own.
- All figures are customer-reported at the Day 60 and Day 120 reviews against a baseline captured before onboarding.
“Our safety board did not want a dashboard. They wanted to know what was tested, who labelled it and how often the scorer agreed with a physician. The golden datasets gave them that, and they signed off on deployment.”
More customer stories
Show all- US regional bank, small-business underwriting agentHow a regional bank's model risk team turned CI runs into adverse action evidenceFinTech
- Legal research copilot companyHow a legal research copilot stopped a broken release in week two and wired the gate in an afternoonLegal
- Top-20 US commercial insurerHow a top-20 US commercial insurer took prompt regressions in claims triage to zeroClaims
Figures are reported by the customer against a baseline captured before onboarding and approved for publication. GONATURELLE provides automated test evidence; it does not certify compliance, and the customer remains responsible for clinical, credit, legal and claims decisions and for regulatory filings.
Stop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.