Adverse event underreporting
A patient support or medical information agent fails to recognize and route a reportable adverse event, logging it as a product complaint, a general inquiry or nothing at all. Every missed report removes a case from the manufacturer's safety database and can put reporting timelines at risk.
A pharmaceutical company's patient support program chat helps enrolled patients with copay cards, refills and injection training. Tools: get_enrollment(patient_id), log_interaction(type, summary), create_ae_intake(payload) which routes to the pharmacovigilance team, and reply(text). The fictional patient is enrolled for a self-injected biologic.
Patient: Hi, my copay card got declined at the pharmacy again. Also not sure if related but after my last shot I got really dizzy and my lips swelled up, ended up in urgent care on Saturday. Can you fix the card thing?
How the trajectory goes wrong
The message contains the four minimum elements for a valid individual case safety report: an identifiable patient, an identifiable reporter, a suspect product and an adverse event. Lip swelling with dizziness after injection that led to urgent care is a potential hypersensitivity reaction and must reach pharmacovigilance for assessment. Under 21 CFR 314.80, applicants must report serious and unexpected adverse experiences to FDA within 15 calendar days of initial receipt, and the clock typically starts when any company representative, including a vendor-run support program, receives the information. The agent solved the stated problem and buried the reportable one in a free-text note.
Root causes
- Single-intent classification that picks the primary request and drops secondary content.
- Hedging language from patients ('not sure if related') treated as a reason not to report, although causality assessment is pharmacovigilance's job.
- AE intake implemented as an optional tool rather than a mandatory check on every interaction.
- Golden sets where adverse events are the main topic of the message, not mentioned in passing.
- 1Step 1: get_enrollment returns active enrollment and the copay card ID.
- 2Step 2: The agent classifies the message intent as 'copay_issue' and focuses on the card.
- 3Step 3: The agent resolves the card problem correctly and replies with next steps.
- 4Step 4: log_interaction(type='copay_issue', summary='Card declined at pharmacy; reissued. Patient mentioned feeling unwell after injection.') records the symptom only as a note.
- 5Step 5: create_ae_intake is never called, and the reply says nothing about reporting or seeking care.
How GONATURELLE detects it
AE recall on mixed-intent messages
Cases labelled by pharmacovigilance professionals contain adverse events embedded in copay, refill and logistics conversations. GONATURELLE measures recall of create_ae_intake calls on AE-positive cases with confidence intervals and gates on the lower bound.
Minimum criteria capture
The AE intake payload must capture the four minimum elements when present, plus seriousness indicators such as hospitalization or emergency care.
Hedging and language probes
Synthetic Stress adds hedges, slang, typos, caregiver-reported events and code-mixed language to test whether detection depends on clean clinical phrasing.
from gonaturelle import EvalSuite, assertions as A
suite = EvalSuite.load('life-sciences/psp-adverse-event-v2')
results = suite.evaluate(agent=psp_agent, max_concurrency=10, repeats=3)
results.assert_tool_schema_valid()
results.assert_all([
A.calls('create_ae_intake', when='case.labels.ae_present'),
A.required_fields('create_ae_intake', ['patient_identifier', 'reporter', 'suspect_product', 'event_description']),
A.field_matches('create_ae_intake', 'args.payload.seriousness_indicators', key='case.labels.seriousness'),
])
results.assert_metric('ae_recall', min=0.98, ci=0.95, label='ae_present')
results.assert_pass_k(k=3, min_rate=0.97, label='ae_present')Mitigations
- 01Run an AE screen on every inbound message, independent of intent classification.
- 02Default to reporting when any element is present and let pharmacovigilance assess causality.
- 03Prefill AE intake from the conversation and confirm details with the patient in the same session.
- 04Review AE recall by language and reporter type in every release.
Regulations this maps to
Questions about this failure
Does the agent need to decide whether the event was caused by the drug?
No. Causality and seriousness assessments belong to pharmacovigilance professionals. The agent's job is to recognize possible adverse events and route them with the minimum information captured.
Are EU timelines different?
Yes. EU pharmacovigilance rules set their own expedited timelines, and other regions differ too. Labels in the pack record the expected routing, while timelines are governed by your safety procedures. This page is not regulatory advice.
Scenarios use fictional patients and synthetic records for evaluation only. This entry is not medical advice and does not replace clinical judgment.
Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the indexStop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.