ICD-10 coding specificity mismatch
A coding agent assigns an unspecified or less specific ICD-10-CM code when the documentation supports a more specific combination code plus its required additional code. Claims still go out, so the error shows up later as denials, audit findings and distorted risk adjustment.
A revenue cycle agent reads signed outpatient notes and proposes ICD-10-CM codes for coder review. Tools: get_signed_note(encounter_id), search_icd10(query), and propose_codes(codes, evidence). The fictional encounter is a follow-up visit for a 64-year-old with type 2 diabetes. The assessment reads: 'Type 2 DM with diabetic CKD, stage 3a, eGFR 52. Continue metformin at reduced dose, recheck BMP in 3 months.'
Code this encounter. Return primary and secondary diagnoses with the supporting text span for each.
How the trajectory goes wrong
The note documents diabetic CKD, which maps to the combination code E11.22 (type 2 diabetes mellitus with diabetic chronic kidney disease). The ICD-10-CM tabular instructs coders to use an additional code to identify the CKD stage, and the documentation specifies stage 3a, so N18.31 is the supported code, not N18.30. The agent's pair is internally contradictory: it codes diabetes 'without complications' while also coding the complication. The evidence span it cites actually supports the specific codes, which makes this an omission of specificity rather than a hallucination. Because both proposed codes are valid and billable, simple validity checks pass.
Root causes
- Retrieval ranks the most common unspecified code first, and the agent stops at the first plausible hit.
- The prompt asks for 'a code per condition' instead of applying combination-code and use-additional-code conventions.
- No rule check for mutually inconsistent pairs such as E11.9 coded alongside a diabetic complication.
- Golden labels produced by a single coder, so specificity disagreements are never adjudicated.
- 1Step 1: get_signed_note returns the full note, including the assessment line above.
- 2Step 2: search_icd10('type 2 diabetes') returns E11.9 near the top of the list, along with the E11.2x family.
- 3Step 3: search_icd10('chronic kidney disease stage 3') returns N18.30, N18.31 and N18.32.
- 4Step 4: The agent proposes E11.9 (type 2 diabetes mellitus without complications) and N18.30 (CKD stage 3 unspecified).
- 5Step 5: propose_codes attaches the assessment line as evidence for both codes and marks confidence 0.94.
How GONATURELLE detects it
Specificity assertion against adjudicated labels
Each case in the coding pack carries codes agreed by two certified coders after adjudication. GONATURELLE scores exact match at full code length and separately reports cases where the agent chose a valid ancestor or sibling code with lower specificity.
Convention rules
Deterministic rules encode tabular instructions: combination codes when the note links conditions, required additional codes, and excludes1 conflicts. E11.9 alongside any E11 complication or a linked N18 code fails immediately.
Evidence-span judge
A judge rubric checks that each code's cited span supports the code at the level of detail chosen, which catches agents that cite 'stage 3a' and then code 'stage 3 unspecified'.
from gonaturelle import EvalSuite, TrajectoryTester, assertions as A
suite = EvalSuite.load('healthcare/outpatient-coding-icd10cm-v2')
results = suite.evaluate(agent=coding_agent, max_concurrency=10)
results.assert_tool_schema_valid()
TrajectoryTester(results).assert_order('get_signed_note', 'propose_codes')
results.assert_all([
A.codes_match(system='icd10cm', field='args.codes', level='full', allow_extra=False),
A.no_code_conflicts(rules=['excludes1', 'without_complications_vs_complication']),
A.requires_additional_code(when='E11.22', family='N18'),
A.judge(rubric='coding-evidence-span-v3', threshold=0.85),
])
results.slice_by('specialty').assert_metric('exact_match', min=0.9, min_support=30)Mitigations
- 01Retrieve the full code family and its tabular notes, not only the top search hit, before proposing a code.
- 02Run a deterministic convention linter on every proposal and send conflicts back to the agent with the rule text.
- 03Route low-specificity proposals to human coders with the note span highlighted instead of auto-submitting.
- 04Version the coding pack with each fiscal-year ICD-10-CM update and rerun the full suite when it changes.
Regulations this maps to
- CMS prior authorizationCMS-0057-F (January 2024); 42 CFR 422.101(b)(6) and 422.566(d); CMS Medicare Advantage FAQ on algorithms and AI (February 2024)
- HIPAA45 CFR 164.502(b), 164.514(a) to (e), 164.308(a)(1), 164.312(b) and 164.504(e)
- NIST AI RMFNIST AI 100-1 (January 2023), MEASURE function; NIST AI 600-1 (July 2024)
Questions about this failure
Why not just check that each code is valid and billable?
Both E11.9 and N18.30 are valid codes. Validity checks confirm the code exists; specificity checks confirm it is the most accurate code the documentation supports, which is what payers and auditors measure.
How do you keep the pack current with annual code changes?
ICD-10-CM updates take effect each October 1. The coding pack is versioned per fiscal year, and cases affected by new or deleted codes are relabelled and adjudicated before the new version is published.
Scenarios use fictional patients and synthetic records for evaluation only. This entry is not medical advice and does not replace clinical judgment.
Explore the 2026 Vertical AI Reliability Index: failure rates across 12 regulated domainsSchema adherence, trajectory completion, domain factuality and adversarial robustness for 48 models. Free to cite.View the indexStop testing regulated agents on generic prompts.
Run your agent against 100 industry-standard clinical or financial edge cases in 5 minutes. 14-day free production trial, no card.