Augmenting Structured Diagnoses Through Effective Use of Pre-Trained Large Language Models on Clinical Notes

Abstract

Objective: Clinical narrative provides a unique window into provider reasoning and attribution for automated diagnosis assignment, but large language models (LLMs) have traditionally not performed well at medical coding. We evaluate a reproducible method for automated diagnosis assignment using LLMs in clinical notes and compare with structured diagnoses. Materials and Methods: We used GPT-OSS for prompt engineering and task segmentation to create a model that extracts ICD-10-CM diagnoses, with estimates of severity, currency, and importance, from progress notes. We assessed performance across multiple cohorts of patients aged 0-21 years. For each, 100 outpatient provider notes were selected across levels of severity, along with coded diagnoses from that visit (electronic health record [EHR]); a subset of 130 notes were subjected to clinical expert review. Results: Comparison showed 18.7% exact code and 33.3% ICD-10-CM category match between EHR and LLM, but semantic similarity of 0.93 at the category level. Compared to expert review, LLM precision was 0.84 and recall 0.49 for exact matches, and 0.92 and 0.62, respectively, for category-level matching. In contrast, coded diagnoses showed slightly higher precision (0.94 for both cases) and substantially lower recall (0.27 and 0.43) versus expert review. Codes not identified by the LLM were more often rated by the reviewer as lower importance or certainty. Discussion: We demonstrate a reusable approach to optimize LLMs for use in diagnosis extraction from clinical notes that can augment structured diagnoses and provide contextualizing metadata. Conclusion: LLMs represent a viable and flexible approach to diagnosis code extraction from unstructured clinical notes.