Methods Supplement to Support Innovative Research on AI and Large Language Models (LLMs) in Patient-Centered CER

Structured clinical data, typically derived from claims or electronic health records (EHR), most closely reflects operational activity, in particular activity that is reimbursable or expends health system resources. These data record actions and results, often in granular detail, but provide limited information about attribution, and may not accurately reflect medical reasoning. Narrative clinical notes are potentially better suited for recording clinical decision-making and planning, and are often richer sources for clinical concepts that do not drive specific utilization (e.g., symptom burden or quality of life), as well as for attribution. However, details may be elided in notes, representation of concepts is not standard, and sentiment can be difficult to discern. In the ideal case, structured and narrative data will be concordant with the major concepts related to the patient’s care. Unfortunately, this is often not the case in practice, and researchers draw on anecdotal experience or limited chart review data to estimate their differences. 
Natural language processing (NLP) methods offer an attractive way to extract features from notes on a large scale. However, existing solutions have limited transfer learning ability to handle new documenting patterns. Therefore, researchers must spend significant effort to annotate local datasets and train NLP models to recognize the new documentation patterns, one of the major barriers to adopting clinical text analytics. Recent breakthroughs in generative large language models (LLMs) and prompt-based learning algorithms provide a major opportunity to improve the portability of clinical NLP. Compared to traditional models, LLMs perform better in few-shot learning (learning from very few labeled samples) and zero-shot learning (learning without labeled samples for the target application). In the clinical domain, LLMs trained on very large corpora and able to respond to a wide range of prompts may improve the ability to detect relevant clinical concepts in notes. In particular, LLMs may provide informative responses for disease activity/severity, concepts often based on a synthesis of multiple facts. In addition, the generality of LLMs may make it possible to incorporate them into studies without needing the resources and expertise for de novo training of NLP models. However, LLMs can still potentially miss concepts not encountered in training, and are subject to confabulation or “hallucination”, in some responses inventing information that is untrue.  
Two LLMs, in particular, have recently generated considerable interest. ChatGPT is widely recognized as a leader in commercial LLMs, is now offered by Microsoft in their Azure cloud, and is actively used by healthcare organizations. The Llama-2 model has been released for public use by Meta for research and non-commercial applications. Both LLMs afford methods to tune or supplement the model training to improve its accuracy with novel materials and methods to reduce the likelihood of hallucination. 
Despite imperfect concordance, the complementarity of structured and narrative data allows each to provide a meaningful quality assessment of the other. For this to be successful, appropriate methods must be developed to interrogate each source of information. In addition, a systematic comparison of the two across a range of patient outcomes will provide an evidence base to develop expectations of external validity when designing DQA for studies.