Systematic Data Fitness Assessment Improves Validity and Replicability of Research Using Real-World Data [Preprint]

Abstract

Research replication in real-world data is essential to build trust in evidence from clinical studies. However, methods for conducting and reporting these efforts are lacking, particularly related to data fitness and limitations. We demonstrate the importance of incorporating systematic fitness testing by replicating a single-center observational study of hydroxyurea for children with severe sickle cell disease (SS/Sβ0) in a multi-institutional learning network using EHR data (PEDSnet). An AS-IS arm applied the original study’s criteria with no major data quality adjustments, while a Data Fitness Enhanced (DFE) arm used systematic data fitness assessment to inform adjustments to cohort eligibility criteria and variable definitions; both arms then replicated the original study’s primary analyses. Data quality checks in the DFE arm refined cohort accuracy and improved hydroxyurea capture, drug era computation, and hematology specialist mapping. The DFE cohort produced average treatment effects with higher face validity and greater concordance with the original study (e.g., change in ED visits: -0.44 (CI -0.60, -0.26) versus -0.36 (CI -0.57, -0.16) in the original study) than the AS-IS cohort (-0.08 (CI -0.26, 0.09)), which yielded several implausible results. These findings show that superficially plausible cohort characteristics do not guarantee valid results without transparent, systematic data fitness assessment.