Development, content validation, and reliability of the Assessment of Real-World Observational Studies (ArRoWS) critical appraisal tool

The objective was to develop and test a pragmatic critical appraisal tool, the Assessment of Real-World Observational Studies (ArRoWS), to quickly and easily assess the quality of real-world evidence studies using electronic health records. The initial ArRoWS tool was developed by identifying items frequently found in existing validated assessment instruments and adapting these items to specifically assess real-world evidence studies. The tool was revised based on recommendations from an expert panel of 14 senior academic individuals specializing in epidemiology and content validity was measured. During March 2018–January 2019, 47 large, observational studies related to cardiometabolic medicine were identified through a search algorithm and assessed by three pairs of raters using the ArRoWS tool. The final version of the ArRoWS had 16 items including nine core items and seven study design-specific items with item-specific content validity indexes ranging from 0.64 to 1.00. The scale-level content validity index of the ArRoWS appraisal tool was 0.91. When the ArRoWS tool was pilot tested, the observed agreement between assessor pairs on whether the study provided high-quality real-world evidence was 85.7%, 68.8%, and 58.8%. The prevalence adjusted bias-adjusted kappa for the assessor pairs was 0.71, 0.38, and 0.18. The ArRoWS is a simple tool to standardize the assessment of real-world evidence studies.