Machine learning–assisted literature screening for a medication-use process–related systematic review

PURPOSE: This article summarizes a novel methodology of applying machine learning (ML) algorithms trained with external training data to assist with article screening for 2 annual review series related to the medication-use process (MUP) generally and the MUP in ambulatory care settings (ACMUP) specifically. As the literature review for these 2 series grew over time, it became essential for the authors to develop methods to be efficient while still capturing most of the relevant literature. The ML model can be used to predict whether search results are likely to be relevant or not relevant. Results least likely to be relevant can then be excluded without manual screening, allowing research teams to save time that would otherwise be spent reviewing a portion of the search results for inclusion. ML models require a large training dataset typically derived from the unclassified corpus. In this study, the authors demonstrate the efficacy of training the ML model using external training data, which is possible in scenarios such as a systematic review update or ongoing review series such as those for the MUP and ACMUP. SUMMARY: The authors ran 3 simulations using screening decisions from previous publications and in-process manuscripts for the MUP and ACMUP review series to test the efficacy of the approach. The simulations were compared to actual manual screening decisions made by the research teams to include or exclude articles using title and abstract text. For each simulation, the authors developed a training dataset using a sample of screening decisions from previous years to predict article relevance in an "unclassified" corpus. In this case, the screening decisions for the unclassified corpus were actually known, allowing us to calculate recall (percent of relevant articles captured) and time saved using the number of articles that would be excluded without manual review. Combined, the ML approach correctly labeled 187 of 192 relevant studies. The 3 simulations included 17,227 unique studies, and using ML the authors demonstrated that 13,201 studies could have been excluded without manual screening while still maintaining recall of relevant articles of 95% or greater. CONCLUSION: This novel approach is applicable to systematic reviews and ongoing review series, including those for the MUP and ACMUP. Pharmacists have a duty to review and incorporate best practices into their organizations to improve the efficiency and cost of care, optimally utilize technology, and reduce the potential for medication errors. This methodology will allow evidence syntheses for the MUP and other disciplines in pharmacy practice to be published more expeditiously by saving significant time during the article screening step.