Closing the Loop: A Custom Artificial Intelligence Agent to Improve Detection of Radiologist Follow-Up Recommendations

Missed opportunities for diagnosis are a critical subset of diagnostic errors that can lead to adverse patient outcomes. These errors frequently arise from failures in the diagnostic process, particularly in ensuring that recommended follow-ups are scheduled and completed. In large health systems, such as Parkland Health in Dallas, Texas, which conducts over 500,000 radiologist studies annually, the challenge of reliably identifying and managing follow-up recommendations is amplified by the reliance on structured note templates (macros) within electronic health records. Improper use or modification of these macros can result in missed notifications and suboptimal care. The authors developed and implemented a custom-built artificial intelligence (AI) agent that uses a pretrained large language model designed to act as an additional safety net for the identification and management of recommended follow-ups from radiologist notes. The AI agent reviews clinical impressions, extracts and standardizes key details for follow-up, and integrates these findings into the digital health workflow for patient outreach. Model performance was evaluated on a sample of 10,000 randomly selected radiologist notes and further assessed during 3 months of silent production mode, encompassing over 120,000 unique imaging studies. The AI agent achieved a balanced accuracy exceeding 97% for identifying radiologist notes requiring follow-up, correctly flagging 6.18 times more cases than the existing macro-based system (513 vs. 83 based on a sample of 10,000 studies). It also demonstrated over 94% accuracy in characterizing the timing of follow-up, the recommended procedure, and the underlying abnormality prompting the follow-up. This approach enabled the digital health team to more reliably identify patients in need of follow-up and improved the integration of actionable findings into patient outreach workflows. Implementation of an AI agent as an additional safety net significantly improved the identification of missed diagnostic opportunities in radiologist notes and accurately extracted key details that aid in patient outreach and scheduling. By enhancing the reliability of follow-up identification and standardizing key details, this approach increases the likelihood that patients receive appropriate care with the intention of optimizing health care outcomes in high-volume clinical settings.