Clinical Evaluation of an Artificial Intelligence-Based Decision Support System for the Diagnosis and American College of Radiology Thyroid Imaging Reporting and Data System Classification of Thyroid Nodules

Background: This study aimed to evaluate the clinical impact of an artificial intelligence (AI)-based decision support system (DSS), Koios DS, on the analysis of ultrasound imaging and suspicious characteristics for thyroid nodule risk stratification. Methods: A retrospective ultrasound study was conducted on all thyroid nodules with histological findings from June 2021 to December 2022 in a thyroid nodule clinic. The diagnostic performance of ultrasound imaging was evaluated by six readers on the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) before and after the use of the AI-based DSS and by AI itself. Results: A total of 172 patients (83.1% women) with a mean age of 52.3 ± 15.3 years were evaluated. The mean maximum nodular diameter was 2.9 ± 1.2 cm, with 11.0% being differentiated thyroid carcinomas. Among the nodules initially classified as ACR TI-RADS 3 and 4, AI reclassified 81.4% and 24.5% into lower risk categories, respectively. Receiver operating characteristic (ROC) curve analysis was performed to evaluate the diagnostic performance of the readers and the AI-based DSS versus histological diagnosis. There was an increase in the area under the ROC curve (AUROC) after the use of AI (0.776 vs. 0.817, p < 0.001). The AI-based DSS improved the mean sensitivity (Sens) (82.3% vs. 86.5%) and specificity (Spe) (38.3% vs. 54.8%), produced a high negative predictive value (94.5% vs. 96.4%), and increased the positive predictive value (PPV) (14.0% vs. 16.1%) and diagnostic precision (43.0% vs. 49.3%). Based on the ACR TI-RADS score, there was significant improvement in interobserver agreement after the use of AI (r = 0.741 for ultrasound imaging alone vs. 0.981 for ultrasound imaging and the AI-based DSS, p < 0.001). Conclusions: The use of an AI-based DSS was associated with overall improvement in the diagnostic efficacy of ultrasound imaging, based on the AUROC, as well as an increase in Sens, Spe, negative and PPVs, and diagnostic accuracy. There was also a reduction in interobserver variability and an increase in the degree of concordance with the use of AI. AI reclassified more than half of the nodules with intermediate ACR TI-RADS scores into lower risk categories.