Expectations vs Reality of an Intraoperative Artificial Intelligence Intervention

Importance: Having significant gaps between the expectations and reality of artificial intelligence-based programs can be a major barrier to successful implementation. This is the first multisite implementation assessment of gaps between surgeon expectations and real-world effects of the Operating Room Black Box, a novel intervention that leverages artificial intelligence to improve surgical outcomes. Objective: To identify barriers and facilitators to implementing artificial intelligence-based interventions that improve intra- and postoperative care. Design, Setting, and Participants: This qualitative study was conducted at 3 large academic centers via semistructured interviews with surgeons and implementation leaders of the AI intervention to identify areas where expectations of the technology misaligned with their experiences. Thirty surgeons and 17 implementation leaders from 3 centers that implemented the AI intervention were interviewed. Data were collected and analyzed between 2021 and 2024. Exposure: Implementation of the AI intervention. Main Outcomes and Measures: The primary outcome was areas of misalignment between participant expectations of the AI intervention technology and actual program deliverables. Results: Of 30 surgeons and 17 implementation leaders interviewed, most surgeons (17 [57%]) were between the ages of 35 and 50 years, and implementation leaders were older, typically between 51 and 80 years old (6 [35%]). Eight surgeons (27%) and 4 implementation leaders (24%) were female. Most surgeons (17 [57%]) had neutral views of the technology, 11 (37%) expressed positive views, and 2 (7%) had negative views. Interviewees identified the following 4 major themes that highlighted misalignment between user expectations and the experience of using the technology: (1) the artificial intelligence model needed considerable additional training to be usable; (2) accessing data on surgical cases was difficult and time consuming; (3) the program showed limited ability to predict postoperative complications; and (4) the program generated few academic deliverables. Conclusions and Relevance: Per the results of this multisite qualitative study, successfully implementing interventions based on artificial intelligence may require deliberate efforts to minimize gaps between what surgeons expect from the interventions and what they can deliver. Our evaluation of this study's AI intervention offers lessons for addressing this critical barrier to implementation.