How Academic Medical Centers Govern AI Prediction Tools in the Context of Uncertainty and Evolving Regulation

Prediction tools driven by artificial intelligence (AI) and machine learning are becoming increasingly integrated into health care delivery in the United States. However, organizational approaches to the governance of AI tools are highly varied. There is growing recognition of the need for evidence on best governance practices and multilayered oversight that could provide appropriate guardrails at the organizational and federal levels to address the unique dimensions of AI prediction tools. We sought to qualitatively characterize salient dimensions of AI-enabled predictive model governance at U.S. academic medical centers (AMCs). We analyzed how AMCs in the United States currently govern predictive models and consider the implications. A total of 17 individuals from 13 AMCs across the country participated in interviews. Half-hour to 1-hour interviews were conducted via Zoom from October 2022 to January 2023. The interview guide focused on the capacity, governance, regulation, and evaluation of AI-driven predictive models. Analysis of interview data was inductive. The research team wrote memos throughout the process of interviewing and analysis. We identified three governance phenotypes: well-defined, emerging, and interpersonal. In the well-defined governance phenotype, health systems have explicit, comprehensive procedures for the review and evaluation of AI and predictive models. In the emerging governance phenotype, systems are in the process of adjusting or adapting previously established approaches for clinical decision support or electronic health records (EHR) to govern AI. In health systems using interpersonal or individual-driven governance approaches, an individual is tasked with making decisions about model implementation without consistent evaluation requirements. We found that the influence of EHR vendors is an important consideration for those tasked with governance at AMCs, given concerns about regulatory gaps and the need for model evaluation. Even well-resourced AMCs are struggling to effectively identify, manage, and mitigate the myriad potential problems and pitfalls related to the implementation of predictive AI tools. The range of governance structures that we identified indicates a need for additional guidance, both regulatory and otherwise, for health systems as prediction and AI proliferate. Rather than concentrating responsibility for governance within organizations, multiple levels of governance that include the industry and regulators would better promote quality care and patient safety. This sort of structure would also provide desired guidance and support to the individuals tasked with governing these tools.