Effective monitoring of online AI decision-making algorithms in just-in-time adaptive interventions

Monitoring just-in-time adaptive interventions (JITAIs) is important both during trialing and when the intervention is deployed in a broader healthcare program. While there is increasing interest in using artificial intelligence (AI) algorithms in JITAIs, these algorithms introduce additional complexity that requires additional monitoring. In this paper, we provide guidelines for monitoring online AI decision-making algorithms. Our guidelines include: (1) identifying potential issues, categorizing them by severity (red, yellow, and green), and (2) developing fallback methods (pre-specified procedures that are executed when an issue occurs). To make ideas concrete, we discuss algorithm monitoring systems in two case studies. In both, the monitoring systems detected real-time issues, and fallback methods both safeguarded participants and ensured quality data for post-deployment data analysis to further refine the JITAI. These guidelines and findings give teams the confidence to include online AI decision-making algorithms in JITAIs.