Development of an Automated Composite Ureteroscopic Efficiency Score Through Simulated Ureteroscopic Skills Assessment

Introduction: Flexible ureteroscopy (fURS) is the most common procedure for treatment of urolithiasis. We previously utilized kinematic evaluations of simulated fURS to demonstrate that certain body movements are associated with efficient ureteroscopic manipulation for complex tasks. In this study, we incorporated computer vision to create an efficiency score using the ureteroscope travel distance (DIST), task time (TIME), spectral arc length (SPARC), and percentage of purposeful wall collisions (COLL). The goal is a simulation-based system that can abstract these automated performance metrics (APMs) to differentiate between novice and expert ureteroscope handling. Methods: A ureteroscopic simulation box was used. Body kinematics, task time, and ureteroscopic movements were analyzed using a motion capture system and video camera. Optical flow computer vision was used to track the ureteroscope. DIST, TIME, and SPARC were automatically calculated. Wall collisions were automatically captured and independently judged by two authors; an algorithm was developed to automatically determine the COLL variable. A mixed-effects model was used to aggregate these variables and distinguish between surgeons' first and final task attempts. Normalized values of these metrics were added to create a composite ureteroscopic efficiency score (CUES). Results: Twelve urologists completed the simulated tasks. The COLL assessment algorithm determined beneficial wall collisions with an accuracy of 77%. Normalized values of TIME, DIST, SPARC, and COLL were combined to create a composite ureteroscopic efficiency score (CUES). Compared with the first attempt, both the second and third attempts showed statistically significant improvements in CUES. The ROC-AUC score reached 0.86, suggesting excellent discrimination between attempts. There was also a statistically significant difference in CUES when comparing resident and attending performance. Conclusions: APMs can be abstracted using computer vision and artificial intelligence; an aggregate composite score (CUES) may be a promising method for evaluation of ureteroscopic efficiency.