A real-time system using deep learning to detect and track ureteral orifices during urinary endoscopy

Abstract Background and objective To automatically identify and locate various types and states of the ureteral orifice (UO) in real endoscopy scenarios, we developed and verified a real-time computer-aided UO detection and tracking system using an improved real-time deep convolutional neural network and a robust tracking algorithm. Methods The single-shot multibox detector (SSD) was refined to perform the detection task. We trained both the SSD and Refined-SSD using 447 resectoscopy images with UO and tested them on 818 ureteroscopy images. We also evaluated the detection performance on endoscopy video frames, which comprised 892 resectoscopy frames and 1366 ureteroscopy frames. UOs could not be identified with certainty because sometimes they appeared on the screen in a closed state of peristaltic contraction. To mitigate this problem and mimic the inspection behavior of urologists, we integrated the SSD and Refined-SSD with five different tracking algorithms. Results When tested on 818 ureteroscopy images, our proposed UO detection network, Refined-SSD, achieved an accuracy of 0.902. In the video sequence analysis, our detection model yielded test sensitivities of 0.840 and 0.922 on resectoscopy and ureteroscopy video frames, respectively. In addition, by testing Refined-SSD on 1366 ureteroscopy video frames, the sensitivity achieved a value of 0.922, and a lowest false positive per image of 0.049 was obtained. For UO tracking performance, our proposed UO detection and tracking system (Refined-SSD integrated with CSRT) performed the best overall. At an overlap threshold of 0.5, the success rate of our proposed UO detection and tracking system was greater than 0.95 on 17 resectoscopy video clips and achieved nearly 0.95 on 40 ureteroscopy video clips. Conclusions We developed a deep learning system that could be used for detecting and tracking UOs in endoscopy scenarios in real time. This system can simultaneously maintain high accuracy. This approach has great potential to serve as an excellent learning and feedback system for trainees and new urologists in clinical settings.