Randomized multi-reader evaluation of automated detection and segmentation of brain tumors in stereotactic radiosurgery with deep neural networks

Abstract Background Stereotactic radiosurgery (SRS), a validated treatment for brain tumors, requires accurate tumor contouring. This manual segmentation process is time-consuming and prone to substantial inter-practitioner variability. Artificial intelligence (AI) with deep neural networks have increasingly been proposed for use in lesion detection and segmentation but have seldom been validated in a clinical setting. Methods We conducted a randomized, cross-modal, multi-reader, multispecialty, multi-case study to evaluate the impact of AI assistance on brain tumor SRS. A state-of-the-art auto-contouring algorithm built on multi-modality imaging and ensemble neural networks was integrated into the clinical workflow. Nine medical professionals contoured the same case series in two reader modes (assisted or unassisted) with a memory washout period of 6 weeks between each section. The case series consisted of 10 algorithm-unseen cases, including five cases of brain metastases, three of meningiomas, and two of acoustic neuromas. Among the nine readers, three experienced experts determined the ground truths of tumor contours. Results With the AI assistance, the inter-reader agreement significantly increased (Dice similarity coefficient [DSC] from 0.86 to 0.90, P < 0.001). Algorithm-assisted physicians demonstrated a higher sensitivity for lesion detection than unassisted physicians (91.3% vs 82.6%, P = .030). AI assistance improved contouring accuracy, with an average increase in DSC of 0.028, especially for physicians with less SRS experience (average DSC from 0.847 to 0.865, P = .002). In addition, AI assistance improved efficiency with a median of 30.8% time-saving. Less-experienced clinicians gained prominent improvement on contouring accuracy but less benefit in reduction of working hours. By contrast, SRS specialists had a relatively minor advantage in DSC, but greater time-saving with the aid of AI. Conclusions Deep learning neural networks can be optimally utilized to improve accuracy and efficiency for the clinical workflow in brain tumor SRS.