Development and validation of a multimodal deep learning model for early esophageal squamous neoplasia detection and invasion depth prediction

Background: Early detection of esophageal squamous cell carcinoma (ESCC) is critical for optimizing patient outcomes. Magnifying endoscopy and endoscopic ultrasonography (EUS) serve as established diagnostic modalities. The multimodal ultrasound and magnifying endoscopic algorithm for early ESCC diagnostics (MUMA-EDx) integrates deep learning-based magnifying endoscopy and EUS imaging to improve early-stage ESCC identification and invasion depth assessment. Methods: Model development and internal validation used a retrospective dataset; external validation used a prospective cohort. MUMA-EDx developed two TResNet_m-based classifiers (magnifying endoscopy/EUS) followed by feature-level fusion. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. Results: MUMA-EDx was developed and validated using a retrospective dataset comprising 358 patients (18 420 images) and subsequently tested prospectively on an independent cohort of 122 patients (8711 images). The feature-level multimodal approach significantly outperformed single-modality models. For tumor discrimination, the model achieved an AUC of 0.94 (95%CI 0.92-0.96) in retrospective validation and a perfect patient-level AUC of 1.00 (95%CI 1.00-1.00) in prospective testing. For the more complex task of multiclass invasion depth classification, it achieved a retrospective AUC of 0.95 (95%CI 0.88-0.99), which remained strong at 0.80 (95%CI 0.67-0.87) in the prospective cohort. In a comparative study on invasion depth classification, MUMA-EDx's performance exceeded that of novice endoscopists and was comparable to expert-level diagnostics. Conclusion: MUMA-EDx demonstrably delivers exceptional early ESCC detection and robust invasion depth classification, achieving performance comparable to expert endoscopists and is poised to significantly enhance diagnostic precision and patient outcomes.