DeepBreastNet: Multi-Modal Deep Learning for Comprehensive Breast Cancer Diagnosis Across Heterogeneous Imaging Modalities

Breast cancer is one of the most common and deadly cancers worldwide, making early and accurate diagnosis essential for effective treatment. While deep learning methods have shown promise in breast cancer detection, most existing approaches focus on individual imaging modalities, such as MRI, ultrasound, or mammography, and rarely incorporate clinical data. This modality-specific focus limits adaptability and reduces diagnostic robustness in heterogeneous clinical settings. To address this gap, this study proposes DeepBreastNet, a multi-modal deep learning framework designed for comprehensive breast cancer diagnosis that is adaptable to all imaging modalities and capable of integrating relevant clinical information. The framework utilizes Attention U-Net for precise lesion segmentation, transformer-based encoders for extracting features from segmented lesions, enhanced platyrhynchos optimization (EPO) algorithm to select the most informative features, and a quantum dilated convolutional neural network (QD-CNN) for final lesion detection and diagnosis. Evaluation was conducted on over 229,000 mammography exams (1 M+ images), multiple ultrasound datasets (TDSC-ABUS, BUSI, GDPH, SYSUCC), and 290 UF-DCE-MRI scans with 1,081 associated radiology reports. The proposed model achieved 99.618% accuracy on mammography, 99.799% on ultrasound, and 99.718% on UF-DCE-MRI, with improvements of 32.4% in benign AUC and 28.8% in malignant AUC over existing state-of-the-art models. These findings validate DeepBreastNet capacity to generalize across several imaging modalities by demonstrating enhanced sensitivity, specificity, and diagnostic reliability. The framework provides a reliable, cohesive AI solution for diagnosing and detecting breast cancer.