Dual-Branch Multimodal Fusion Network for Skin Lesions Diagnosis using Clinical and Ultrasound Image

Skin lesions describe the abnormal skin tissue that may be an indicator of cancer. Early diagnosis of benign and malignant diseases is crucial. The use of deep learning-based methods can reduce the influence of personnel subjectivity and improve the accuracy of diagnosis. This paper studies the benign and malignant skin disease diagnosis using a multi-modal fusion network. Specifically, we design a two-branch learning network, including global branches and local branches, to learn different information respectively. In addition, we design an attention-based feature fusion strategy to focus more on learning the features of the lesion area, which enhance more discriminative feature information of each modality. Note that this method focuses on learning feature information from clinical images (CN) and grayscale ultrasound images (US). Our method achieves the best performance compared to other methods by conducting experiments on self-collected dataset.