Colposcopic multimodal fusion for the classification of cervical lesions

Objective: Cervical cancer is one of the two biggest killers of women and early detection of cervical precancerous lesions can effectively improve the survival rate of patients. Manual diagnosis by combining colposcopic images and clinical examination results is the main clinical diagnosis method at present. Developing an intelligent diagnosis algorithm based on artificial intelligence is an inevitable trend to solve the objectification of diagnosis and improve the quality and efficiency of diagnosis.Approach: A colposcopic multimodal fusion convolutional neural network (CMF-CNN) was proposed for the classification of cervical lesions. Mask region convolutional neural network was used to detect the cervical region while the encoding network EfficientNet-B3 was introduced to extract the multimodal image features from the acetic image and iodine image. Finally, Squeeze-and-Excitation, Atrous Spatial Pyramid Pooling, and convolution block were also adopted to encode and fuse the patient's clinical text information.Main results: The experimental results showed that in 7106 cases of colposcopy, the accuracy, macro F1-score, macro-areas under the curve of the proposed model were 92.70%, 92.74%, 98.56%, respectively. They are superior to the mainstream unimodal image classification models.Significance: CMF-CNN proposed in this paper combines multimodal information, which has high performance in the classification of cervical lesions in colposcopy, so it can provide comprehensive diagnostic aid.