MMCN: Multi-Modal Co-attention Network for Medical Visual Question Answering

Medical visual question Answering (MedVQA) attempts to answer medical questions posed by a correlative medical image. Although it has vast potential in the medical domain, this technology is still difficult to apply in real-life. It has not been widely adopted because accurate answer prediction requires a refined understanding of medical images and question text. Existing methods directly use the whole image and the whole question for multi-modal fusion to predict the answer. However, for a question, important information only exists in a small part of the whole image and a few critical words in the question, and extra information may interfere with the answer prediction. To this end, we introduce an effective multi-modal co-attention network (MMCN) for learning essential words in the question and essential regions in the image. Each word and region is scored by the attention weighting method, which is used to indicate the importance of each word and region in the process of model reasoning. Experimental comparisons show that our MMCN is superior to the most advanced methods of the public RAD-VQA dataset.