An overview of deep learning methods for multimodal medical data mining

Deep learning methods have achieved significant results in various fields. Due to the success of these methods, many researchers have used deep learning algorithms in medical analyses. Using multimodal data to achieve more accurate results is a successful strategy because multimodal data provide complementary information. This paper first introduces the most popular modalities, fusion strategies, and deep learning architectures. We also explain learning strategies, including transfer learning, end-to-end learning, and multitask learning. Then, we give an overview of deep learning methods for multimodal medical data analysis. We have focused on articles published over the last four years. We end with a summary of the current state-of-the-art, common problems, and directions for future research.