Multi-Modal Deep Learning for Pulmonary Embolism Classification: A Late Fusion Approach Combining CTPA Imaging and EMR Data

Early identification of Pulmonary Embolism (PE) is vital, as it is a perilous medical state associated with significant mortality. Traditional diagnostic approaches often rely on a single modality, such as imaging, which can result in delayed and suboptimal predictions. To overcome this limitation, recent progress in Deep Learning (DL) has facilitated the integration of Computed Tomography and Pulmonary Angiography (CTPA) images with Electronic Medical Records (EMR), providing complementary information that improves diagnostic precision. This study explores the late fusion strategy for multimodal PE classification. In the late fusion approach, independent DL models are trained separately on image and EMR data, and their outputs are combined at the decision level, preserving modality-specific learning. Multiple DL architectures, including LSTM, Bi-LSTM, Atten LSTM, GRU, Bi-GRU, and Atten GRU were evaluated using the Stanford University Medical Centre (SUMC) dataset. Among the proposed models, Atten LSTM consistently outperformed others, achieving a PE classification accuracy of 90.86%. These findings highlight the potential of the late fusion technique in enhancing PE prediction by leveraging the synergy between imaging and textual EMR data.