Multi-modal data fusion for enhanced pancreatic cancer detection

Pancreatic cancer is among the deadliest cancers globally, primarily due to diagnosis at advanced stages. While deep learning has significantly advanced computer-aided diagnosis (CADx), most models operate in a single-modality framework. The integration of complementary information from multiple data sources in the medical domain could enhance CADx performance. To explore this potential, this work investigates the effectiveness of different data-fusion strategies across different image encoders. More specifically, the paper evaluates three fusion methods: (1) data-level fusion, (2) decision-level fusion and (3) feature-level fusion. To construct a baseline, these methods are evaluated on a novel, multi-modal animal dataset, comprising 2D natural images accompanied by self-annotated attributes. The study is then extended to the medical domain using an internal pancreatic cancer dataset composed by 3D abdominal CT scans and corresponding clinical features. For both datasets, the integration of image and attribute features shows significant improvements in classification performance. Specifically, for the pancreas dataset, data-level and decision-level fusion consistently outperform one modality experiments. Our best performing model achieves an area under receiver operating characteristics curve (AUC) of 0.94±0.01, significantly surpassing both its image only (0.87±0.01) and attributes only (0.87±0.02) baselines. This study highlights the effectiveness of multi-modal CADx in the medical domain. Our comparison to a larger-scale natural image dataset underscores the potential for even greater improvements, compared to image-only and attribute-only approaches, through more advanced fusion methods as more data becomes available. To encourage further research, the code and the newly introduced animal dataset will be made publicly available in a repository accessible at: https: //github.com/ConstancaSilva07/Multi-modalData-Fusion-for-Pancreatic-Cancer-Detection.