Abstract A049: Development of a tri-modal contrast learning model integrating pathology-text and CT-text for clinical oncology tasks

Abstract This study aims to develop a path-image-text tri-modal representation learning framework (Tri-MCR) without paired data by integrating pathology-text and CT-text models to improve the performance of clinical cancer tasks. Aiming at the challenge of scarce multi-modal data pairing in the cancer field, we project the pre-trained pathology-text and CT-text models into a shared semantic space by using text (i.e., Electronic Health Record) as an intermediate modality and optimize the cross-modal alignment by using a semantic enhancement strategy. Tri-MCR projects pre-trained pathology-text and CT-text models into a shared semantic space, leveraging text as an intermediary modality. Cross-modal alignment is optimized via a semantic enhancement strategy involving two key components: (1) Dynamic Semantic Enhancement: Gaussian noise injection and cross-modal attention mechanisms are employed to dynamically aggregate text-guided pathology and radiology features, thereby enhancing the semantic integrity of the embeddings. (2) Dual-Alignment Strategy: Cross-modal contrastive loss enforces semantic consistency between pathology-text and CT-text representations within the shared space, while intra-modal contrastive loss mitigates representation shifts between pathology and CT modalities. Furthermore, an Interpretability Mapping technique visualizes pathology-CT-text semantic associations through a cross-modal similarity matrix, offering biological insights for clinical decision-making. In the validation of the colorectal cancer single-center cohort (N=1015) of the ChangKang (Healthy Bowel) project, Tri-MCR significantly outperforms the baseline model in tumor biomarker prediction and mortality risk prediction tasks. This method provides a new idea for efficient representation learning of tumor multimodal data, provides an interpretable multimodal analysis tool for tumor precision diagnosis and treatment, and reduces the dependence on large-scale paired data, which has important clinical application value. In the future, further model multi-task performance evaluation and reliability verification in multi-center and multi-cancer cohorts are considered. Citation Format: Zhicheng Du, Hui-Yan Luo, Lijin Lian, Vijay Kumar. Pandey, Jiansong Ji, Peiwu Qin. Development of a tri-modal contrast learning model integrating pathology-text and CT-text for clinical oncology tasks [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Artificial Intelligence and Machine Learning; 2025 Jul 10-12; Montreal, QC, Canada. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(13_Suppl):Abstract nr A049.