Abstract 5008: A multi-dimensional vision-language deep learning model for radiology and pathology imaging

Abstract Multimodal vision-language models (VLM) have shown promising biomarker classification performance in individual data modalities such as radiology and pathology. By learning joint representations of visual and textual information, these methods have shown to provide better features from medical images. However, current methods are limited to the multimodal landscape of 2D images in radiology and pathology, leaving a substantial gap for the creation of models with 3D data. In this study, we introduce a novel, multimodal VLM that can process both radiology and pathology images of varying dimensionalities. Our approach processes 3D CT reconstructions and 2D digitized hematoxylin- and eosin-stained histology slides simultaneously, learning a joint representation which is applicable for biomarker classification in either modality. Our zero-shot approach was validated using two distinct datasets, eliminating the need for model fine-tuning. The first evaluation assessed whole-body CT lesion classification across multiple organs (lung, liver, kidney, and soft tissue) in 839 patients. The second evaluation focused on distinguishing between lung adenocarcinoma and squamous cell carcinoma in 974 patients. The model achieved an average Area Under the Receiver Operating Characteristic (AUROC) curve of 0.70 for lesion detection and 0.60 for lung cancer subtyping in a zero-shot setting. Our findings demonstrate the potential of multimodal vision-language models to perform medical image analysis without task-specific training, which offers a promising direction for developing more flexible and generalizable image processing and multimodal integration towards an artificial intelligence-driven molecular tumor board. Citation Format: Marta Ligero, Tim Lenz, Jakob Nikolas Kather. A multi-dimensional vision-language deep learning model for radiology and pathology imaging [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 5008.