Enhancing Pathology Vision-Language Pre-Training: A Robust Baseline

Contrastive Language-Image Pretraining (CLIP) has demonstrated remarkable performance in aligning image and text representations across various domains, including medical imaging. However, CLIP relies heavily on large-scale paired datasets, which are scarce in the medical field, particularly in pathology imaging. Existing methods rely on web-scraped datasets that often contain noisy and irrelevant information, leading to poor generalization across different datasets, magnifications, and clinical settings due to incomplete or low-quality text. To address these limitations, we propose a hierarchical self-supervised vision-language framework that combines CLIP with Self-Supervised Learning (SSL), called PIVLM, to enhance feature representation for pathology images. Our framework introduces multi-level alignment at the instance, patch-token, region, and prototype levels, enabling the model to capture finegrained semantic correspondences between images and texts. By leveraging image augmentation for SSL, our method reduces reliance on noisy and incomplete textual information, ensuring robust feature learning. Experiments on five datasets for pathology images show that PIVLM achieves state-of-theart performance on zero-shot tasks compared to its CLIP-like counterparts. These results present a robust baseline for future research in medical image analysis, particularly in pathology, and opens new avenues for addressing the challenges of cross-modal alignment in medical imaging. The code is available at https://anonymous.4open.science/r/PIVLM-F9B1/.