计算机科学
人工智能
自然语言处理
语言模型
计算机视觉
作者
Haojie Zhang,Min Zeng,Jinfeng Ding,Yixiong Liang,Ruiqing Zheng,Zhe Qu,Min Li,Shichao Kan
出处
期刊:
[Institute of Electrical and Electronics Engineers]
日期:2024-12-03
卷期号:: 2874-2879
被引量:2
标识
DOI:10.1109/bibm62325.2024.10822357
摘要
Large Vision-Language Models (LVLMs) have garnered substantial attention in the biomedical image analysis domain due to their robust vision understanding capabilities. However, current methods rely heavily on dataset- and modality-specific fine-tuning. This involves tuning separate models for each dataset and biomedical modality. In this paper, we introduce a method for aligning multimodal biomedical images and language using a single LVLM, dubbed UniMed-LVLM. Specifically, we devise a General Projection Module (GPM) by integrating multiple image projection branches and implementing dynamic routing between the vision encoder and language decoder within the LLaVA-Med framework. Subsequently, we progressively align multiple biomedical modalities using a Parameter-Efficient Fine-Tuning (PEFT) technique known as Low-Rank Adaptation (LoRA). The model is initially trained on the LLaVA-Med dataset and then fine-tuned on four biomedical image analysis datasets: PathVqa, Slake, VqaRad, and Fitzpatrick17k, enabling the simultaneous analysis of radiology, pathology, and dermatology images. A single model is fine-tuned on three modalities across these datasets and evaluated on all test sets. Experimental results show that UniMed-LVLM improves the average evaluation score by 1.88% across the four datasets, validating its effectiveness in handling multimodal biomedical images.
科研通智能强力驱动
Strongly Powered by AbleSci AI