Glio-LLaMA-Vision: A Vision-Language Model for Molecular Prediction, Radiology Report Generation, and VQA in Adult-type Diffuse Gliomas

Motivation: Leveraging a pre-trained large vision-language model may show robust performance for radiology of adult-type diffuse gliomas. Goal(s): To establish a robust vision-language model for molecular subtyping, radiology report generation, and visual question answering (VQA) in adult-type diffuse gliomas. Approach: MRI and paired radiology reports from 1,001 adult-type diffuse gliomas patients were included as the institutional training set. A vision-language model, Glio-LLaMA-Vision, was developed from LLaMA 3.1 pre-trained on 2.79 million biomedical text-image pairs and was optimized via fine-tuning from the training set. The performance was validated on external test sets. Results: Glio-LLaMA-Vision showed robust performance on molecular subtyping, radiology report generation, and VQA. Impact: Glio-LLaMA-Vision shows promising performance in molecular subtype prediction, radiology report generation, and VQA in adult-type diffuse gliomas. Notably, our current study provides a practical paradigm of adapting general domain LLMs to applications in a specific medical domain.