Region-Guided Vision-Language Modeling for CT Report Generation and Question Answering

We present a method for interpreting computed tomography scans by generating radiology reports and answering visual questions using a region-guided, slice-based approach. While 3D imaging provides rich spatial detail, most existing vision-language models are trained on 2D data and are not designed to process volumetric scans directly. In addition, pretrained models specifically tailored for 3D medical imaging remain scarce. To address this challenge, we propose a pipeline that leverages a 2D vision-language model for analyzing computed tomography scans. For report generation, the region is predefined by the task; for visual question answering, a language model agent determines the relevant region based on the question. In both cases, automatic segmentation is used to define the target region and extract a representative slice. These slices are then processed by a shared vision-language model to generate either region-level radiology reports or answers to clinically relevant questions. We evaluate our method on the publicly available AMOS-MM dataset, which includes region-annotated CT scans with corresponding reports and more than 16,000 visual question-answer pairs. Experimental results show good performance on both tasks. Our results also show that finetuning is required to achieve meaningful accuracy, as zero-shot models do not generalize well to the specific demands of radiology reporting and question answering. Our findings demonstrate that anatomical guidance combined with multi-task supervision enables effective adaptation of 2D models to the domain of 3D medical imaging, providing a promising foundation for future research and real-world clinical deployment.