Less can be better: decomposing clinical data modalities in large language model-based healthcare applications

OBJECTIVE: To systematically examine different clinical data modalities in large language models (LLMs) and multimodal large language models (MLLMs), and to quantify the contribution of data modalities in early inpatient risk prediction and decision support tasks. MATERIALS AND METHODS: We conducted a systematic analysis using MIMIC-IV, MIMIC-IV-Note, and MIMIC-CXR-JPG datasets to create a unified cohort of 22 254 hospital admissions containing structured electronic health records (EHRs), radiology reports (clinical notes), and chest X-ray images. We evaluated general-purpose and medical-adapted LLM/VLMs across uni-, bi-, and tri-modal configurations on 2 risk prediction tasks (in-hospital mortality and length of stay [LOS] prediction) and 2 clinical decision support (CDS) tasks (discharge diagnosis phenotyping and medication-use prediction). RESULTS: For risk prediction tasks, structured EHR data alone achieved the best or comparable performance (best mortality AUROC: 0.849; LOS AUROC: 0.868), with limited incremental benefit observed from adding radiology reports or medical images. For CDS tasks, multimodal integration yielded substantial improvements: the best tri-modal configuration achieved F1-scores of 0.589 (diagnosis) and 0.405 (medication), representing 21.4% and 18.4% improvement over the best unimodal approach. Radiology reports consistently outperformed raw single-view chest radiographs as a supplementary modality. MLLMs demonstrated better zero- and few-shot performance than unimodal LLMs. Multi-view imaging consistently improved performance over single view across all tasks. CONCLUSION: The benefits of multimodal data integration are task dependent. Healthcare LLMs should examine clinical data modalities according to specific tasks for efficient integration. These findings provide practical guidance for designing efficient clinical decision support systems.