A generative vision-language model for holistic pathological assessment using preoperative imaging in hepatocellular carcinoma

BACKGROUND: Pathological evaluation of hepatocellular carcinoma (HCC) traditionally relies on surgical resection, posing risks of infection and complications while failing to provide comprehensive pathological insights preoperatively. This study aims to develop HepaPathGPT, which utilises preoperative imaging to deliver detailed pathological interpretations, enabling non-invasive, real-time pathological assessments for patients with HCC. METHODS: A retrospective study of 1091 patients with HCC from 10 independent cohorts was used. SegFormer-b5 segmented tumour regions, and vision-language alignment mapped imaging features to pathology descriptions. We fine-tuned four pretrained frameworks using Low-Rank Adaptation (LoRA) to efficiently translate imaging features into structured histological reports, enabling real-time evaluation via an interactive interface. FINDINGS: HepaPathGPT showed robust tumour segmentation (mean Intersection over Union: 0.883 ± 0.007, Dice: 0.934 ± 0.006) and an average accuracy of 0.697 ± 0.024 for six pathological markers in external validation (n = 109). For text generation, BLEU-4 and ROUGE-1 scores were 62.7 ± 1.7 and 84.2 ± 1.1. Five pathologists rated 92.5% and 87.4% of reports as acceptable for accuracy and completeness. INTERPRETATION: HepaPathGPT offers a approach for non-invasive pathological analysis in patients with HCC. This technology holds significant clinical value for decision-making in patients with HCC and promises scalability to other diseases in the future. FUNDING: National Natural Science Foundation of China (82090053, 82090052, 12326618, 82272703, 82473201); Tsinghua University Initiative Scientific Research Program of Precision Medicine (2022ZLA007); CAMS Innovation Fund for Medical Sciences (2019-I2M-5-056); Elite Youth Project of Natural Science Foundation of Fujian Province (2023J06056); Science-Health Joint Medical Scientific Research Project of Chongqing (2023MSXM092).