Diagnosis assistant for liver cancer utilizing a large language model with three types of knowledge

Abstract Objective. Liver cancer has a high incidence rate, but experienced doctors are lacking in primary healthcare settings. The development of large models offers new possibilities for diagnosis. However, in liver cancer diagnosis, large models face certain limitations, such as insufficient understanding of specific medical images, inadequate consideration of liver vessel factors, and inaccuracies in reasoning logic. Therefore, this study proposes a diagnostic assistance tool specific to liver cancer to enhance the diagnostic capabilities of primary care doctors. Approach. A liver cancer diagnosis framework combining large and small models is proposed. A more accurate model for liver tumor segmentation and a more precise model for liver vessel segmentation are developed. The features extracted from the segmentation results of the small models are combined with the patient’s medical records and then provided to the large model. The large model employs chain of thought prompts to simulate expert diagnostic reasoning and uses Retrieval-Augmented Generation to provide reliable answers based on trusted medical knowledge and cases. Main results. In the small model part, the proposed liver tumor and liver vessel segmentation methods achieve improved performance. In the large model part, this approach receives higher evaluation scores from doctors when analyzing patient imaging and medical records. Significance. First, a diagnostic framework combining small models and large models is proposed to optimize the liver cancer diagnosis process. Second, two segmentation models are introduced to compensate for the large model’s shortcomings in extracting semantic information from images. Third, by simulating doctors’ reasoning and integrating trusted knowledge, the framework enhances the reliability and interpretability of the large model’s responses while reducing hallucination phenomena.