Large‐language‐model empowered 3D dose prediction for intensity‐modulated radiotherapy

Abstract Background Treatment planning is currently a patient specific, time‐consuming, and resource demanding task in radiotherapy. Dose‐volume histogram (DVH) prediction plays a critical role in automating this process. The geometric relationship between DVHs in radiotherapy plans and organs‐at‐risk (OAR) and planning target volume (PTV) has been well established. This study explores the potential of deep learning models for predicting DVHs using images and subsequent human intervention facilitated by a large‐language model (LLM) to enhance the planning quality. Method We propose a pipeline to convert unstructured images to a structured graph consisting of image‐patch nodes and dose nodes. A novel Dose Graph Neural Network (DoseGNN) model is developed for predicting DVHs from the structured graph. The proposed DoseGNN is enhanced with the LLM to encode massive knowledge from prescriptions and interactive instructions from clinicians. In this study, we introduced an online human‐AI collaboration (OHAC) system as a practical implementation of the concept proposed for the automation of intensity‐modulated radiotherapy (IMRT) planning. Results The proposed DoseGNN model was compared to widely employed DL models used in radiotherapy, including Swin Transformer, 3D U‐Net CNN, and vanilla MLP. For PTV, DoseGNN achieved the mean absolute error (MAE) of , , , and between true plans and predicted plans that were 64%, 53%, 64%, 61% of the best baseline model. For the worst case among OARs (left lung, right lung, chest wall, heart, spinal cord), DoseGNN achieved the mean absolute error of , , that were 85%, 91%, 80% of the best baseline model. Moreover, the LLM‐empowered DoseGNN model facilitates seamless adjustment to treatment plans through interaction with clinicians using natural language. Conclusion We developed DoseGNN, a novel deep learning model for predicting delivered radiation doses from medical images, enhanced by LLM to allow adjustment through seamless interaction with clinicians. The preliminary results confirm DoseGNN's superior accuracy in DVH prediction relative to typical DL methods, highlighting its potential to facilitate an online clinician‐AI collaboration system for streamlined treatment planning automation.