Data Extraction and Curation from Radiology Reports for Pancreatic Cyst Surveillance Using Large Language Models

BACKGROUND: Manual curation of radiographic features in pancreatic cyst registries for data abstraction and longitudinal evaluation is time-consuming and limits widespread implementation. We examined the feasibility and accuracy of using large language models (LLMs) to extract clinical variables from radiology reports. STUDY DESIGN: A single-center retrospective study included patients under surveillance for pancreatic cysts. Nine radiographic elements used to monitor cyst progression were included: cyst size, main pancreatic duct (MPD) size (continuous variable), number of lesions, MPD dilation 5 mmol/L or more (categorical), branch duct dilation, presence of solid component, calcific lesion, pancreatic atrophy, and pancreatitis. LLMs (generative pretrained transformer [GPT]) on the OpenAI GPT-4 platform were used to extract elements of interest with a zero-shot learning approach using prompting to facilitate annotation without any training data. A manually annotated institutional cyst database was used as the ground truth for comparison. RESULTS: Overall, 3,198 longitudinal scans from 991 patients were included. GPT successfully extracted the selected radiographic elements with high accuracy. Among categorical variables, accuracy ranged from 97% for solid component to 99% for calcific lesions. In the continuous variables, accuracy varied from 92% for cyst size to 97% for MPD size. However, Cohen’s kappa was higher for cyst size (0.92) compared with that for MPD size (0.82). Lowest accuracy (81%) was noted in the multiclass variable for a number of cysts. CONCLUSIONS: LLM can accurately extract and curate data from radiology reports for pancreatic cyst surveillance and can be reliably used to assemble longitudinal databases. Future application of this work can potentiate the development of artificial intelligence–based surveillance models.