Development and validation of a novel multimodal deep learning model integrating histopathology, radiology, and clinical data to predict primary non-response to infliximab in patients with Crohn’s disease

BACKGROUND AND AIMS: Crohn's disease (CD) is a chronic inflammatory condition of the gastrointestinal tract. While infliximab (IFX) offers significant benefits, 10%-30% of patients remain non-responders initially. This study employs artificial intelligence with multimodal integration to improve treatment response prediction and advance precision medicine. METHODS: We conducted a retrospective analysis of clinical data from patients with CD. The endpoint event was defined as primary non-response within 14 weeks of treatment. The multimodal dataset included laboratory indices, computed tomography enterography (CTE), and endoscopic histopathology based on whole-slide biopsy images. A TabNet model, originally designed for tabular data and here applied to clinical and laboratory features, was developed using a multi-instance learning framework to incorporate this multimodal information for predicting primary non-response to IFX. Finally, the multimodal model was validated in an independent external test cohort. RESULTS: The study included 188 patients, with 93 in the internal training set, 38 in the internal validation set, and 57 in the test set from an independent external cohort. The model utilizing pathological features achieved an area under the receiver operating characteristic (AUC) of 0.789 in internal validation. When combining pathological and radiological features, the AUC was 0.844. The optimal multimodal model integrating histology, radiology, and clinical features achieved an AUC of 0.852 in the internal validation set and 0.858 in the external test set. CONCLUSIONS: The study developed a multimodal deep learning model accurately predicting IFX primary non-response, offering a tool to guide individualized therapy and improve CD outcomes.