Multimodal deep learning improving the accuracy of pathological diagnoses for membranous nephropathy

OBJECTIVES: Renal biopsy is the gold standard for the diagnosis of glomerular diseases including membranous nephropathy (MN), however, it faces challenges in accuracy, objectivity, and reproducibility of tissue evaluation. This study aims to develop a multimodal pathological diagnosis system to assist pathologists in diagnosing MN in morphology. METHODS: Using PASM-stained, immunofluorescence, and electron microscopy images from MN patients, we built three deep-learning models to detect lesions. The outputs of these models were combined to provide a comprehensive pathological diagnosis. Our system was compared with pathologists, validated on external test sets, and detected in 138 patients with various kidney diseases. RESULTS: Considering PASM-stained images, our model had a classification accuracy of 91.74%, a recall of 81.97%, and an F1 score of 86.58% for spike identification. For immunofluorescence images, our model had an accuracy rate of 98.97%, a recall rate of 99.65%, and an F1 score of 99.31% for MN classification. Regarding the segmentation of electron-dense deposits, the segmentation model had a Dice coefficient of 85.66% and an IoU of 75.93%. Our model presented superior performance to that of pathologists in fluorescence image classification and segmentation of deposits, achieved high accuracy in spike identification and fluorescence image classification in external test sets, and could be targeted to diagnose MN in a wide range of glomerular diseases. CONCLUSIONS: This multimodal pathological diagnosis system can not only assist pathologists in diagnosing MN rapidly and accurately but also lays the foundation to develop diagnostic models for other glomerular diseases.