Multimodal Machine Learning Using Visual Fields and Peripapillary Circular OCT Scans in Detection of Glaucomatous Optic Neuropathy

PurposeTo develop and validate a multimodal artificial intelligence algorithm, FusionNet, using the pattern deviation probability plots from visual field (VF) reports and circular peripapillary OCT scans to detect glaucomatous optic neuropathy (GON).DesignCross-sectional study.SubjectsTwo thousand four hundred sixty-three pairs of VF and OCT images from 1083 patients.MethodsFusionNet based on bimodal input of VF and OCT paired data was developed to detect GON. Visual field data were collected using the Humphrey Field Analyzer (HFA). OCT images were collected from 3 types of devices (DRI-OCT, Cirrus OCT, and Spectralis). Two thousand four hundred sixty-three pairs of VF and OCT images were divided into 4 datasets: 1567 for training (HFA and DRI-OCT), 441 for primary validation (HFA and DRI-OCT), 255 for the internal test (HFA and Cirrus OCT), and 200 for the external test set (HFA and Spectralis). GON was defined as retinal nerve fiber layer thinning with corresponding VF defects.Main Outcome MeasuresDiagnostic performance of FusionNet compared with that of VFNet (with VF data as input) and OCTNet (with OCT data as input).ResultsFusionNet achieved an area under the receiver operating characteristic curve (AUC) of 0.950 (0.931–0.968) and outperformed VFNet (AUC, 0.868 [95% confidence interval (CI), 0.834–0.902]), OCTNet (AUC, 0.809 [95% CI, 0.768–0.850]), and 2 glaucoma specialists (glaucoma specialist 1: AUC, 0.882 [95% CI, 0.847–0.917]; glaucoma specialist 2: AUC, 0.883 [95% CI, 0.849–0.918]) in the primary validation set. In the internal and external test sets, the performances of FusionNet were also superior to VFNet and OCTNet (FusionNet vs VFNet vs OCTNet: internal test set 0.917 vs 0.854 vs 0.811; external test set 0.873 vs 0.772 vs 0.785). No significant difference was found between the 2 glaucoma specialists and FusionNet in the internal and external test sets, except for glaucoma specialist 2 (AUC, 0.858 [95% CI, 0.805–0.912]) in the internal test set.ConclusionsFusionNet, developed using paired VF and OCT data, demonstrated superior performance to both VFNet and OCTNet in detecting GON, suggesting that multimodal machine learning models are valuable in detecting GON. To develop and validate a multimodal artificial intelligence algorithm, FusionNet, using the pattern deviation probability plots from visual field (VF) reports and circular peripapillary OCT scans to detect glaucomatous optic neuropathy (GON). Cross-sectional study. Two thousand four hundred sixty-three pairs of VF and OCT images from 1083 patients. FusionNet based on bimodal input of VF and OCT paired data was developed to detect GON. Visual field data were collected using the Humphrey Field Analyzer (HFA). OCT images were collected from 3 types of devices (DRI-OCT, Cirrus OCT, and Spectralis). Two thousand four hundred sixty-three pairs of VF and OCT images were divided into 4 datasets: 1567 for training (HFA and DRI-OCT), 441 for primary validation (HFA and DRI-OCT), 255 for the internal test (HFA and Cirrus OCT), and 200 for the external test set (HFA and Spectralis). GON was defined as retinal nerve fiber layer thinning with corresponding VF defects. Diagnostic performance of FusionNet compared with that of VFNet (with VF data as input) and OCTNet (with OCT data as input). FusionNet achieved an area under the receiver operating characteristic curve (AUC) of 0.950 (0.931–0.968) and outperformed VFNet (AUC, 0.868 [95% confidence interval (CI), 0.834–0.902]), OCTNet (AUC, 0.809 [95% CI, 0.768–0.850]), and 2 glaucoma specialists (glaucoma specialist 1: AUC, 0.882 [95% CI, 0.847–0.917]; glaucoma specialist 2: AUC, 0.883 [95% CI, 0.849–0.918]) in the primary validation set. In the internal and external test sets, the performances of FusionNet were also superior to VFNet and OCTNet (FusionNet vs VFNet vs OCTNet: internal test set 0.917 vs 0.854 vs 0.811; external test set 0.873 vs 0.772 vs 0.785). No significant difference was found between the 2 glaucoma specialists and FusionNet in the internal and external test sets, except for glaucoma specialist 2 (AUC, 0.858 [95% CI, 0.805–0.912]) in the internal test set. FusionNet, developed using paired VF and OCT data, demonstrated superior performance to both VFNet and OCTNet in detecting GON, suggesting that multimodal machine learning models are valuable in detecting GON.