Reliability, reproducibility, and potential pitfalls of the O-RADS scoring with non-dynamic MRI

Background The O-RADS scoring has been proposed to standardize the reporting of adnexal lesions using magnetic resonance imaging (MRI). Purpose To assess intra- and inter-observer agreement of the O-RADS scoring using non-dynamic MRI and its agreement with pathologic diagnosis, and to provide the pitfalls in the scoring based on discordant ratings. Material and Methods Adnexal lesions that were diagnosed using non-dynamic MRI at two centers were scored using O-RADS. Intra- and inter-observer agreements were assessed using kappa statistics. Cross-tabulations were made for intra- and inter-observer ratings and for O-RADS scores and pathological findings. Results Intra- and inter-observer agreements were assessed for 404 lesions in 339 patients who were admitted to center 1. Intra-observer agreement was almost perfect (97.8%, kappa = 0.963) and inter-observer agreement was substantial (83.2%, kappa = 0.730). The combined data from center 1 and center 2 included 496 patients; of them, 295 (59.5%) were operated. There was no borderline or malignant pathology for the lesions with O-RADS 1 or 2. Of those with an O-RADS score of 3, 3 (4.1%) lesions were borderline and none were malignant. The O-RADS scoring in discriminating borderline/malignant lesions from benign lesions was outstanding (area under the ROC curve 0.950, 95% CI = 0.923–0.971). Sensitivity, specificity, positive, and negative predictive values of O-RADS 4/5 lesions for borderline/malignant lesions were 96.2%, 87.1%, 72.8%, and 98.4%, respectively. Conclusion The O-RADS scoring using non-dynamic MRI is a reproducible method and has good discrimination for borderline/malignant lesions. Potential factors that may lead to discordant ratings are provided here.