Accuracy and Reliability of Peripheral Artery Calcium Scoring Systems Using an Intravascular Ultrasound Reference Standard

Background Peripheral artery calcium scoring systems are commonly used in clinical trials to categorize calcium severity but there are little data on their accuracy and reliability. The purpose of this study was to investigate the accuracy and reliability of these systems. Methods Angiographic, computed tomography angiography, and intravascular ultrasound (IVUS) imaging were obtained from 47 consecutive cases sourced from a prospectively collected database of patients undergoing femoropopliteal artery endovascular intervention. Two independent blinded readers graded calcium severity using the Peripheral Arterial Calcium Scoring System, Peripheral Academic Research Consortium, and Fanelli calcium scoring systems. IVUS maximum arc of calcium and calcium length were compared between severity grades for each scoring system. The diagnostic accuracy of each scoring system for identifying severe calcium was calculated using the reference standard of an IVUS maximum calcium arc ≥ 180°. Agreement testing was performed between scoring systems and between and within observers for each system. Results IVUS identified calcium in 85% (42/47) of cases, compared to 68% (32/47) of cases with angiography. There were no differences in IVUS calcium parameters between grades of calcium for any of the scoring systems. Severe calcium was detected by IVUS in 30 cases, in 23 cases by Peripheral Arterial Calcium Scoring System (sensitivity: 73%, specificity: 33%, positive predictive value [PPV]: 83%, negative predictive value [NPV]: 22%), in 12 cases by Peripheral Academic Research Consortium (sensitivity: 42%, specificity: 83%, PPV: 92%, NPV: 25%), and in 10 cases by Fanelli (sensitivity: 39%, specificity: 100%, PPV: 100%, NPV: 27%). Agreement between scoring systems was weak to moderate (range: k = 0.55–0.74). Interobserver agreement was weak (k = 0.41–0.54) and intraobserver agreement was highly variable ranging from k = 0.41 to k = 0.92. Conclusions The poor diagnostic accuracy and weak-to-moderate reliability of calcium scoring systems raise doubts about the use of current calcium scoring systems for use in clinical trials. Peripheral artery calcium scoring systems are commonly used in clinical trials to categorize calcium severity but there are little data on their accuracy and reliability. The purpose of this study was to investigate the accuracy and reliability of these systems. Angiographic, computed tomography angiography, and intravascular ultrasound (IVUS) imaging were obtained from 47 consecutive cases sourced from a prospectively collected database of patients undergoing femoropopliteal artery endovascular intervention. Two independent blinded readers graded calcium severity using the Peripheral Arterial Calcium Scoring System, Peripheral Academic Research Consortium, and Fanelli calcium scoring systems. IVUS maximum arc of calcium and calcium length were compared between severity grades for each scoring system. The diagnostic accuracy of each scoring system for identifying severe calcium was calculated using the reference standard of an IVUS maximum calcium arc ≥ 180°. Agreement testing was performed between scoring systems and between and within observers for each system. IVUS identified calcium in 85% (42/47) of cases, compared to 68% (32/47) of cases with angiography. There were no differences in IVUS calcium parameters between grades of calcium for any of the scoring systems. Severe calcium was detected by IVUS in 30 cases, in 23 cases by Peripheral Arterial Calcium Scoring System (sensitivity: 73%, specificity: 33%, positive predictive value [PPV]: 83%, negative predictive value [NPV]: 22%), in 12 cases by Peripheral Academic Research Consortium (sensitivity: 42%, specificity: 83%, PPV: 92%, NPV: 25%), and in 10 cases by Fanelli (sensitivity: 39%, specificity: 100%, PPV: 100%, NPV: 27%). Agreement between scoring systems was weak to moderate (range: k = 0.55–0.74). Interobserver agreement was weak (k = 0.41–0.54) and intraobserver agreement was highly variable ranging from k = 0.41 to k = 0.92. The poor diagnostic accuracy and weak-to-moderate reliability of calcium scoring systems raise doubts about the use of current calcium scoring systems for use in clinical trials.