The impact of different reference standards when estimating the accuracy of tests for diagnosing asthma

Background: Improving the accuracy with which asthma is diagnosed requires methodologically robust evaluation of diagnostic tests. This is complicated in asthma due to the lack of a true reference 'gold standard'. Aims: We investigated how the diagnostic accuracy of a test for asthma varies when compared against two reference standards: bronchial provocation assessed by 1) spirometry 2) whole body plethysmography (WBP). Methods: We conducted a secondary analysis of a diagnostic study. Individuals with symptoms suggestive of asthma were enrolled and completed a symptom questionnaire, lung function tests, Fractional exhaled Nitric Oxide (FeNO) and bronchial provocation. Index tests were 12 symptoms and FeNO. Diagnostic accuracy of every index test was calculated against both reference standards. Results: 393 participants (158 men) completed the study. Using these alternative reference standards made no difference to the estimates of sensitivity for each of the 12 symptoms. Using WBP as the reference standard, 11 of 12 symptoms had a higher specificity (not statistically significant) than spirometry. For FeNO, area under the receiver operating characteristic curve was non-significantly greater when WBP (0.66 95%CI 0.60 to 0.71) was used as the reference standard instead of spirometry (0.62 95%CI 0.55 to 0.69). Conclusions: Overall, no statistical difference in diagnostic accuracy of clinical symptoms or FeNO was found when spirometry or WBP reference standards were used. Measures of diagnostic accuracy were non-significantly greater with WBP as reference standard. Accurate evaluation of new tests and diagnostic algorithms for asthma requires careful consideration of the reference standard used.