A systematic comparison of non-overlap metrics and visual analysis in single-case experimental design

Visual analysis of graphed data is a prominent strategy employed by single-case researchers to document data stability and to ascertain whether a causal relation is demonstrated between the intervention and the dependent variable. Contemporary best practice also includes the use of non-parametric non-overlap metrics or parametric effect sizes to systematically quantify the magnitude of intervention effects. In the present study, 132 studies with 924 AB phase contrasts were analyzed using visual analysis and three non-overlap metrics. Sensitivity and specificity of Tau-U, improvement rate difference, and baseline-corrected Tau were tested using receiver operating characteristic analysis to detect the magnitude of intervention effects. Agreement with visual analysis was examined using kappa coefficient, and Pearson’s correlation coefficient. Each of the three non-overlap metrics performed moderately well in distinguishing effective from non-effective interventions and had a moderate to substantial level of agreement with visual analysis. Further research on the impact of confounding factors as well as the use of sensitive effect size metrics and visual aids to improve the reliability and accuracy of visual analysis is recommended.