Editorial: Robust Sensitivities

In this issue of the Journal, Lash et al. (1), separate commentaries by Greenland (2) and Gustafson (3), and a rejoinder by Lash et al. (4) comprise a rare and important example of extended scientific discourse. This set of papers discusses current use of sensitivity analysis, particularly quantitative bias analysis, in the field of epidemiology. Lash et al. (1) detail 3 cases of published bias analyses in which, they are concerned, the results might mislead more than inform readers regarding the expected impact of measurement error (example 1 by Chien et al. (5)), selection bias (example 2 by Di Forti et al. (6)), and uncontrolled confounding (example 3 by Mørch et al. (7)). Gustafson (3) deepens our view by extending the bias analysis of Chien et al. (5), making it more rigorous, and rightly calling attention to the fact that our intuitions around bias analysis are not yet well honed. In contrast, Greenland (2) broadens our view to a continuum of possible bias analyses, anchored by conventional analysis at one end (i.e., assuming no uncontrolled bias), and some, admittedly elusive, definitive bias analysis at the other end. In their rejoinder, Lash et al. (4) further highlight the difficulties inherent in conducting rigorous bias analyses despite best intentions, related to both understanding of the methods and consideration of how bias parameters might be correlated. In addition to calling for more and more transparent examples of bias analyses (a call we echo here), they emphasize the utility of bias analyses to inform study design and data collection. Two central themes in this set of papers that we wish to highlight are the call for “principled” bias analysis and the question of “optimal” bias analysis. We all call for principled analyses of epidemiologic data (8). All the authors are champions of principled bias analysis. Such a principled or formal approach to bias analysis could improve our collective understanding of and expectations for bias analyses and help hone our intuition. In describing the 3 case studies they present as “suboptimal,” Lash et al. rely on an informal definition of optimality (1). Perhaps optimal bias analysis is unobtainable given human limitations, and we ought instead to seek some kind of “type 2” optimality (9), some practical albeit nebulous optimality. Given that no definitive optimal bias analysis is on offer, rather than concede in the despair of conventional analyses, Greenland calls for “satisfactory” bias analysis. Any satisfactory, principled bias analyses ought to include sufficient detail regarding methods used and the source of bias parameters for its utility to be evaluable, as well as meaningful engagement with the leading sources of bias specific to the study, rather than merely providing values for bias parameters sufficient to falsify results without evaluating their plausibility. If we are rational, then there can hardly be any strong justification to prefer conventional analysis over the results that would have been found with slight perturbations in unidentified bias parameters. It is akin to choosing one set of results from a large set and pretending that these results have some particularly strong value that other members of this set do not have, even as we admit, in our discussions, that we might have omitted control for a confounder or mismeasured key variables. Perhaps this overly firm preference for the observed data is a remnant of logical positivism. However, we cannot escape assumptions when making factual claims. Any utility in the observed data is dependent on many assumptions, many of which go unnamed. There can be no “model-free” model of our world. Classification of assumptions can be helpful, particularly for identifying key threats to inference, although any classification system will be an oversimplification. Below the set of “identification” assumptions on which epidemiologists concentrate (e.g., no selection bias) are a set of statistical “regularity” conditions (e.g., smoothness) (10). Below these are a deeper set of obscure conditions, such as everything is self-identical, the external world exists, nothing is both true and false, and the British regularity condition, “whatever else is required.” To conclude, we affirm the Journal’s support for the use of principled, practical, sufficiently “satisfactory” analyses to explore the sensitivity of results to assumptions. We suggest that authors consider not only detailing the set of assumptions made in their discussion of limitations but, to the extent possible, that they discuss their assumptions in rank order of importance and explore potential violations of those assumptions quantitatively. Perhaps the progress of a science, or at least the progress of epidemiology, can be seen in how topics previously discussed only loosely in the natural language of a Discussion section move into the more formal language of the Methods and Results sections, freeing Discussion for both broader and deeper issues, which, in turn, might—or might not—become topics of study by future generations. Author affiliations: Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States (Catherine Lesko); Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina, Chapel Hill, Chapel Hill, North Carolina, United States (Stephen R. Cole); and Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States (Enrique Schisterman). This work was funded by National Institutes of Health (grant K01 AA028193). Conflict of interest: none declared.