Replicating Effects and Biases

AbstractOne common view is that a well-designed empirical study will reach conclusions that can be found again and again if the study is replicated, whereas a poorly designed study is unlikely to replicate. In opposition to this view, it is argued that a well-designed empirical study reaches conclusions that tend to replicate when correct and are less likely to replicate when incorrect, whereas in a poorly designed study the conclusions tend to replicate even when incorrect. In observational studies of treatment effects, the same hidden bias may occur repeatedly in a series of studies, so the studies reproduce the same distorted estimates of treatment effects. The purpose of this article is to point to strategies in research design that make it less likely that biased estimates will replicate, and to illustrate these strategies with examples. To replicate effects without replicating biases, vary the treatment assignment mechanism, so that the reasons subjects are spared treatment are different in the original and replicated studies. Also, vary the treatment envelope—that is, the ostensibly irrelevant features in which the treatment is packaged.KEY WORDS: Assignment mechanismHidden biasObservational studyTreatment envelope