Rethinking the Role of Teams and Training in Geopolitical Forecasting: The Effect of Uncontrolled Method Variance on Statistical Conclusions

Using data from a geopolitical forecasting tournament, Mellers et al. (2014) [Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25, 1106–1115] concluded that forecasting ability was improved by allowing participants to work in teams and providing them with probability training. Here, we reevaluated Mellers et al.’s conclusions using an item response theory framework that models latent ability from forecasting choices. We found that the relationship between latent ability estimates and forecast accuracy differed from the interpretation of the original findings once key extraneous variables were statistically controlled. The best fit models across the first 2 years of the tournament included one or more extraneous variables that substantially eliminated, reduced, and, in some cases, even reversed the effects of the experimental manipulations of teaming and training on latent forecasting ability. We also show that latent traits associated with strategic responding can discriminate between superforecasters and non-superforecasters, making it difficult to identify the latent factors that underlie the superforecasters’ superior performance.