Predicting human decisions with behavioral theories and machine learning

Predicting human decision-making under risk and uncertainty represents a quintessential challenge that spans economics, psychology, and related disciplines. Despite decades of research effort, no model can be said to accurately describe and predict human choice even for the most stylized tasks like choice between lotteries. Here, we introduce BEAST Gradient Boosting (BEAST-GB), a novel hybrid model that synergizes behavioral theories, specifically the model BEAST, with machine learning techniques. First, we show the effectiveness of BEAST-GB by describing CPC18, an open competition for prediction of human decision making under risk and uncertainty, in which BEAST-GB won. Second, we show that it achieves state-of-the-art performance on the largest publicly available dataset of human risky choice, outperforming purely data-driven neural networks, indicating the continued relevance of BEAST theoretical insights in the presence of large data. Third, we demonstrate BEAST-GB's superior predictive power in an ensemble of choice experiments in which the BEAST model alone falters, underscoring the indispensable role of machine learning in interpreting complex idiosyncratic behavioral data. Finally, we show BEAST-GB also displays robust domain generalization capabilities as it effectively predicts choice behavior in new experimental contexts that it was not trained on. These results confirm the potency of combining domain-specific theoretical frameworks with machine learning, underscoring a methodological advance with broad implications for modeling decisions in diverse environments.