Sequential Search Transformer: A Deep Structural Econometric Model

Modeling and leveraging consumers’ dynamic search behaviors presents significant business opportunities. Although deep learning methods excel at processing vast consumer data for predictive tasks, their opaque nature limits interpretability and fails to explicitly model consumer decision making. In contrast, economic theory suggests that consumers follow a sequential search strategy, evaluating alternatives until they find the best match for their preferences. To bridge this gap, we propose the sequential search transformer (SST), a deep structural econometric model that integrates deep learning with sequential search theory to model search and purchase decisions. SST unifies these two approaches into an end-to-end trainable model, improving both predictive accuracy and policy evaluation capabilities. Unlike conventional deep learning models, SST explicitly models consumer decision making, and unlike existing sequential search models, it enables consumer behavior modeling across sessions and sequentially resolves utility uncertainty for searched items. We provide a theoretical analysis of the identification strategy for the SST model and show that all parameters can be identified under the proposed framework. Then, we apply SST to a data set with detailed clickstream data collected from a U.S. e-commerce website. Empirical evaluations show that SST outperforms state-of-the-art deep learning and structural models in predicting consumer searches and purchases. Moreover, policy experiments demonstrate SST’s effectiveness in optimizing product recommendations and new product promotion strategies, ultimately enhancing consumers experience and driving revenue growth. This paper was accepted by Hemant Bhargava, information systems. Supplemental Material: The online appendices and data files are available at https://doi.org/10.1287/mnsc.2024.04540 .