Is Relevancy Everything? A Deep-Learning Approach to Understand the Effect of Image-Text Congruence

Firms increasingly use a combination of image and text description when displaying products and engaging consumers. Existing research has examined consumers’ response to text and image stimuli separately but has yet to systematically consider how the semantic relationship between image and text impacts consumer choice. In this research, we conduct a series of multimethod empirical studies to examine the congruence between image- and text-based product representation. First, we propose a deep-learning approach to measure image-text congruence by building a state-of-the-art two-branch neural network model based on wide residual networks and bidirectional encoder representations from transformers. Next, we apply our method to data from an online reading platform and discover a U-shaped effect of image-text congruence: Consumers’ preference toward a product is higher when the congruence between the image and text representation is either high or low than when the congruence is at the medium level. We then conduct experiments to establish the causal effect of this finding and explore the underlying mechanisms. We further explore the generalizability of the proposed deep-learning model and our substantive finding in two additional settings. Our research contributes to the literature on consumer information processing and generates managerial implications for practitioners on how to strategically pair images and text on digital platforms. This paper was accepted by Duncan Simester, marketing. Funding: J. Cao acknowledges financial support from Young Scientists Fund of National Natural Science Foundation of China [Grant 72402192], the General Research Fund [Grant 17501423] and Early Career Scheme [Grant 27502521] of the Research Grants Council of Hong Kong, and the Institute of Behavioural and Decision Science, the University of Hong Kong (HKU). Supplemental Material: The online appendices and data files are available at https://doi.org/10.1287/mnsc.2022.01896 .