Use of Event-Related Potentials to Assess Visual–Auditory Multisensory Information in the Decision-Making Processes Related to Fast-Consuming Behavior

AbstractPeople are accustomed to using mobile phones to purchase products online. Previous studies identified brain activities associated with online buying decisions but did not assess mismatch negativity (MMN), P300 (P3a and P3b), and slow waves. We applied visual animation and sound effects to test visual–auditory multisensory concepts in product promotion strategies. Event-related potential (ERP) responses from 53 participants were analyzed to help design sound effects based on MMN, P300, and slow waves. We designed five short visual–auditory animation videos: visual animation sound effect (VASE), VA sound beep (VASB), visual standard sound beep (VSSB), visual animation–no sound (VANS), and visual standard–no sound (VSNS). These five apps were used to test how matched and unmatched multisensory user experience (UX) influences online shopping decisions. We also used Arrow's impossibility theorem to design the purchasing experiment. VASE evoked MMN (cluster p = 0.0184) compared with VSSB. VASE evoked the strongest P300 (P3a and P3b) responses (cluster p = 1 × 10−5), and VASB evoked more negative slow wave amplitudes (cluster p = 1.7 × 10−4) compared with VASE. Participants buy products unconsciously via most VASE and VANS mobile apps compared with via VASB and VSNS. Our findings suggest that sensory memory from multisensory UX can promote product consumption in mobile phone shopping interfaces. This study's testing of memory and attention from MMN, P300, and slow waves makes a novel contribution to the marketing discipline. The implications can motivate researchers to apply ERP to analyze the multisensory concept in future interdisciplinary research.Keywords: Multisensory designdecision-making processmismatch negativityproduct promotionP300slow waves AcknowledgmentsFirst, Zhepeng Rui would like to sincerely thank my Ph.D. supervisor, Professor Dr. Zhenyu Gu, who critically reviewed the manuscript and supported the study funding. The encouragement and guidance provided by my supervisor have greatly improved this study. Second, we would like to thank Dr. Eric Field and Dr. David Groppe for creating MUT and FMUT toolboxes and helping us with technical questions. Third, we would like to thank Jianrong Zhang for analyzing and helping with the sound design. Finally, we wish to thank all my friends and colleagues at the university and in the industry, who contributed to the research experiments.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementRui, Zhepeng; Gu, Zhenyu (2023), "Data for: Use of event-related potentials to assess visual–auditory multisensory information in the decision-making processes related to fast-consuming behavior", Mendeley Data, V6, doi: 10.17632/ht5dnfgdsk.6. https://data.mendeley.com/datasets/ht5dnfgdsk/6Stimuli availability statementTo better replicate our study, we have uploaded our stimuli along with the research data in the Mendeley data repository.Six videos on visual animation and sound effects in the pre-experimental stageExample stimuli video displays VASE, VASB, VSSB, VANS, and VSNS in stage 1 and stage 2.Example stimuli displays VASE video in stage 3 because it is the most promotional and representative in this study.Additional informationFundingThis study was supported by the National Natural Science Foundation of China [Grant no. 71802132].Notes on contributorsZhepeng RuiZhepeng Rui earned Bachelor's and Master's degrees in Design Computing, Interaction Design, and Electronic Art from the University of Sydney. Currently pursuing a PhD at Shanghai Jiao Tong University, he has 4 years of industry experience in visual-sound design and UI/UX. His research covers art, HCI, user-centric marketing, and neuroscience.Zhenyu GuZhenyu Gu, a design professor at Shanghai Jiao Tong University's School of Design, holds a PhD from Hong Kong Polytechnic University. He serves as the director of the industrial and interactive design committee. His research spans industrial and interactive design, design computing, and cognitive intelligent product design.