Approaching Intelligent In-vehicle Infotainment Systems through Fusion Visual-Speech Multimodal Interaction: A State-of-the-Art Review

Advanced in-vehicle infotainment systems are part of the future's intelligent autonomous vehicles. Developing such systems requires advanced interaction modalities to be utilized. On the other hand, using sophisticated applications on an unimodal touch display for manual driving vehicles could endanger drivers' lives and result in a poor user experience. Offering in-vehicle fusion multimodal interaction could broaden the types of applications and enhance user experience while keeping safety and low distraction into account. Since in-vehicle interaction is bi-directional, both driver-vehicle and vehicle-driver sides are equally important to achieve and develop advanced infotainment systems. Searching in related literature, it has been found that there is good progress in driver-vehicle fusion multimodal interaction; in comparison, only a scarce amount of research on the vehicle-driver side is available. This paper presents the state-of-the-art in vehicle-driver fusion multimodal interaction for infotainment systems. This type of interaction is essentially a form of human-computer interaction. However, when we specify that the computer is a vehicle computer, certain specific factors become essential to consider when developing the user interface for this type of computer. Furthermore, a research agenda together with challenges and opportunities is proposed.