A Multimodal Human-Computer Interaction for Smart Learning System

The rise of digitalization and computing devices has transformed the educational landscape, making traditional teaching methods less productive. In this context, early and continuous user interaction is crucial for designing and developing effective learning applications. The field of Human-Computer Interaction (HCI) has seen significant technological growth, enabling educators to provide quality educational services through smart input and output channels. However, to prevent students from discontinuing their studies and help them grow their careers, a multimodal HCI approach is needed. This paper proposes a multimodal deep learning multi-layer Convolutional Neural Network (CNN) to improve the educational experience. Our designed system aims to create a promising solution for improving the educational experience and enabling educators to provide high-quality educational services to students. Our implementation results show promising real-time performances, including a high success rate in a constriction learning concept, a quality interaction experience, and enhanced educational services. We evaluated the accuracy of five multimodal inputs, including Finger Touch (FT), Hands Up (HU), Hands Down (HD), Voice Command (VC), and Click/Typing (CT). The results indicate an average accuracy of 90.8%, 87%, 88.6%, 91.8%, and 87%, respectively, demonstrating the effectiveness of our proposed approach.