Multi-Sensory Visual-Auditory Fusion of Wearable Navigation Assistance for People With Impaired Vision

Visually impaired individuals face limited mobility and restricted independent navigation in complex environments. Improving mobility and independent navigation for people with visual impairments is essential. In this paper, we introduce wearable electronic glasses (E-Glasses) that utilize a target detection network to fuse visual and auditory information for searching desired targets. Integrated with the electronic glasses, we present a neural path planning that combines spiking neural and convolutional neural networks. Several participants took part in experiments that showcased the remarkable capabilities of the developed system in target detection and navigation. The experimental results revealed an impressive success rate of 95.46% for the target detection network, providing participants with more accurate target information. Additionally, the neural path planning network achieved a success rate of 92.60%, demonstrating a significant speed advantage compared to the enhanced $A^{\ast}$ algorithm. Note to Practitioners —This article aims to address the mobility and navigation challenges faced by visually impaired individuals in complex environments. Many solutions have been proposed to assist with wearable navigation by utilizing sensors to perceive the environment and provide path information to users. In this article, we introduce wearable electronic glasses (E-Glasses) with advanced target detection and path planning capabilities to help visually impaired individuals navigate more effectively in complex environments. We have developed a target detection network that integrates visual and auditory information to effectively search for desired targets. Additionally, we have developed a neural path planning algorithm that combines spiking neural networks and convolutional neural networks. Furthermore, experiments conducted in indoor navigation challenges demonstrate the feasibility of this approach. In future research, we will focus on further exploring the target detection network and refining the neural path planning algorithms to improve the overall performance of the system.