Precise localization method for faulty idlers in belt conveyors based on visual and auditory bimodal fusion

Idlers are key load-bearing components in belt conveyors, and their proper functioning is essential for ensuring the continuous and efficient operation of bulk material transportation systems. Traditional methods for monitoring idler health are typically based on a single modality, such as sound or vibration, which has limitations in monitoring range and accuracy. This study presents a novel method for accurately locating faulty idlers by fusing image and audio modalities. This approach achieves pixel-level segmentation of faulty idlers guided by audio signals, addressing the limitations of image-only methods, which cannot detect sound sources, and sound-only methods, which struggle with accurate localization. Specifically, this work proposes a module for converting one-dimensional audio features into two-dimensional image features, a module for correlating and spatially aligning audio and image features, and a pixel segmentation module. These innovations enhance both the monitoring range and accuracy of fault detection. Experimental results demonstrate that the proposed method achieves high detection accuracy and strong anti-interference capability, enabling precise localization of a single sounding idler in a large field of view. With the mean intersection over union (MIOU) as the evaluation metric, the proposed algorithm achieves 73.8% accuracy on a custom dataset and shows robust performance even under image and audio noise interference. This method plays a significant role in advancing the effective integration of multisource heterogeneous information and improving the intelligent health management of belt conveyor systems in complex environments. Our source code is available at https://github.com/jisi123/Belt-conveyor .