Cross-Modal Attention-Based Multimodal Speech Enhancement Framework

In loud settings, improving speech quality for better communication is still a tough task. This paper introduces an audio-visual multimodal speech enhancement framework that leverages lip-reading information from visual modalities to guide audio denoising. The proposed model integrates cross-modal attention mechanisms to fuse audio spectrograms with visual lip features, achieving superior noise suppression compared to traditional audio-only methods. Experimental evaluations on a simulated dataset demonstrate significant improvements in key metrics: SNR reaches 16.2 dB, STOI 0.88, and PESQ 3.5. Ablation studies confirm the visual modality's contribution, with a 26% SNR drop when removed. The framework's robustness is validated across noise types, with Gaussian noise yielding SNR 17.2 dB. Visualizations, including spectrogram comparisons and attention maps, illustrate effective cross-modal fusion. This approach advances multimodal signal processing, offering practical applications in video conferencing, hearing aids, and surveillance systems. Future work may explore real-time implementation and diverse datasets for broader generalization.