Deep Learning-Based Audio-Visual Speech Enhancement with Adaptive Noise Suppression and Perceptual Loss Optimization

In noisy environments, traditional audio-only automatic speech recognition (ASR) systems suffer significant performance degradation. Leveraging visual cues from lip movements can enhance robustness, as demonstrated in early works on audio-visual feature fusion. This paper proposes a novel deep learning-based framework, AV-DenoiseNet, which integrates convolutional neural networks (CNNs) for visual feature extraction, recurrent neural networks (RNNs) for audio processing, and a transformer-based fusion module for multi-modal speech enhancement. Building on foundational methods like linear filtering for noisy audio enhancement and adaptive noise suppression, we incorporate recent trends such as attention mechanisms and end-to-end training. Experiments on the AVSpeech and GRID datasets show up to 35% relative reduction in word error rate (WER) at low signal-to-noise ratios (SNRs) compared to baselines. Simulation results, including spectrograms and performance curves, validate the model's efficacy for real-time applications.