Robust audio–visual speech enhancement under visual occlusion using TITR-CNN and silence-aware MALA-EM

Audio-visual speech enhancement (AV-SE) leverages the strong correlation between lip motion and speech acoustics to improve robustness in noisy environments. However, practical AV-SE systems remain limited by three major challenges: (i) visual corruption caused by partial occlusions in the lip region, (ii) spurious “hallucinated” harmonics during silence segments where only background noise is present, and (iii) the restricted expressiveness of deterministic posterior inference in recurrent variational autoencoders (RVAEs). This paper introduces a unified generative framework that jointly addresses these limitations by integrating an occlusion-robust TITR-CNN visual reconstruction front-end, a silence-aware latent noise prior, and hybrid MALA-Barker and MALA posterior samplers within the AV-RVAE architecture. TITR-CNN reconstructs occlusion-free lip-region frames through trendremainder temporal decomposition and gated inpainting, yielding stable visual embeddings even under 20–60% irregular occlusions. A silence-aware latent prior estimated from noise-only frames regularizes posterior samples and suppresses structured artifacts during non-speech intervals. Finally, MALA sampling enables accurate and stable posterior exploration, outperforming deterministic EM inference.Experiments on NTCD-TIMIT demonstrate consistent improvements across SI-SDR, PESQ, and STOI metrics, with the combined TITR+Sampling+NoisePrior configuration recovering most of the performance lost due to visual occlusion and achieving results close to clean-video inference. These findings highlight the complementary roles of visual reconstruction and stochastic latent inference in building robust multimodal generative speech enhancement systems.