Enhanced Audio-Visual Speech Enhancement via Attention-based Sparse Graph Convolutional Neural Networks with Fire Hawk Optimization

Audio-Visual Speech Enhancement (AVSE) has the vision to enhance speech quality and requite intelligibility by using both audio and video inputs, which is quite useful in noisy conditions. This approach acknowledges the fact that modalities such as lip movement may be equally informative of the auditory signals as the actual sound in cases where audio signals are distorted. In this research, an innovation approach to the AVSE is presented, which involves several state-of-the-art techniques. The data preprocessing step uses Z-score Min-Max Normalization to normalize the input data to help enhance the feature extraction process. Differentiable adaptive short-time Fourier transform (DA-STFT) is employed as the method of audio feature extraction that provides key temporal information. In the feature extraction process, a local directional structural pattern (LDSP) is utilized for the extraction of visual features since it examines the moving patterns on the face effectively. The fusion of data is done using a quadruplet depth-wise separable fusion convolutional neural network (QDSFCNN), which enhances the amalgamation of the audio and visual cues. Classification is done using an attention-based sparse graph convolutional neural network, known as ASGCNN, since the focus is on the relevant stipulations. Finally, to achieve the highest performance level, the fire hawk optimizer (FHO) is used for tuning parameters in the presented model. This combined approach presents advanced developments in AVSE, which provide improved speech quality in difficult conditions.